Generative AI in DevOps Toolchains: Incident Summarisation, Runbook Generation, and On-Call Automation

Abstract

Generative AI models are increasingly embedded in DevOps toolchains to automate toil-heavy operational tasks. This paper evaluates three GenAI use cases�automated incident summarisation, runbook generation from postmortems, and on-call assistant chatbots�deployed in production at two large-scale SaaS organisations. Using a combination of expert evaluation, BLEU/ROUGE scoring, and controlled A/B testing of engineer response times, we find incident summaries achieve 4.1� faster triage initiation, runbooks generated from postmortems are rated as accurate and actionable in 74% of cases, and on-call chatbot interactions resolve 38% of pages without human escalation. Hallucination risks and mitigation through retrieval-augmented generation are discussed.