ITIL defines an incident as an unplanned interruption to a service or reduction in the quality of a service. Incidents have four priority levels: Critical, High, Medium and Low. Major incidents are typically classified as high-priority events based on the urgency and business impact of the situation.
A major incident impacts major business operation. Major incidents bring an organization’s entire operation to a standstill and impacts their revenue and bottom line. This can also have far-reaching consequences for the company’s reputation.
Major incidents commonly include:
Time is critical during a major incident and an organization’s ability in bringing back business to normalcy makes all the difference. Major incident management (MIM) helps distinguish between a real major incident versus an outage. The goal of MIM is to manage the incident life cycle and remediate the issue while minimizing disruption.
The Cloudflare 2019 global outage is an example of a major incident. A minor change in the rules used to detect anomalies resulted in major outages. Per Cloudflare’s estimates, systems were down for approximately 27 minutes and affected almost half of the internet’s accessibility.
The first step in the MIM process is identifying a major incident. Organizations encounter incidents every few minutes, so the challenge is distinguishing major incidents from the rest.
A key indicator of a major incident is that it affects many users, disrupting one or several critical services of a business. Incidents are often reported to a service desk technician or detected by monitoring tools that automatically trigger notifications when anomalies are identified.
When a major incident is detected, the relevant stakeholders need to be engaged to contain the situation and minimize business losses. There are three key groups that must be informed of the situation:
Assemble the major incident team to remediate the major incident. The team must consist of engineers, incident commanders and other key stakeholders, such as external consultants. All parties aim to minimize damage and resolve the issue.
In critical situations, emails and SMS are ineffective message channels. Messages are often missed and unaddressed, and the channels are unable to elevate high-priority messages. Incident alert management applications allow for real-time, secure messages for team collaboration.
The resolution stage occurs when the outage has subsided, and systems have been restored to full functionality.
Once resolution is achieved, the team should document the entire incident management process for future reference. If necessary, carefully implement process changes and ensure that other dependencies are not affected.
Post-incident analysis measures the performance of resources and systems in place. Incident managers conduct post-incident reviews to gain insight into the situation and event resolution process. This helps organizations become well prepared for future incidents. Message audit trails can be analyzed to get information about the team’s incident response.
While you can catalog an incident using your IT service management (ITSM) ticketing tools, there is very little you can do to manage the incident by simply using tickets. Ticketing tools only allow for unprioritized SMS and email incident alerts. This limitation inhibits team collaboration and slows down the incident remediation process.
Integrate ticketing tools with an incident alert management system. This way, you can convert tickets into intelligent alerts that can be sent to response teams whenever there is a critical incident. Teams can further collaborate on tickets and include resources into the conversation thread if required.
The incident alert management tool needs to be more than just an alerting service. Here are some requirements of any incident alert management solution:
