Incident Response Methodology

Incident Response Methodology plays significantly into how quickly the issue is resolved. The cause of the outage could be the result of a network configuration change, software upgrade, scheduled maintenance, surge capacity failure or simply a code change. Any one of these issues could cause hours of downtime. Knowing that an hour of IT downtime can easily cost over $100,000, it is important for every IT team to have a preconfigured, well-considered incident response plan to minimize downtime and keep key stakeholders informed.

The incident response management team that responds to an IT outage will be successful if they have a strong leader along with a strong team to manage the outage. The IT outage team must have a preassigned leader as well as an assigned team whose job is to manage the outage. Otherwise, the resulting disorder will take away critical time from responding to the incident. Time will be wasted trying to figure out who should handle which part of the outage. If however these leadership and management roles are predetermined, then the team members can get to work right away and start resolving the issue at hand.

To ensure that there is no guessing as to who will be notified, the response team needs to have their names listed in a digital scheduler so that as soon as the outage occurs they are notified of the issue.  This team should also have back-up responders included in the digital schedule so that if anyone is out sick, they are sure to have a backup.

Alerting of the IT outage team should occur on multiple channels such as SMS, email, phone call and smartphone application. Just about everyone in tech has their smartphone surgically attached to their hand. As such, getting an alert via smartphone should be the first step. The smartphone alert will be the most effective way of grabbing the engineer’s attention. SMS, email and phone calls probably best serve as effective backups. The goal is to provide primary and secondary forms of alerting to make sure that there is virtually no chance of letting the team members remain unaware of a brewing situation.

Once the IT outage response team has been alerted to the issue, IT needs an incident response management tool through which to communicate and make sure their messages are received immediately and prominently. In the high stakes game of managing IT outages, emails and SMS are not effective tools. Email is really a form of communication focused on an exchange between two people. As soon as multiple people get involved in an email thread, the communication gets muddled. Additionally, email is not good for real-time communications. There is inevitably a delay which prevents rapid resolution of the issues at hand.

Similarly, SMS faces similar issues to email in that it is not meant as a collaborative tool or a tool that enables work to get done. The text messages are not integrated into the work thread and so they remain separated. In addition, and this is also an issue faced by email, it is impossible to query databases or execute functions from the command line of SMS or email.

More importantly, these tools don’t encourage collaboration, which is exactly what the team needs to effectively manage outcomes. Instead, if a team has an incident management system that allows for real time chat, they will be much more effective. Indeed, the strength of an IT team revolves around its ability to quickly resolve incident, chat plays a crucial role in rapid incident resolution.

Instead, teams should grab onto a smartphone application that elevates high stakes communications and separates it from the standard chat that occurs on straightforward ChatOps platforms. A technology like OnPage has the ability to link into standard apps like Slack and ensure that messaging continues on a separate, high priority channel.

At the same time that the members of the outage team are communicating, the leader of the team needs to make sure that any updates are reported to important stakeholders. These individuals can receive an alert on their smartphone application that keeps them apprised of important developments. Later on, when the situation is resolved, these VIPs can view the reporting details to learn about the meeting of SLAs and how well the team performed.

For incident response managers to further ensure success, the actions taken by the incident response team need to be documented and measured. This type of visibility can only really occur when there is effective reporting attached to the critical alerting platform.

The reporting tool should provide summaries and insights through data. This information should highlight a team’s effectiveness across multiple shifts and time zones. Team leaders can thus easily see trends, performance, productivity and understand how well their team is doing.