By David Spark, CSO
Having poor security is bad. Not having an incident response plan is far worse.
“It is not a question of ‘if’ but more a question of ‘when,’” added Hayter. “With a tested data breach response plan in place your chances of minimizing the impact are greatly enhanced.”
How should you respond? How soon does management need to know? What do you say to the public? Is there one guidebook? We asked experts for advice on what to do when you’re breached. Here’s what they recommend.
1: Before discovery, assume you’re already breached
“We all need to start operating under the assumption that in today’s threat environment someone or something malicious is already on our networks,” forewarned David Gibson (@dsgibson), VP of marketing, Varonis.
Looking for holes in your perimeter will help you prioritize what to strengthen.
“Make a hacker’s job much harder even if they do find their way inside again,” added Gibson. “Set up controls that monitor and baseline normal user behavior on the data itself, identifying and locking down sensitive data, and enforcing a strict least-privilege model.”
2: Assess and contain the problem
“Isolate the affected devices from the rest of the network to prevent the attack from expanding and possibly completing its mission,” said Michael Pittenger (@black_duck_sw), VP of product strategy, Black Duck Software. “Sophisticated attacks take several steps after the initial infection.”
“Usually we don’t want to actually shutdown servers as that may remove evidence, but instead block the servers from the public internet, and block the public internet from the servers by using the firewalls,” added Denny Cherry (@mrdenny), owner and principal consultant, Denny Cherry & Associates Consulting.
“After you’ve removed all previously authorized users from the environment, it becomes easier to spot the inappropriate activities – anything that moves becomes suspicious,” said Dwayne Melançon (@ThatDwayne), CTO, Tripwire.
3: Wait, maybe you don’t want to shut everything down
“Decide whether to shutdown immediately and thus notify the hackers they have been discovered, or wait and potentially get better information about the extent of the breach,” explained Jeffrey Bolden (@jbolden1517), managing partner, Blue Lotus SIDC. “The people breaching your system to collect information are professionals and work hard to avoid detection so that they can continue to collect information. If you shut down immediately the people organizing become notified they’ve been detected, and the breachers will take precautions for months so as to avoid detection while you are auditing, thus harming your counter-intelligence efforts.
“On the other hand, not shutting down immediately creates liability issues. You are now knowingly allowing information to be disclosed. Any damages move from a failure of best practices (hopefully), or negligence, to potentially a deliberate act. This is a difficult choice that you are going to need to make almost immediately at a time of high emotion and low information.”
4: Move quickly
“Think in terms of minutes, not hours,” added G DATA’s Hayter. “Every minute a breach of your IT systems goes on could result in the exfiltration of millions of pieces of information.”
“You need to decide how long to wait until you disclose the breach to the rest of the organization and to the public,” continued Scap who advises you “learn as much as you can as quickly as possible about what happened, before disclosing the breach.”
5: Talk to your lawyers
“The knee-jerk reaction is to fix all the things, patch all the holes, admonish all the sysadmins and then tell everyone what a great job you did. Technical approach aside there is so much more to look at,” said Thom Langford (@ThomLangford), CISO, Publicis Groupe. “Was it really a breach? What were the real implications? What is the actual impact? Do you need to respond to third parties from a legal perspective? Who really is to blame?
“If there was one thing I would say about any breach, and it pains me to say it, is that the first people you need to speak to are the lawyers (followed closely by the communications folks).”
“Place the investigation under attorney/client privilege,” added Kirk Herath, VP, chief privacy officer, associate general counsel, Nationwide. “The privilege will permit open and frank discussion.”
“Gone are the days of a breach simply being about technology, it is now just as much about legal obligations, market perception, and often just the ability to publicly say sorry,” concluded Langford.
6: Understand who the real victims are
“Often companies make the mistake in believing that they are the victims,” said Quentyn Taylor (@quentynblog), director of information security, Canon Europe. “If the breach involves people’s data my key takeaway would be to treat the loss of data as if it was your own personal data that had been lost. How would you want the company you represent to act? When you put yourself in the mind of the customer the path will be clear.”
“Apply the ‘golden rule’ – treat others as you would wish to be treated if your personal information had been compromised,” suggested Bill Schrier (@BillSchrier), senior policy advisor, State of Washington Office of the CIO.
7: Inform your customer base quickly to preserve trust
“There are many prescribed ways to handle the response to a security breach, but in my professional opinion, the best approach is to respond quickly and honestly,” suggested Kris Lahiri (@KrisLahiri), co-founder and CSO, Egnyte.
“Transparency is key for businesses, especially with customers, and letting them know of any breach as soon as it occurs is a must,” advised Rick Spurr (@ZixCorp), CEO, ZixCorp. “After all they are the ones being affected, and in some cases it can impact their lives on a much larger scale.”
“Understandably, most organizations fear a loss of reputation from public acknowledgement of an intrusion. However, a prompt, open, and technically accurate response can actually enhance an organization’s image by conveying integrity and responsibility,” said Don Maclean (@DLTSolutions), chief cyber security technologist, DLT Solutions. “A breach can make any organization look bad, but if they bite the bullet, state publicly what happened, and share the technical analysis, they can minimize the damage, or even come out with a net gain in reputation.”
8: Be transparent about how information will be released
“Even if you do not know the root cause or extent of damage, you should be proactive in telling your stakeholders and customers what happened, what you have done so far, what you will do next, and when you will update them again,” said Terence Ngai (@TerenceCNgai), head of cloud delivery management, HP.
“Be transparent about the steps, but don’t give information until it has been validated,” added Adam Ely (@adamely), co-founder, Bluebox Security. “Telling the media that 1.4M records have been compromised and then 24 hours later updating that to 9M serves no one. It just leads to more confusion and more effort. By stating there was a compromise of at least a certain number and setting a timeline to give more information, the situation can be better controlled and those that actually matter, the victims, can know when to expect information.”
9: Be open about what you do and don’t know
“Stating what you do and don’t know helps to build trust through humility and transparency. Along the same lines, it’s also important to quantify or qualify your degree of confidence in what you think you know,” Jeff Lowder (@agilesecurity), director of global information security and privacy, OpenMarket. “For example, ‘We think the breach only impacted customers in New York who did business with us between 2014 to the present, but it may also include customers in other states. We’re continuing to investigate and will post an update on the scope of the breach as soon as it is available.’”
10: Keep an open mind when digging in
“A breach is generally not discovered when it first happens and on the system it originated on,” said Blue Lotus SIDC’s Bolden. “Don’t assume that you caught the breach on day one and system one.”
“You might think you have solved the issue to later realize you only fixed part of the condition,” said Amir Mizhar (@SafeTData), founder and CSO, Safe-T Data. “It is though safe to ‘assume’ that you are being hacked by one or more criminals and your data is being sold!”
11: Ensure easy access to critical data
“We put all of our efforts up front in order to deploy the right tools and solutions in place to access necessary data in minutes or seconds,” explained Demetrios ‘Laz’ Lazarikos (@vArmourNetworks), CISO, vArmour.
“Make sure your systems indeed have logs which can be collected,” added Safe-T Data’s Mizhar. “This information is critical and must be retained for proper analysis and identification of the cyber criminals.”
12: Coordinate your communications
To achieve this synergy, Andy Ellis (@CSOAndy), CSO, Akamai, suggests that “all communications be jointly written by the technical, legal, and communications teams; not in a stove-piped fashion, but with actual collaboration. That way, differences of opinion can be quickly identified and sorted out to line everyone up with the business approach.”
“If your teams aren’t already interacting like cogs in a well-oiled machine at the moment your servers are compromised,” said Daniel Page (@aseohosting), director of business development, ASEOHosting, “Then it’ll be that much longer before you’re able to break the bad news to your clients – and you need to do so.”
13: Don’t panic. Have a plan.
“Do not panic. Panic takes way too much time you could be using to plan. Planning should also take very little time,” advised Edward Haletky (@texiwill), managing director, The Virtualization Practice.
“Don’t rely on your instincts,” continued Wright. “Pull out your security incident response policy and procedure, and follow it.”
“If a breach occurs, our team organizes quickly and starts going through the standard operating procedures. There’s no chaos, no heroes, and everything is done by the letter in order to bring certainty to the situation, and to understand the breadth and depth of the breach,” said Ty Rollin (@tyrollin), CTO, Mobiquity.
“Hopefully you respond by looking at your incident response plan that you have been practicing as a company for the life of your business,” said Daniel Riedel (@riedelinc), CEO, New Context. “If you haven’t already, start writing that document today.”
14: Be prepared, but be flexible
“You need to strike the right balance between predictability and flexibility, because everything about a particular incident response is going to be slightly different,” added Marcus Ranum (@mjranum), CSO, Tenable Network Security, who suggests having a process framework that allows the flexibility to deal with the unknown and unexpected.
15: Change your environment
“With breach response, an immediate set of instructions must be deployed to clients and end users regarding password changes and other security measures,” advised Steve Prentice (@stevenprentice), senior writer, CloudTweaks. “Given that such messages are both scary and inconvenient, I always recommend a company perform a drill, which would help create the right type of messaging and guidance.”
16: Prevent the second wave of attacks
“After suffering a breach, it’s easy to assume that the worst is over. But, that’s when cybercriminals will try to leverage the heightened sensitivity and security awareness of your customers and employees,” said Greg Mancusi-Ungaro (@gmancusiungaro), CMO, BrandProtect. “The breach is often only the beginning of a stream of phishing emails, impersonation schemes, and fraudulent social activities, targeting your people and customers when they are at their most vulnerable state… Don’t get caught looking only at your perimeter.”
17: Try not to be overwhelmed
“From the technical perspective, you will be overwhelmed – no matter what,” warned Tripwire’s Melançon. “Take a top-down, risk-based approach so you start with the systems, processes, and business services that matter most to your business. Focus on a process of analyzing the state of your systems to quickly divide them into three groups – systems you know match your expected configuration guidelines, those that definitely don’t, and those that are questionable. Take the known bad systems offline immediately, protect the trustworthy ones, and begin digging deeper on the ones you’re unsure about.”
18: Practice, practice, practice
“The single most important response to a breach happens long before the breach ever occurs. It is to prepare an incident management plan,” said Eric W. Cowperthwaite (@e_cowperthwaite), VP, advanced security and strategy, Core Security.
Orlando Scott-Cowley (@orlando_sc), cybersecurity specialist at Mimecast recommends “getting key executives in a room for an immersive simulation so as to set expectations and guide how everyone responds after a real breach takes place.”
“If you simply have a policy along with paper procedures, then your response will be disjointed, poorly conceived and you will be in hero mode,” said Jeff Bardin (@bardinjs), chief intelligence officer, Treadstone 71. “Instead, practice your incident response procedures in military fashion (learn the academics; practice a table top walk through; test to a table top practical exam; dry run through several examples, and then wet runs) so that your organization will be prepared.”
19: Get external support
Once you know you’ve been breached, you need to “figure out how bad the damage is, and how much data the attacker got access to. This is where the specialists come into play,” said Cherry of Denny Cherry and Associates’ Consulting.
Not only will an outside firm have more expertise dealing with breaches, but “they can be more objective when things are crazy,” added Tripwire’s Melançon.
You can often find breach expertise among your industry’s colleagues.
“Immediately upon identifying an incident of concern, a security response team should share its findings with a trusted network of peers so the information can be correlated with other reported incidents,” said Paul Kurtz (@trustartech), CEO, TruSTAR Technology. “Sharing incident information early in the response process provides an opportunity for the security team to learn how other organizations may have dealt with a similar attack, and collaborate on a mitigation strategy. Leveraging external peer expertise is perhaps the most overlooked step in the incident response chain, but this is beginning to change in innovative organizations.”
20: Use the experience to move security forward
“You likely are going to be given a little more latitude to fix things,” said Mark Herschberg, principal, White Knight Consulting. “You may find a bigger budget or more authority for policy changes (e.g., requiring two factor authentication). It doesn’t mean go hog wild, but it does mean you can use the current spotlight to move along changes you know will improve security.”
Cherry of Denny Cherry and Associates’ Consulting thinks you should be aggressive with your spending after a breach: “Review the network security budget that you have in place today, and triple it. You apparently are going to need it.”
CONCLUSION: You can survive a breach
No one wants to be breached, but it’s a reality that happens even to the best. People point to huge enterprises and the U.S. government being breached, but even companies who do security for a living, such as Kaspersky, are susceptible. What is evident from all the conversations I’ve had with security professionals is that no one boasts that they’re impenetrable. In fact, if they haven’t been breached, they know it’s a very real possibility, and they’re wisely preparing their team and the rest of the company for the inevitable case of ‘when,’ not ‘if.’