Who ya gonna call?

March, 1999

Do you have an approved incident response procedure in place? Getting your IRP set up before something happens could save you a lot of headaches -- not to mention your job

Summary
You're sitting at your desk, browsing through the system and notice something isn't right. Further investigation shows that there may be an intruder in your system. This wakes you up faster than a can of Jolt Cola, and you're all primed to take action. The question is: what kind of action? A system administrator's perception of an appropriate incident response may be very different from corporate management's. An approved incident response procedure can save you from major troubles -- and embarrassment. Here's how to do it right. (3,600 words)

  WIZARD'S GUIDE TO SECURITY  
By Carole Fennelly

Let's consider how you would react to the following scenarios:

  1. A user calls complaining about sexually explicit images that have appeared on her computer.

  2. An administrator at another site contacts you regarding the actions of one of your users. This could involve anything from nasty e-mail to attacks on the site. The administrator may be:

  3. You are experiencing repeated attacks from the same address. You've contacted the registered technical admin who:

  4. An authoritative agency such as the FBI or CERT contacts you directly and informs you (sounding very official) that it is conducting an investigation of one of your users and expect full cooperation.

  5. You notice that an executive in the company is browsing child porn sites, and you suspect he may be saving some images on company systems.

  6. A new employee brags to you about a back door she left in her last company.

  7. You think your system may have been compromised.

In each of these cases, your first response should be to inform management. Unfortunately, this is usually the last thing a system administrator actually does. The administrator will often make an assumption about the appropriate response. The problem is that there are political and legal issues to consider that the administrator is not qualified to address. In the precommercial days of the Internet, system administrators in research organizations freely exchanged information without informing management. Most, if not all, of the decision making was left up to the administrator. Today, corporations have strict policies regarding the release of proprietary information. In a commercial environment, even releasing information to CERT could get an administrator in a lot of trouble with management.

System administrators often complain that management doesn't understand technical issues. If management can't understand the technical issues, perhaps they need to be explained in clearer terms. The movie Armageddon has a scene in which the scientist is attempting to explain to the president the meteor's trajectory, mass, and velocity. The president is confused until the explanation is reworded to "a rock the size of Texas is going to hit Earth and destroy everything." Management just needs to know the issues so they can make a decision. The administrator's job is to understand the technology and produce the results management asks for. Don't give management data paralysis -- when they start giving you that deer-in-the-headlights look, you know you've gone too far.

The system administrator can make the process easier by interpreting the technical issues into business terms and developing an incident response procedure that has management's approval. This way, the administrator knows what actions are preapproved. For example, you may be authorized to call the technical contact for an address that is conducting repeated probes of your site.

Incident response procedures are nothing new in the industry. Many of the guidelines for developing an IRP were established during the precommercial days of the Internet and focus on how to proceed if your site has been compromised. While you should certainly have such a procedure in place, hopefully you'll be prepared to respond to an incident before it gets that far. Many sites are installing intrusion detection systems -- the technical equivalent of a burglar alarm. But what good is such a system if you're unprepared to react to the alarms?

The sidebar at the end of this column includes an outline I use for preparing an incident response procedure. Often, the issues that must be considered are the same across sites, but the details may vary, depending on the site's security policy and configuration. Hopefully, this outline will help you develop your own procedure.

Examples of response to above scenarios

While the appropriate response to a situation depends on your site's policies and procedures, it's very important to keep accurate chronological logs of events with details such as file names and the names and phone numbers of all people involved in any security incident. Check with your legal department for guidelines on evidence gathering. This is how I might respond to the hypothetical situations I mentioned earlier:

Final thoughts

You may disagree with my suggested responses to these security scenarios. If so, that's all the more reason to develop an incident response procedure that has the approval of your management. If getting started is your problem, use the attached IRP and edit it for your site. Come up with some of your own scenarios and discuss with management how they should be handled -- then put it in writing. Once you have a response procedure, it should be tested by the audit department as part of the regular security assessment. Whatever you do, don't wait for a major incident to occur before you write one.

Next month

Just as I was about to write about installing Sendmail 8.9.2 on a firewall, an announcement was made about the release of Sendmail 8.9.3. Oh, well -- that saves me from explaining how to install the patch. Next month, I'll describe the installation steps for a firewall and internal mail gateway and include sample configuration templates. <

Disclaimer: The information and software in this article are provided as-is and should be used with caution. Each environment is unique and the reader is cautioned to investigate with his or her company as to the feasibility of using the information and software in the article. No warranties, implied or actual, are granted for any use of the information and software in this article and neither author nor publisher is responsible for any damages, either consequential or incidental, with respect to use of the information and software contained herein.

Resources and Related Links
   

Sample Computer Security Response Procedure
 


Introduction
The purpose of this document is to provide a procedure to recognize a security incident and take appropriate action. This document also defines the chain of command for reporting security incidents and details the responsibilities of each level. A computer security incident is defined as use of company computing facilities in an unauthorized manner. See your Company Security Policy for further details.

Chain of command
The company incident response team must be notified of all security incidents. The incident response team is responsible for coordinating the response and is preauthorized to take specified actions. The incident response team will include two or more of the representatives of the following organizations to ensure redundancy: System Administration, Security Administration, Management, Audit, and Legal. The technical staff should focus on the technical issues while the other members direct policy issues. All members of the IRT must be reachable at all hours or designate a backup. Attached will be the contact information for the IRT members, including pager and home phone numbers. All IRT members will have an emergency contact list for senior management in the company, including home phone numbers. This list is confidential, but must be accessible at all hours.

Categorizing a security incident
The first priority for a security incident is to determine the type and status of the incident.

  • Is the incident a penetration attempt or a violation of the security policy?
  • Is the incident presently active?
  • What is the source of the incident?
  • Has the system been compromised?
  • Which organizations are involved?

Penetration attempts
The most obvious indicator of a penetration attempt is your Intrusion Detection System (IDS) informing you that you have one. You should be familiar enough with your IDS to recognize whether or not the alert is important. Remember -- an IDS can only tell you about the traffic it monitors. Internal users are still considered the biggest security threat. You also must have a mechanism to receive alerts by pager since you will not be in front of the monitor 24 hours a day. The best way to be able to recognize a security incident is to be observant and familiar with normal conditions. System administrators should be alert to:

  • Anomalies in log files, such as a user logging in at odd hours or frequent failed login attempts
  • The "last" command showing a user who doesn't travel logging in from an unusual location
  • System programs disappearing or behaving strangely
  • Unusual processes running
  • Entries added to the password file
  • Security monitoring software complaining (obviously)
While these signs may not conclusively prove a security incident, they warrant further investigation. The incident response team should be informed even if an incident is not confirmed, unless it can be dismissed outright. While attempting to confirm whether an incident has occurred, the system administrator should capture the output of commands run and log files. It is also helpful if information that appears on the screen can be printed and saved. All this data should be kept off of the system in question and write-protected. No discussion of a possible incident should occur via e-mail. Even if it turns out not to be an incident, it doesn't hurt to save the data.

Violation of security policy
Because security policy violations are committed by internal users, some delicacy is required to handle the situation since it may result in legal action. It is very important to involve senior management, Legal, and Human Resources to limit the company's exposure to liability. Violation of the security policy may be determined either by routine examination of system log files, or by a reported incident.

Assigning priority and responding to a security incident
How you respond to a security incident is dependent on the severity and type of incident. In all cases, details of the incident should be discussed only on a need-to-know basis. A system administrator may find it tempting to tell everyone about the porno sites that an executive was caught browsing, but confidentiality is crucial. A chronological log must be maintained of all investigative measures. If this log might be used as legal evidence, it is to be signed by all involved members of the IRT. Also, a backup should be made of the data, preferably on write-once media. At the very least, the media should be write-protected.

Priority 1
An active penetration attempt that is currently in progress. This may either be directed at the company internal network from outside, or originating from the internal network and directed either inside or outside the external network. This list is a bare minimum and should be modified to reflect your site's configuration. The point is that you have to act fast, and you might forget important details. The incident response team should not be constrained by waiting for management approval to take action.

  • Call incident response team, if not already involved.
  • Technical staff should get as much information from screens and log files as quickly as possible and store offline. Files to be saved include: 
    		
/etc/passwd	
/var/log/syslog	
/var/adm/messages	
/var/adm/sulog
/var/adm/lastlog
/var/adm/utmp
/var/adm/wtmp
    Also save the log files from any application or security software you may be running, such as tcpd.log and http.log. Make a list of all your firewall log files and include them here. You may not have time to analyze whether the intruder is currently in the system. Get as much data as you can and analyze it offline later. It may be helpful to have a script written that you can run. Just don't make it too complicated: 
    /usr/bin/ls -laR /tmp > /archive/tmp.save
/usr/bin/ps -aux > /archive/ps.save
    Add to this list any other commands that would be useful. Run the commands in order of priority. Get the quick commands early in the list in case you run out of time. It would be very useful if /archive is a write-once CD, but failing that, get the files off the system and on backup media as soon as possible.
  • In the meantime, the management members of the incident response team should be informing management and relevant organizations.
  • If there is no management representative immediately available, all members of the IRT are preauthorized to take the following actions:
    • Contact the involved Internet service providers and provide appropriate log data to trace the attack.
    • Disconnect the company network from the Internet.
    • Request that the ISP for the originating attacker disconnect the attacker at its end.
    • Under no circumstances will any administrator or IRT member be permitted to take retaliatory action against the originating network with an intention to cause harm. No matter how you are provoked, don't open the company up to criminal or civil liability.
    • Only senior management may authorize informing outside agencies such as law enforcement, the media and security organizations.
    • No attempt will be made to communicate directly with the attacker.
    • A complete backup of the system must be performed to preserve the data. Media must be clearly labeled, dated, and signed. You may need to demonstrate the integrity of the backup by showing comparative checksums against the original file. A WORM drive is preferred as a backup device.
    • If the system has been successfully penetrated and compromised, the following steps must be taken:
      • Disconnect from the Internet, backup the system and begin analysis
      • Reinstall system binaries from distribution media
      • Install all recommended patches and all security patches
      • Change all passwords, including users
      • Have the system audited by an external security consultant

Priority 2
An active violation of the company security policy that could expose the company to criminal or civil liability. An example may be that a user is currently sending or receiving harassing images or messages.

  • The incident response team managers will inform senior management and follow Human Resources guidelines.
  • The technical members of the IRT will discreetly gather all log information and save to backup media.
  • There will be no discussion of the incident with anyone who is not on a need-to-know basis, as approved by senior management.
(The following describes incidents that are not currently active and do not have the same degree of urgency.)

Priority 3
There is evidence that a serious penetration attempt has been made, but the attack is not currently active. Follow the same procedure as in Priority 1, with the following exception: The IRT must get senior management approval to disconnect the company from the Internet or before sending log data to the ISP.

Priority 4
There is evidence or a report of a security policy violation that could expose the company to legal liability. Follow the same procedures as in Priority 2.

Priority 5
There is evidence of minor penetration attempts or policy violations. The technical members of the IRT may follow up by contacting the originating ISP's technical contact and monitor the log files. The management members of the IRT will inform appropriate management of policy violations.

 
Return to article

Copyright 2003, ITworld *-* Please direct questions and comments to Carole Fennelly. Permission is granted to quote, reprint or redistribute provided the text is not altered, and the author and ITworld are credited.