Forensics

Getting to the bottom of a security breach

Summary
What are some of the technical steps you can take to trace an intruder? In this article, Carole Fennelly describes an actual security breach and the methods she and some other security professionals used to trace it. Included as sidebars are the actual emails sent back and forth among the parties involved. (4,000 words)

As the purpose of this article is to discuss techniques, I am taking some license by combining the text of several emails in order to make them more readable. I am also sanitizing the information -- replacing actual user names, domain names, and IP addresses with pseudodata -- in order to maintain the privacy of the parties involved. Any similarity to actual names and addresses is purely coincidental.

Were this information being utilized in a legal case, the unadulterated and unabridged emails, addresses, and names would need to be preserved.

Houston, we have a problem

BNC (BouNCe) is an IRC proxy daemon written by James Seter. With it, users can bounce IRC traffic to mask the traffic's originating IP address.

BNC isn't malicious code in and of itself. It can be used for quite legitimate purposes, such as protecting a PC by covering its real IP address with the address of a system better able to withstand an attack. While this is a form of security through obscurity, a little obscurity can be helpful, especially when facing DoS attacks.

Computer crackers might install BNC on a site they broke into in order to cover their tracks as they attack another site. They could also use it to masquerade as an employee of Company X in a chat room. It should be noted that the person using BNC must already have a way to get into the network in order to use it. BNC itself is not a back door, though its presence may indicate that there is an intruder in the network.

In this particular case, Mac suspected that BNC was being used because he could not see a user logged in at the time of the incident.

Initial recommendations

?r--r--r--   1 root     root            0 May  6  1998 .name_service_door
?rw-r--r--   1 root     root            0 May  6  1998 .syslog_door

On my Solaris 7 system, the file looks like this:

Drw-r--r--   1 root     root           0 Mar  6  1999 /etc/.syslog_door

It looked to me as if the ? representation was there because the system was an older release; in the present version, D indicated a door. I could not confirm this in the manual pages, however, and later resorted to asking Casper Dik, the walking manual of Solaris, for clarification.

Casper confirmed that, in Solaris 2.5.*, doors weren't documented and file/ls didn't know about them. What's more, .name_service_door is used by the libraries to contact nscd and .syslog_door is used by syslog() to check whether syslogd is still up and running.

A Puppy to the rescue

RFP started his investigation by tracing the suspect IP addresses to their source ISP (see Sidebar 4). From there, he picked up the technical contact for the site, Bailey Rabbit (not his real name). Because there was no email address listed for Mr. Rabbit, RFP looked him up by name in the Network Solutions database and identified the ISPs where the rogue connections originated. Further investigation required an account on the compromised system, which Mac was authorized to provide.

In his original mail, Mac noted a copy of httptunnel in the account of a user who had left. It was initially thought that this was the source of the compromise, though this turned out not to be the case (see Sidebar 5). This illustrates that there are often anomalies found on a compromised system that have nothing to do with the event being traced (see Sidebar 6).

A little help from proc

Using the /proc tools, RFP proceeded to find the location of the BNC proxy (see Sidebar 9). He also discovered that ls wasn't behaving properly, which suggested that a Trojan ls had been installed. I later confirmed with RFP that the ls -l command gave no output, when it should have at least given total 0 if there were no files to be listed in a directory. Because RFP had previously remarked that ps was not helpful, it looked like a rootkit was installed on the system.

Mac had a few questions about the methods that RFP used (see Sidebar 10).

Suggestions

Brian Martin (of the DSIC Security Group) and Jay Dyson (of OneSecure) suggested running the find command to look for SUID and SGID binaries:

find /  \( -perm -4000 -o -perm -2000 \) -exec ls -al {} \; > /tmp/suids

RFP noted that on Linux +6000 would report SUID, SGID, or SUID+SGID. On Solaris, looking for +6000 will only report binaries that are both SUID/SGID.

Jonathan Klein (of Wizard's Keys) suggested using lsof, a publicly available tool that lists open files. It is only useful for active processes, but it does provide a great deal of information. Of course, it is better to already have a copy of lsof on the system before you have an incident. See Sidebar 11 for a sample of the information it provides.

Jon also suggested having a copy of traceroute on the system, which Sun (at last!) included with Solaris 8.

Post mortem

For now, the systems have been patched, though, as Jay Dyson pointed out, "Patching an already-compromised box is like slapping on band-aids to stop a cerebral hemorrhage. It won't work. Take the box offline, reinstall the OS, and lock it down. Anything else is a waste of time and effort."