Forensics

Determining whether you've had a security incident can be a difficult, tedious task. We'll help walk you through the basics of when to look and what to look for as well as who to call if you're just not sure. We'll give specific examples for Sun Solaris and Red Hat Linux. Other Unix operating systems will have similar files and commands, but may vary slightly.

While this document is meant to be easy to understand, some of the information we present is potentially damaging and should not be undertaken lightly. That said, don't be afraid to look around--that's the best way to learn. If you feel like you're in over your head or don't have the time to make a thorough search, call CITES WSG. That's what we're here for.

And of course, the easiest way to clean up a security incident is to be proactive. For an Unix OS-specific guide to security best practices, visit the WSG Security Pages. Non-Unix security information can be found at http://www.cites.uiuc.edu/security.

Before you do anything, you need to decide what you are hoping to accomplish. In most cases, the goal will simply be to restore service and close the hole that allowed the exploit. To do that, you need to:

• Determine what vulnerability was exploited that allowed the incident to occur.
• Determine the extent of the incident. Did the attacker gain root access or just a user account?

If the attacker used an undocumented vulnerability, it is important to find that point of entry and release it through the proper channels to avoid its being used in the future. If it might be necessary to pursue disciplinary action for the attacker, it is critically important that you document your steps and the evidence you find.

For a formal investigation, it is generally best to collect and archive everything, but that is not always practical. Guidelines are available in RFC 3227. One of the more widely used computer forensics applications is the Coroner's Toolkit, which is especially useful in conducting a formal investigation.

If you intend to use your evidence in court, you are faced with some important considerations:

• Log everything and log frequently. Take notes with a pen (not a pencil) and paper.
• Maintain a chain of evidence. In court, you will need to be able to testify that there is no way the evidence could have been tampered with. Anyone who handles your notes or the computer in question should sign in and sign out.
• Get the complete picture. Using a tool like The Coroner's Toolkit (TCT) can help.
• Take advantage of redundancy. Don't trust any single source of information. Try to have multiple witnesses of any event.

If you're looking for evidence, chances are that you have a good idea whether there's been a security incident. While you're looking, try to compare recent system events to their possible causes. A machine may exhibit suspicious behavior for many reasons, so try to compare recent system events to their possible causes. For example, if you've been getting error messages about a failing hard drive, it's possible that that's what caused your file system corruption. Possible culprits for suspicious system behavior include:

• Hardware problems
• Software problems
• accidental deletion of system or user files
• malicious user
• hacked user account
• hacked root account
• Denial of service (DoS) attack

Keep in mind that no networked, multiuser system is ever truly secure. There are ways to see if you have been hacked, but you will never really be able to prove that you haven't. Some people get hacked and don't realize it for days, weeks, or even months! Most of the time, though, you'll know within a day. Here are some of the more obvious signs:

• Someone tells you your machine is attacking other machines. This may be a message from your network administrator, another sysadmin, or from the CITES Security Group.
• You see odd messages in your system logs. Examples include long strings of gibberish or messages from daemons you know should not be running.
• You notice your files have been modified. Extra directories may have been added or config files changed.
• Your system "feels" different. Perhaps it's a little bit slower or commands aren't running the way they normally do.

When investigating a machine that may have been hacked, it is important to remember that not all system utilities can be trusted. A smart attacker will replace crucial administrative tools with their own trojaned binaries. In one case, the attacker left the binaries alone, but modified the libraries that the utilities used! Typical targets for modification include /bin/ps, /sbin/ifconfig, /bin/netstat, /usr/bin/top, /usr/bin/slocate, /usr/bin/find, /usr/bin/md5sum, /sbin/syslogd, and /bin/ls.

Did you notice that md5sum was in that list? The new md5sum binary returns the old checksums to help the attacker avoid detection. Similarly, the t0rnkit matches the length of the files to the previous length (so an ls -l won't show a filesize difference) and matches the modify and access times to the previous values.

The point here is to remind you to investigate your system using trusted binaries from a system that is known to be good.

Keep in mind that no networked, multiuser system is ever truly secure. There are ways to see if you have been hacked, but you will never really be able to prove that you haven't. Now that you're good and paranoid, we'll show you some basic search procedures.

• Check your log files. Log files are typically found in /var/log/ or /var/adm/. You should be checking your logs anyway, which will make it easier to determine what is suspicious. Note that a missing or empty log may be suspicious.
• Look at running processes. Use ps -aux or ps -ef to see who's doing what.
• Watch for modified system files. Use ls -lart or ls -lartc to list all the files in a directory with the most recently modified listed at the bottom.
• Check your ports with netstat -an | grep LISTEN or by running nmap against your machine. Are extra services running? (Hint: you're probably in trouble if you find something on port 31337.) Are vulnerable services running?
• Check for hidden files and directories. Use find / -name \.\.\?\* -print to list all files and directories beginning with .. (excluding the previous directory, ..).
• Look for suid binaries by running find / \( -perm -4000 -o -perm -2000 \) -print.
• Use Tripwire, ls -lc, and md5sum to look for modified files.
• Check your crontabs (both root and user) and at jobs.
• Look at your patch history. Attackers will sometimes patch a machine to avoid having others "steal" a machine using the same vulnerability the original attacker used. This may seem helpful, but be aware that the attacker will typically leave a back door open in your system.
• Look for files hidden in plain sight. The t0rn rootkit, for example, adds the text file /lib/lidps1.so use file * to look for directories and ASCII files in directories that should only contain binaries or devices.

Because Unix filesystems are optimized for performance, deleted files are generally gone for good. In some cases, however, it may be possible to undelete files depending on the specifics of your filesystem.

The Coroner's Toolkit contains a utility to undelete files from non-journaled filesystems. Be warned that it requires 220% of the unused space on the filesystem. That means if you're using 8 GB of a 10 GB partition, you will need 4.4 GB of free space on a separate partition. Even if you have the space, there's no guarantee your data will still be there. Even if the data still exists, there's no guarantee that you'll be able to find it in the unused 2 GB of random data. Nevertheless, if it is a critical file and you know a few keywords, it may be worthwhile to try.

If you're trying to recover files from an ext2 filesystem, you might want to look at the information available at http://e2undel.sourceforge.net/.

CITES Workstation Services Group - The CITES Workstation Services Group (WSG) specializes in Unix general support, contract system administration, and hourly Unix support (including Unix security).

WSG Unix Security Guides - WSG's guide to Unix security best practices, both general and OS specific.

CITES Help Desk - The CITES Help Desk staff is available to answer Windows and Mac OS questions and can help direct you to the appropriate resources if they are meet your needs.

CITES Security Group - The CITES Security Group is responsible for enforcing the campus Information Technology Policy. They also maintain information on SPAM, viruses, denial of service attacks, and rootkits.