A FAM-based Intrusion Detection System

For a while now, the Red Hat and Fedora distributions have included a nifty thing called a "fam", or "file alteration monitor" server. Fam was originally developed by SGI, but has now been ported to Linux. Here's a description of what it does, from the man page:
       fam  is a server that tracks changes to the filesystem and
       relays these changes to interested applications. ... In  
       the  absence  of  fam,  these applications  ... are forced 
       to poll the filesystem to detect changes.  fam is more 

       Applications can request  fam  to  monitor  any  files  or
       directories  in  any filesystem.  When fam detects changes
       to monitored files, it notifies the  appropriate  applica­
       tion.   The  FAM  API provides a programmatic interface to
These features provide the basis for a useful intrusion detection system (IDS).

How to Use It
Here's one way to do it. Take a look at the following Perl script. The script uses a package called SGI::FAM that's available from your friendly local CPAN archive. (NOTE: Under Fedora you can use Eric Calder's "cpan2rpm" program to fetch the perl module and pack it up into an easy-to-install rpm.) On a RedHat/Fedora machine, I could have written the program in C without installing anything else, but I've implemented it in Perl here to make it easier to play with.

(NOTE: Since Fedora Core 3, FAM has been replaced by gamin. Gamin is a drop-in replacement for FAM in most respects, but you'll find you need to make one small change to use perl's SGI::FAM module with it. Namely, you'll need to create the file /etc/fam.conf, which can be empty if you like. Without this file SGI::FAM will fail.)


my $notify = 'root';
my $config = '/etc/famids.conf';

open ( CONFIG, "<$config" );
my @lines = grep {!/(^\s*#|^\s*$)/} <CONFIG>;
chomp @lines;
close ( CONFIG );

use SGI::FAM;
my $fam=new SGI::FAM;

use Text::ParseWords;

my %action;
my %dirfiles;
for my $line (@lines) {
    my @data = parse_line( '\s*,\s*', 0, $line );
    chomp @data;
    -e $data[0] && $fam->monitor($data[0]);
    $action{$data[0]} = $data[1];
    if ( -d $data[0] ) {
	my @files = split(/\s*\|\s*/,$data[2]);
	push (@{$dirfiles{$data[0]}},@files);

my $waited = 0;
EVENT: while (1) {
    my $event;
    eval {
	$event = $fam->next_event; # Blocks
    if ($@) {
	if ( $waited ) {
	    die "$0: ERROR: $@\nPossibly fam server died?\n";
	} else {
	    sleep ( 15 ); # Give the server another chance.
	    next EVENT;
    } else {
	$waited = 0;
    unless ( $event->type eq "exist" |
	     $event->type eq "end_exist" ) {
	if ( @{$dirfiles{$fam->which($event)}} ) {
	    my $filename = $event->filename;
	    next EVENT unless grep(/^$filename$/,
	open ( MAIL, "|/bin/mail -s 'famids report' $notify" );
	print MAIL "Possible intrusion:\n";
	print MAIL "Pathname: ", $event->filename,
	"\nEvent: ", $event->type, "\n";
	close ( MAIL );
	if ( $action{$event->filename} ) {
	open ( MAIL, "|/bin/mail -s 'famids action report' $notify" );
	print MAIL "Taking action in response to event for ".$event->filename."\n";
	print MAIL "Action: $action{$event->filename}\n";
	print MAIL `$action{$event->filename}`;
	close ( MAIL );

And here's a sample configuration file:

# Famids configuration file.
# List of files/directories to monitor:

/bin/login, rpm -V util-linux
/bin/ls, rpm -V fileutils
/bin/ps, rpm -V procps
/sbin/ifconfig, rpm -V net-tools

How It Works
The "famids" program requests that the fam server monitor the list of files given in "famids.conf". Once famids has told the server which files its interested in, the program sits back and waits for the fam server to send it an event. Whenever one of the monitored files changes, the fam server sends famids an event containing data that identifies the file and says what happened to it. For example, the event type might be "Changed", indicating that the file size, date or ownership has changed. Other types include "Moved" and "Deleted".

When famids receives such an event, it sends a descriptive e-mail to an address specified at the top of the script. After sending the e-mail, famids goes back to looking for events from the fam server.

Optionally, the file name in famids.conf may be followed by an action to be taken when events involving this file are seen. For example, when /bin/ls changes, you might want to verify the entire rpm package that contains /bin/ls, to look for changes in other associated files. Any output from these actions will be mailed to the same address as the event notifications. The "file,action" pairs are just separated by a comma. The program uses Text::Parsewords to parse the line, so you can enclose odd filenames in quotes if you want to.

Sometimes we need to work around a couple of idiosyncrasies of FAM. First, we might try monitoring a directory (/root for example) instead of a file. If you try this, you'll rapidly find that a LOT of files are created/deleted/changed when someone logs in, because you'll get an e-mail message about each creation/deletion/change every time you log in as root. This gets really annoying really quickly, and even more so if you have automated tasks that log in as root.

Secondly, you might sometimes want to look for the creation of a file that doesn't normally exist. For example, I use tcsh, so I don't normally have a .bash_history file in my home directory. If such a file appears, it's a good indication that something fishy is going on. Unfortunately, FAM won't monitor a file that doesn't initially exist.

To get around these problems, my famids script lets you specify a directory to monitor, and then specify a list of "important" files in that directory for which you'd actually like to receive e-mail reports. The script asks FAM to monitor the directory, then whenever FAM signals that a file has been created/deleted/changed, the script checks the list of important files to see if an e-mail needs to be sent. So, to monitor for the creation of a .bash_history file in /root, I can just monitor /root and ask to only be notified about events involving the file .bash_history. The way to do this is shown in the example famids.conf file above.

Finally, I'll note a couple of other annoyances that could be removed by improving famids. Since Red Hat Linux transmogrified into Fedora, the program "prelink" has been included in the distribution, and (by default) is run automatically every day by cron. Prelink tries to make programs load faster by "pre-linking" them with the shared libraries they use. That's a good thing, but it means that whenever the shared libraries change, prelink will modify the programs that depend on these libraries, and this will trigger a warning e-mail from famids if any of these program files are being monitored. One way to avoid this would be to modify famids so that monitoring is suspended during the prelinking process.

Similarly, program files may be modified by automatic updates. In my case, I have a nightly update script that uses yum and other tools to update each computer. In principle, the famids script could be modified to suspend monitoring during these updates.

What Should You Monitor?
The list of files in the sample famids.conf file above includes the passwd and group files, so you'll be notified if any new accounts are created. It also includes ssh, sshd and login, which are commonly altered (trojaned) by rootkits. Next, it includes several utilities (ls, ps, top, etc.) that are commonly altered to mask an intruder's activities. Finally, the list includes the famids configuration file itself.

Starting famids and Keeping it Running
If an intruder knows that famids is running, he or she can just kill the process. How can you make sure it's restarted? One way is by starting famids from /etc/inittab. For example, add the line:

to the bottom of your inittab and restart init with a "kill -HUP 1". (This assumes that you've put famids in the directory /local/sbin. Alter as appropriate.) The "respawn" causes init to automatically restart famids if the process dies.

What if the intruder modifies /etc/inittab before killing famids? Notice that famids.conf tells famids to monitor /etc/inittab, so an alarm will be set off if anyone changes the file.

What if the intruder modifies famids.conf, so that inittab isn't monitored anymore? Note that famids.conf also monitors famids.conf.

And, before you ask, notice that the famids program itself is also monitored. All of this makes it difficult for an intruder to shut off famids without setting off an alarm.

Other Security Measures
Remember, famids is only one part of a security system. By the time an intruder sets off a famids alarm, he or she has already broken into your computer. Here are some other things you need to do to make your computer secure:

Try out famids. Play around with it. Don't blame me if it doesn't work.