Summertime potluck

This mixed bag of goodies includes additional tidbits on sendmail and chroot

Summary
In this month's column, Carole ties up some loose ends left over from previous columns on sendmail and chroot (padded cells). The checkcompat() routine is discussed, as well as a method for defeating a possible exploit against chroot. As an added bonus, some review of the Black Hat briefings and Defcon. (3,000 words)

This month's installment is in the spirit of that traditional part of the dog days of summer, the potluck supper. I'm using this month's column to tie up some loose ends with sendmail and chroot, based on reader feedback. Also, I regret that I could not attend the Black Hat Briefings and Defcon; to get the scoop, I asked some technical friends for feedback to decide if I should go next year. There has already been a tremendous amount of press coverage, but reporters usually cannot evaluate the technical value of a talk.

Sendmail leftovers

checkcompat()

Logging sender/recipient/subject
A reader sent me mail asking how to have sendmail log the subject of each message, as well as the sender and recipient. I failed to come up with a clean conventional solution (doesn't mean there isn't one -- just that I didn't find it), but discovered that it could be done very effectively using checkcompat():

#ifdef FIREWALL
#define LOOP_CHECK "X-Loop-Check"
#endif
int
checkcompat(to, e)
        register ADDRESS *to;
        register ENVELOPE *e;
{
#ifdef FIREWALL
        int cnt;
        HDR *h;
#endif
# ifdef lint
        if (to == NULL)
                to++;
# endif /* lint */

        if (tTd(49, 1))
                printf("checkcompat(to=%s, from=%s)\n",
                        to->q_paddr, e->e_from.q_paddr);

        /* this is a locally submitted message  ? */

#ifdef FIREWALL

        if (hvalue(LOOP_CHECK,e->e_header) != NULL) {
                /* Been here, done this */
                return(EX_OK);
        }

        /* Add a loop back check header */

        addheader(LOOP_CHECK, "", &e->e_header);

        for (cnt = 0, h = e->e_header; h != NULL; h = h->h_link) {

                /* If not a subject, then skip it */
                if (strcasecmp(h->h_field,"subject") != 0)
                        continue;

                if (h->h_value)
                        sm_syslog(LOG_INFO, e->e_id,
                                "to=%s,from=%s,Subject: '%s'",
                                to->q_paddr, 
e->e_from.q_paddr,h->h_value);
        }
#endif

        return (EX_OK);
}

Below is sample output from the log file that shows the subject as "New Test." You can take this as input to a script, which can then render the data into a more readable form:

Jul 18 14:02:02 rahl sendmail[1990]: OAA01990: from=klein, size=56, class=0,
pri=30056, nrcpts=1, msgid=<199907181802.OAA01990@rahl.wkeys.com>,
relay=root@localhost
Jul 18 14:02:02 rahl sendmail[1992]: OAA01990: to=klein,from=klein,Subject:
'New Test'
Jul 18 14:02:02 rahl sendmail[1992]: OAA01990: to=klein, ctladdr=klein
(1002/10), delay=00:00:00, xdelay=00:00:00, mailer=local, stat=Sent

Inserting special text
The managers of many sites want the ability to insert text that makes some statement about the site into every mail header. It can be done using checkcompat(), provided that you don't need to change your message very often. Why would you want to put text in your mail header at all? Well, you may want some sort of disclaimer statement, or some other legalese. Or, you may want to advertise your Web page. It shouldn't be very lengthy, and it should be something that is appropriate for all users (including the CEO).

#ifdef FIREWALL
char *copyright[] = {
"This material is property of the XYZ Corporation. Any misuse or",
"unauthorized redistribution of this e-mail is a violation of",
"state and/or federal laws. This notice must appear in all e-mails",
"and may not be removed without prior written consent of XYZ",
"Corporation"
};
int size=(sizeof(copyright)/sizeof(char *));
#define LOOP_CHECK "X-Loop-Check"
#endif

int
checkcompat(to, e)
        register ADDRESS *to;
        register ENVELOPE *e;
{
#ifdef FIREWALL
        int cnt;
        HDR *h;
#endif
# ifdef lint
        if (to == NULL)
                to++;
# endif /* lint */

        if (tTd(49, 1))
                printf("checkcompat(to=%s, from=%s)\n",
                        to->q_paddr, e->e_from.q_paddr);

        /* this is a locally submitted message  ? */

#ifdef FIREWALL

        if (hvalue(LOOP_CHECK,e->e_header) != NULL) {
                /* Been here, done this */
                return(EX_OK);
        }

        /* Add a loop back check header */

        addheader(LOOP_CHECK, "", &e->e_header);
        for (cnt = size - 1; cnt >= 0; cnt--)
                addheader("X-Ownership",copyright[cnt],&e->e_header);

#endif

        return (EX_OK);
}

This is what the mail message would look like:

>From klein Thu Jul 22 21:39 EDT 1999
Received: (from root@localhost)
        by rahl.wkeys.com (8.9.3/8.9.3) id VAA06567
        for klein; Thu, 22 Jul 1999 21:39:16 -0400 (EDT)
Date: Thu, 22 Jul 1999 21:39:16 -0400 (EDT)
From: Jonathan Klein <klein>
Message-Id: <199907230139.VAA06567@rahl.wkeys.com>
To: klein
Subject: Test Messages
X-Ownership: This material is property of the XYZ Corporation. Any misuse or
X-Ownership: unauthorized redistribution of this e-mail is a violation of
X-Ownership: state and/or federal laws. This notice must appear in all
e-mails
X-Ownership: and may not be removed without prior written consent of XYZ
X-Ownership: Corporation
Content-Type: text
Content-Length: 5
Status: RO

Test

Padded cell leftovers

In the the Simon Burr example above, the program opens a directory to the current directory, chroot's into a directory called temp, and then uses fchdir and chdir to follow the parent path back up the line to get to the real root, thus breaking the padded cell.

In an effort to defeat this program, my partner, Jon Klein, came up with a method that keeps chroot from having a valid link to the real system. The way this is done is a bit of a hack -- essentially you are creating a broken filesystem on top of the filesystems that contain your padded cells. This filesystem contains a loop; as a result, it must be remade on every reboot, because fsck will not be happy with the corrupted parent loop and will try to fix it. The script to recreate the top-level filesystem follows. It is recommended that the chroot script be run after only all the mounts are complete. In my implementation, I added the chroot script to S01MOUNTFSYS in /etc/rc2.d after the mountall:

...code deleted
# Added by JIK. Do not erase. This is needed for the padded cells

/sbin/chroot-help /dev/rdsk/c0t1d0s5 /dev/dsk/c0t1d0s5 /var/cells
/sbin/mount /dev/dsk/c0t1d0s5 /var/cells

/sbin/mount /var/cells/cell_top/sendmail
/sbin/mount /var/cells/cell_top/extra
/sbin/mount /var/cells/cell_top/extra2

# nothing changed below..just left in a place reference -CF

# The following counts the number of filesystems with quotas enabled
NQUOTA=`cut -f 4 /etc/mnttab | egrep -c "^quota|,quota"`

if [ $NQUOTA -gt 0 ]
then
        echo "Checking UFS quotas: \c"
        /usr/sbin/quotacheck -a -p
        echo "done."
        /usr/sbin/quotaon -a
fi

I put a commented line in /etc/vfstab to keep a marker for the mini-filesystem I created, and allowed the mounts for the filesystems that rely on the broken filesystem (the padded cell filesystems) to fail. After the chroot script, make sure you add mounts for the padded cell file systems so that they get online. Before you use the script, make a small filesystem (about one megabyte will do). The script creates a new filesystem and mounts it. Then, it makes directories cell_top, cell_top/tmp, and cell_top/tmp/loser, and whatever other filesystems you want under cell_top, in this case sendmail extra and extra2.

Edit the script to create whatever other directories you need. The most important thing is to create cell_top and cell_top/tmp/loser, and to put all the directories for the padded cells under cell_top. In addition, because the cell_top filesystem is recreated each time the system is booted, you should not put anything in this filesystem that you want to survive a reboot. As separate filesystems, the padded cell filesystems retain their data and are mounted after the script is run.

After the directories are created, we locate the inode numbers for both cell_top and cell_top/tmp/loser. Now comes the fun part. The script uses fsdb (filesystem debugger -- a favorite of old timers) to edit the filesystem and do the following:

• Decrement the link counter for the top level directory (inode 2); this drops the ".." reference in cell_top for this directory

• Increment the counter in cell_top/tmp/loser; this will allow for a new reference for ".." in the cell_top directory

• Go back to the head of the filesystem, cd into cell_top and assign ".." to be the directory cell_top/tmp/loser

At this point, mount all the other filesystems that are to go under cell_top. Essentially, what you've done is create a parent loop between cell_top and cell_top/tmp/loser. When you try to run the breakout program, it will go into an infinite loop. You should also try to avoid passing through the parent directory in cell_top, as it causes things like sh and pwd to act oddly. Use the absolute path if you need to pass up through cell_top. You can traverse downward through cell_top with no adverse problems.

Lastly, here is the chroot filesystem script:

!/sbin/sh
# Create a broken filesystem to prevent chroot breakout
#
if [ "$1" = "" -o "$2" = ""  -o "$3" = "" ]
then
        echo "Usage: chroot-help <char device> <block device> 
<mountpoint>"
        exit 1
fi

# Make sure it is unmounted

/sbin/umount $2

# Make file system
/usr/sbin/newfs $2 <<END
y
END

if [ $? -ne 0 ]
then
        echo "Could not make file system"
        exit 2
fi

# Mount file system

/usr/sbin/mount $2 $3
if [ $? -ne 0 ]
then
        echo "Cannot mount file system"
        exit 3
fi
cd $3

# Make top level directories

/bin/mkdir -p cell_top/tmp/loser
/bin/mkdir -p cell_top/sendmail
/bin/mkdir -p cell_top/extra
/bin/mkdir -p cell_top/extra2

# Get inode numbers

top=`ls -id . | sed -e 's/^ *//' | cut -f1 -d" "`
c=`ls -id top/tmp/loser | sed -e 's/^ *//' | cut -f1 -d" "`
echo $top $parent $c

# Unmount directories

cd /
/usr/sbin/umount $2

# Create corrupted directory parent
/usr/sbin/fsdb -o w $1 << END
2:inode
:ln=-1
0t$c:inode
:ln=+1
2:inode
:cd cell_top
1:dir=0t$c
:quit
END

# We are done

Mail regarding chroot
I ran this method past Simon to see what he thought of it. His comments follow:

"The method to defeat the break out is interesting and quite nice. However, I really doubt that I'd be allowed to install it on a production server. Sysadmins tend to get a little uneasy when people start to mess with the filesystems at that level. (They still shudder when they remember the fun they had when a system changed ".." to ".," (note the comma) on a certain directory for some strange unknown reason occasionally.) Their argument would be that the risk of someone getting root within a chroot() jail is less than the risk of something going wrong with the filesystem hack. In a well-designed system, this should be the case. Unfortunately, wu-ftpd is not well designed (the internal structure and code is absolutely horrendous -- I'm amazed that there haven't been more problems with it). There is another problem. If the attacker has root access, then there is nothing to stop him or her from doing an lstat(), using mknod() to create a device entry, and then looking at the device directly to fix the problem. It does require a high level of knowledge to do this, however."

My response to Simon's comments are this:

I agree that hacking the filesystem inode is tricky and potentially dangerous, if you don't know what you're doing. While I have used it in production, it was only on systems that I would be administering. If I implemented a solution that was to be turned over to another group for administration, I would not do this. Simon makes a very good point that the dangers far outweigh the advantages in this case. Still, I hope that demonstrating how it can be done sparks some interest in learning more about how the internals of the filesystem work. Use with caution!

Road trips

Black Hat

Organization
The hotel was impressive, but just barely finished. Environmental alarms were constantly going off in the main conference room during talks. There were a couple of changes announced at the opening; otherwise, things went pretty smoothly. The most common complaint was poor scheduling: often two or more intriguing talks would be happening at once, while at other times there was nothing of interest going on.

Content of talks

The general consensus was that the talks were rather light on technical content. Most of the talks were targeted for law enforcement and upper management. Those that did have a technology focus were on an introductory level -- nothing new. The most notable session was Jennifer Grannick's talk on computer crime law. I specifically asked one person for a comparison to the SANS Network Security Conference (NetSec), which was held last October in Orlando. He felt the NetSec conference was more technical and more polished. It was also only half of Black Hat's price. (Disclaimer: I don't work for SANS and, in fact, did not care for the NetSec conference.)

Overall

The general impression I got was that Black Hat was interesting to attend and a good way to make contacts with people -- as long as someone else was paying for it.

Defcon

Organization

I heard overwhelming complaints about the facilities and lack of organization. Some specific points made were:

• There were supposed to be three tracks, but only two rooms were available.

• Some talks were canceled at the last minute, and it was very difficult to get information about rescheduled talks. One talk ended up at the pool area two hours after it was originally scheduled.

• There were some grumblings at the BO2K event when the first few rows were asked to relinquish their seats for the press. These should have been reserved in advance.

• The organizers did not seem prepared to handle a crowd of 3,000 attendees.

Content of talks

No one I spoke to seemed to be too happy with the quality of the talks presented, either. Overall, the technical content was considered light. Some specific points made were:

• A few talks were duplicated from Black Hat.

• A lock-picking demonstration did not seem to be very useful for someone interested in computer security. Ditto for a discussion on military weaponry. What would have been interesting would be if the speaker divulged how the information was obtained, rather than the information itself.

The most useful presentation (some said the only reason for going) was the Cult of the Dead Cow (cDc) Back Orifice 2000 launch, which had all the restraint of a rock concert. A few dozen advance copies were thrown out to the crowd. Unfortunately, these were accidentally infected with the CIH virus. While I think it's commendable that cDc members eventually accepted responsibility, perhaps more attention should have been paid to quality control and less on theatrics (my opinion).

Overall, the motivation for attending either conference seemed to be the ability to network with people. It's a great opportunity to meet government agents, law enforcement, security professionals, hackers and, of course, the media. Who knows? Maybe your picture will show up on the news. I'm still not sure if I really want to spend my summer vacation in the desert. Why don't they do these things at the beach?

Acknowledgements

Disclaimer: The information and software in this article are provided as-is and should be used with caution. Each environment is unique and the reader is cautioned to investigate with his or her company as to the feasibility of using the information and software in the article. No warranties, implied or actual, are granted for any use of the information and software in this article and neither author nor publisher is responsible for any damages, either consequential or incidental, with respect to use of the information and software contained herein.