Hardening RedHat Linux with Bastille

Securely installing a bastion host

By Seán Boran

This article presents a concise step-by-step approach to securely installing RedHat Linux for use in a firewall DMZ, or other sensitive environment, using Bastille. Linux has progressed rapidly and can be configured to be as secure as, if not better, than commercial UNIX.

The focus in this article is on RedHat 6.2 on SPARC + x86, RedHat7 on x86 and Mandrake 7.0 on x86.

DRAFT: Includes bastille 1.1.1

We welcome your feedback on this article.


Table of Contents:

  1. Preparation
  2. Initial OS installation
  3. Install SSH
  4. Bastille: Introduction, Running, Checking, Problems.
  5. Installing tools & sysadmin software
  6. Patches and Logging
  7. Integrity Checking and backup
  8. Install, test, harden applications.
  9. Going Live

Regular maintenance
Additional Notes
References
Changes to this article


1. Preparation


2. Initial OS installation

On x86 hardware, screen, keyboard and mouse are needed. Boot from CDROM or the boot floppy and choose install.

On SPARC hardware, the entire installation can be done without screen or keyboard (also called a "headless server"). Connect the serial console, switch on, halt to the OK prompt by sending a Stop-A (~#, ~%b, or F5 depending on whether you use tip, cu or a vt100 terminal), then start the installation procedure:  boot cdrom - install.

The RedHat install is pretty buggy and options differ between releases:

The Mandrake 7.0 install does not suffer from these shortcomings.

So, choose a server or custom install, set hostname, IP parameters, timezone, etc. Don't enable any naming services like NIS or NFS. Choose manual disk partitioning:

Set a strong password (At least 8 chars with numbers, letters and punctuation) for root. Create an additional test user, as you won't be able to login over the network as root.

The "init level" should be set to 3 (command line login), rather than 5 (graphical login). If a GUI is needed, it can always be started manually with startx.

To login via the 'serial port A' on x86 Hardware, which is useful for troubleshooting, installations and getting to know the command line (it is not necessary for headless SPARCs which do this automatically). Add the following to /etc/inittab.
con:23:respawn:/sbin/getty ttyS0 VC

To allow root to login via this serial port, add ttyS0 to /etc/securetty,
echo "ttyS0" >> /etc/securetty



3. Install SSH

Install SSH, the secure Shell for login access (Bastille can do this, but you need Internet access, I had problems with SPARC, and prefer to know exactly what options are used to compile SSH). SSH is already included in some distributions, such as Redhat7.

There are two key implementations for Linux 'ssh1' and 'OpenSSH', here we use ssh1 as an example. OpenSSH is more interesting in some ways, but ssh1 also supports securid (which is useful to me). See also [7] for a detailed discussion of SSH and it's various implementations.

Either download sources or RPMs (see sites listed under [7]):

1) Sources: 
zcat ssh-1.2.30.tar.gz | tar xf - 
cd ssh-1.2.30; ./configure --prefix=/usr --without-none --without-rsh --without-idea
make && make install

2) RPMs (SPARC example shown):
rpm -i ssh-1.2.30-7i.sparc.rpm        ssh-clients-1.2.27-7i.sparc.rpm
rpm -i ssh-extras-1.2.30-7i.sparc.rpm ssh-server-1.2.27-7i.sparc.rpm

Copy a startup file (example sshd) to /etc/rc.d/init.d/sshd and setup links, unless it was done as part of the previous step.
chkconfig --add sshd

Configure an appropriate /etc/sshd_config file (see also [7]), so that access is restricted to named hosts with known public keys (/etc/ssh_known_hosts) and rhosts authentication is disabled. Avoid trusts. Only allow specific users and hosts to access SSH.

Deny daemon accounts access, for example:
DenyUsers daemon bin sync adm lp shutdown halt mail news uucp nobody operator sympa, squid, postgres, gopher, postfix, xfs.



4. Bastille

Introduction

Bastille is set of open source scripts designed to harden a virgin Red Hat 6.0 or 6.1 installation (6.2 support is planned soon). The first release was in December 1999 and significant progress has been made. V1.1 (released June 2000) was used here. What does Bastille do? Unneeded daemons are stopped, logging enabled/improved, SUID and file permissions tightened, account security improved and even a chroot environment is provided for DNS servers.
Note that Bastille does not run on other Linux variants such as SuSE (which have their own mechanisms and different startup files).

An automated and interactive interface is available. The text based menus provided by the interactive install are very useful for explaining the different options involved and generate a configuration file (tui-generated-raw-config) which is used in the next step by the Bastille back end to do the actual hardening. The configuration file can be edited or copied to other machines to speed up hardening.

Running Bastille
  1. Download Bastille and extract into /root.
  2. Shutdown the network interface during this next phase, just in case (the interface name may vary):
  3. ifconfig eth0 down

  4. Run the Bastille interactive script:
    cd /root/Bastille; ./InteractiveBastille.pl

    Note: On SPARC the Interactive script won't work:
    Can't load './Curses.so' for module Curses: ./Curses.so: ELF file data encoding not big-endian at /usr/lib/perl5/5.00503/sparc-linux/DynaLoader.pm line 169.
    Fix: Download Curses-1.02.tar.gz from CPAN and install:
    perl Makefile.PL; make && make install
    Then change to the Bastille dir & remove the i386 Curses library:
    cd /root/run-Bastille; mv Curses.* /tmp;
    And run InteractiveBastille.pl again.

    InteractiveBastille.pl runs through hardening setup on a step-by-step basis, asking the user what should or should not be tightened down. Unneeded daemons are stopped, logging enabled, SUID and file permissions tightened, account security improved and even a chroot environment is provided for DNS servers.
    Default answers except for the following were used (see tui-generated-raw-config):

  5. Start the actual hardening process:
    ./BackEnd.pl < config > screen.log

Bug: the first time BackEnd fails with the following message:
# ./BackEnd.pl < config > screen.log
/bin/cp: /etc/banners: omitting directory

Run it again and it works fine!

Checking the results
  1. Review the log of the hardening process: /root/Bastille/screen.log and /root/bastille-action-log
  2. Reboot.
  3. Login as root and check the process list, it should be something like:

    tests# ps -ef
    UID PID PPID C STIME TTY TIME CMD
    root 1 0 10 17:07 ? 00:00:03 init [3]
    root 2 1 0 17:07 ? 00:00:00 [kflushd]
    root 3 1 0 17:07 ? 00:00:00 [kupdate]
    root 4 1 0 17:07 ? 00:00:00 [kpiod]
    root 5 1 0 17:07 ? 00:00:00 [kswapd]
    root 6 1 0 17:08 ? 00:00:00 [mdrecoveryd]
    root 276 1 1 17:08 ? 00:00:00 syslogd -m 0 -a /home/dns/dev/lo
    root 286 1 0 17:08 ? 00:00:00 klogd
    root 301 1 0 17:08 ? 00:00:00 crond
    root 310 1 3 17:08 ? 00:00:00 /usr/sbin/sshd
    root 369 1 0 17:08 ttyS0 00:00:00 login -- root
    root 370 1 0 17:08 tty1 00:00:00 /sbin/mingetty tty1
    root 371 1 0 17:08 tty2 00:00:00 /sbin/mingetty tty2
    root 372 1 0 17:08 tty3 00:00:00 /sbin/mingetty tty3
    root 373 1 0 17:08 tty4 00:00:00 /sbin/mingetty tty4
    root 374 1 0 17:08 tty5 00:00:00 /sbin/mingetty tty5
    root 375 1 0 17:08 tty6 00:00:00 /sbin/mingetty tty6
    root 378 369 1 17:08 ttyS0 00:00:00 -bash
    root 390 378 0 17:08 ttyS0 00:00:00 ps -ef

  4. Check the network connections, for example:

# netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
raw 0 0 *:icmp *:* 7
raw 0 0 *:tcp *:* 7
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 3 [ ] DGRAM 239 /dev/log
unix 0 [ ] STREAM CONNECTED 108 @0000000f
unix 0 [ ] DGRAM 241 /home/dns/dev/log
unix 0 [ ] DGRAM 353
unix 0 [ ] DGRAM 282
unix 0 [ ] DGRAM 254

Bastille Problems/limitations

The Bastille Linux hardening script is a community consensus project: it attempts to integrate existing "best practices" documents and the shared knowledge of many administrators.
The Bastille team are to be congratulated on their good work, but a few problems have been noticed. It has been indicated that the following problems should be addressed in future releases, together with support for RH6.2.



5. Installing tools & sysadmin software

At this stage standard tools/utilities are going to be installed. These tools should already have been compiled and tested extensively on another machine. They are typically transferred as tar files, by CD or FTP.  Some of the following settings can also be configured via linuxconf.


6. Patches and Logging

Patches

Patch management is still very basic in Linux, there is no unique numbering, or dedicated tools for automated checking of patch levels (like Solaris for example), and no "patch bundles" to enable you to quickly get you system unto a current level.

Redhat 6.1, patches can be found at www.redhat.com/errata. Security advisories, bug fixes and package updates are on offer. Mandrake updates are at www.linux-mandrake.com/en/fupdates.php3, but Mandrake also offers a nice Patch checking and installation tool under KDE, that is worth trying out.
Only select patches that are relevant to tools/applications actually in use on your system.
Once you have downloaded the relevant patches apply them:

rpm - Uvh filename.rpm

There are also specific kernel patches to add security features such as strong crypto to the kernel, see [9].

Configure logging:

Syslog logging: The additional syslog configuration setup by Bastille is useful, but only on hosts with local logging or loghosts (syslog servers).

Syslog "clients": It is suggested that most hosts log to central loghost, with an /etc/syslog.conf such as the following:

# syslog.conf for clients
#
# send all messages to "loghost":
*.* @loghost
*.emerg *
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# Log all logins to /var/log/loginlog
auth.*;user.*;daemon.none /var/log/loginlog

# Log additional data to the Alt-F7 and Alt-F8 screens (TTY 7 and 8)
*.info;mail.none;authpriv.none /dev/tty7
authpriv.* /dev/tty7
*.warn;*.err /dev/tty7
kern.* /dev/tty7
mail.* /dev/tty8
###########

Syslog "loghost"

Give the loghost a whopping great /var disk for logs.
Use the default syslog.conf, with the additions from Bastille.
Set up log pruning as described below.

Root cron entries:

## Synchronise the time:
0,15,30,45 * * * * /usr/bin/rdate timehost1 >/dev/null 2>&1

# process the mail queue hourly during the week:
0 * * * 1-5 /usr/lib/sendmail -q

File permissions: tighten more permissions, and restrict certain tools to root or disable:

chmod u-s /usr/sbin/sendmail       #Not for mailgateways or multi-user hosts
chmod 400 /.shosts
chmod 444 /etc/sshd_config /etc/ssh_known_hosts

Documentation:

Document configuration changes in a text file such as /etc/mods, update after each change, with date, author, files affected, description.
cat > /etc/mods <<EOF
19.12.00  sb  New install of RH6.1+Bastille & tools according to hardening guidelines
EOF

Banners.

Change login banners to warn users about unauthorised access - you'll need this if you want to prosecute intruders. Edit /etc/issue (for pre-login) and /etc/motd (for post-login). Bastille does not change /etc/issue, one might might want to remove indications about the Operating System type and add a banner (especially when the console is not in a secured room).

Reboot.

Test - Do SSH and the standard tools work? Check log entries, check console messages. Does the system behave as expected?



7.Integrity Checking and backup

At this stage, we need to install a file integrity checker that uses secure hashing algorithms, initialise it's database and run regular checks to monitor for changes. If possible keep the master database on another machine or offline or on write-once media. Even if you can't run regular checks, take a master copy and store in on a floppy, it will be a great help in detecting what has changed, should the system be penetrated at a later stage.

What options do we have for integrity checking?

An example using the free Tripwire Version 1.2:


8. Install, test, harden applications

Depending on the function of the server, applications such as ftpd, BIND, proxies, etc. are installed at this point.

Hardening of specific applications like ftp, DNS, Email and also general application tips are discussed in a separate document [16].


9. Going Live


Preparing to go live

Consider installing a script to check that important daemons are running. Install monitor_processes.pl and add a root cron entry:
## Check that important processes are running during office hours:
## [If you run 7x24, modify accordingly]
0,30 8-19 * * 1-5 /secure/monitor_processes.pl inetd sshd httpd

If data partitions had to be mounted read-write during the application install/testing, consider mounting them read-only now.

Reinitialise tripwire (or equivalent integrity checker).

Backup the system, again to two tapes, one offsite.

Run a network scan on the system, to ensure that only expected services are visible. A commercial tool such as ISS or a free one like Nessus, nmap or Satan should do the job. Print out the results and archive.

If possible, have additional people do the final testing, just in case something was forgotten. Test in detail - What works? What is forbidden? Check console/log entries. Does the system behave as expected? Watch the logs very frequently during the first few days of production.

Going Live

Test in detail. Check log entries. Does the system behave as expected?

Have applications been tested in detail, by different people with different points of view, from different access points on the network?



Regular maintenance

The following activities should take place hourly, daily, weekly or monthly, depending on how critical the system is:


Additional Notes

This article has been very specific, in the interest of making it practical. However, each security administrator has his own methods and each site has different requirements.



References

[0] www.RedHat.com   www.mandrake.com
Linux on Sun SPARC: www.ultralinux.org   www.sparclinux.com 
[1] sourceforge.net/projects/bastille-linux   www.Bastille-Linux.org
[2] Mandrake User and Reference Manual (also in /doc on the Mandrake CD)
www.linux-mandrake.com/userguide/en/reference/000.html
[3] Scripts/configs included with this article: monitor_socket.pl, monitor_processes.pl, trip_linux.sh, trip_host.shlogcheck.sh, tui-generated-raw-config, sshd, msec.txt.

Saveit  script:
Original spanish version: saveit-sp.sh, my tweaked version: saveit (less verbose messages, fix for OpenBSD, english translations, if target exists save with a time postfix).
[4] Klaxon and tocsin www.eng.auburn.edu/users/doug/second.html
[5] Tripwire:
Free version V1.2 www.cert.org/ftp/tools/tripwire (last updated in 1994). It was difficult to find rpms, but I eventually found them at www.lj.net/~jht/rpms
Commercial Version www.tripwiresecurity.com (starts at $495.-/server) also runs on NT.
OpenSource Version www.tripwire.org Planned for late 2000, on Linux only.
Sunworld article.
[6]  Sample tools for analysing logs:
Logcheck www.psionic.com/abacus/logcheck (see also my improved version of logcheck.sh)
Swatch  ftp.stanford.edu/general/security-tools/swatch
[7] All About SSH PartI and Part II, an article written for SecurityPortal by the author.
All about SSH - PartI securityportal.com/direct.cgi?/research/ssh-part1.html,
All about SSH - Part II securityportal.com/direct.cgi?/research/ssh-part2.html  
Downloading SSH Sources:
SSH1 sources: ssh-1.2.30.tar.gz from ftp.cs.hut.fi/pub/ssh
OpenSSH sources: www.OpenSSH.com

Downloading rpm binaries:
SSH1 for RH SPARC ftp.zedz.net/pub/crypto/linux/redhat/sparc
OpenSSH and SSH1 for RH x86: ftp.zedz.net/pub/crypto/linux/redhat/i386
OpenSSH for Mandrake x86:  ftp.zedz.net/pub/crypto/linux/mandrake/7.0
   ftp.sunet.se/pub/Linux/distributions/mandrake-crypto
Also:
  ftp.cryptoarchive.net
  www.cryptoarchive.net/ftp.cryptoarchive.net/SSH/OpenSSH
[8] AIDE, a GPL file integrity checker. www.cs.tut.fi/~rammer/aide.html
[9] SecurityPortal's Linux Security Knowledge Base
  1. File system considerations in Linux kben10000036.html
  2. XNTP - accurate time synchronization for Linux kben10000029.html
  3. Linux kernel security patches kben10000021.html
  4. Securing the Linux console kben10000016.html
  5. Securing the LILO bootloader kben10000002.html
  6. Secure administrative access tools for Linux kben10000011.html, kben10000012, kben10000013, kben10000014.
  7. Limiting user access to cron kben10000014.
  8. Article Index www.securityportal.com/lskb/articles
[10] New reports on Bastille or Linux Hardening since April'00:
- Bastille Walkthrough, by Jay Beale [STALE LINKS]
- Building a Secure Gateway System, By Chris Stoddard
- Shredding Access in the Name of Security: Set UID Audits, by Jay Beale [STALE LINKS]
- Why Do I Have to Tighten Security on My System?, by Jay Beale [STALE LINKS]
-
How Do I Tighten Security on My System?, by Jay Beale [STALE LINKS]
[11] Patching RedHat, tools: rh-errata
[16] Hardening Applications
[17] syslog replacements:
syslog-ng www.balabit.hu/products/syslog-ng (tcp connections, content filtering, encryption, authentication)
secure syslog  www.core-sdi.com/english/slogging/ssyslog.html
Nsyslogd coombs.anu.edu.au/~avalon/nsyslog.html (tcp connections & SSL)
[18] Security Advisories: www.cert.org, www.first.orgwww.ciac.org
CERT provide several useful firewall/hardening/intrusion detection papers online www.cert.org/tech_tips.
[19] Security Newsletters: SecurityPortal, SecurityFocus, SANS.


Other links:


Changes to this article

25.Apr.'00  Improve tripwire, ssh, references, Mandrake. Spelling. Feedback from Kurt+Jay.
29.Apr.'00  Initial Publication
28.Jun.'00  Add links to new reports [10], Correct links[9] and snort.
09.Aug.'00 New: saveit, Update: Bastille 1.1, Links.
19.Dec.'00 Update for RH7

Last Update: 11 December, 2001


Seán Boran is an IT security consultant based in Switzerland and the author of the online IT Security Cookbook.

© Copyright 2001, Sean.Boran, All Rights Reserved

By default after 20 reboot, the filesystems are checked (fsck). If you have several filesystems, it's better to stagger the checking so that all filesystems are not checked at the same time, avoiding a slow "21 reboot".

Each filesystem has a mount count which can be interrogated as follows:
dumpe2fs /dev/hda1 | grep 'Mount count'

Set this to a staggered number between 1 and 19, for example:
tunefs -C 6 /dev/hda1
tunefs -C 12 /dev/hda4
tunefs -C 18 /dev/hda7

Warning: umount filesystems before running tunefs.

The number of reboots between check can also be changed:
tunefs -c 10 [check every 10 reboots]
tunefs -i 5 [check every 5 days]