The following is based on my recommendations for setting up Debian 3.0 (woody) servers in a very specific production environment on x86 architecture. This is what works for me here and now (2003), but hopefully it will be useful to others as well. As always, remember that YMMV.
This document is broken up into the following sections:
Think carefully about what hard drive partitions you would like before you install. I typically make the first partition the root (/) partition and use the second partition for swap. I tend to create separate partitions for /usr and /var, as well as a separate partition (or two for RAID-1 mirroring) for /home on user machines. A separate small (100 MB) /tmp partition is also a good thing to have on user machines. The size of the partitions varies from server to server and with the amount of disk available. /usr needs to be large enough to hold binaries, libraries, documentation, and source. /var needs to be large enough to hold logs and dpkg information. I find that I need more disk space in /usr than in /var than in /. Assuming that /home is a separate partition or that there really aren't going to be any files in /home, my absolute minimum recommendation is 1 GB each for /usr and /var and 512 MB for /. Hard drives are huge these days. A 9 GB hard drive (the smallest available) could be partitioned into 4 GB for /usr, 2 GB for /var, 1 GB for /, and 512 MB for swap, leaving 1.5 GB to spare for future expansion or backups or whatever.
Certain services work best with a separate partition. postfix servers should have a small (100 MB) spool partition (/var/spool/postfix). squid proxies should have a separate large (2-8 GB) cache partition (/var/spool/squid). Separate partitions for web pages or databases may also be useful. Use your judgment.
Journaling files systems are good. I use ext3, although xfs looks like it may be promising.
Boot from the Debian CD and use a 2.4 kernel ('bf24' at the boot prompt). In brief, installation consists of hitting Enter a lot. Really. In most cases, you want the default option. Don't worry too much about kernel modules except the ethernet card. Say yes to both password questions. For more information, see Debian's Installation Notes and/or read on for the very brief version.
This will likely only make sense if you have an installation in front of you: en (english), US, qwerty/us; partition the hd; activate swap partitions; initialize linux partions; set the file systems (journaling); reboot (so the partitions know they exist); mount the partitions; install kernel driver modules (make sure to select the ethernet card from kernel/drivers/net/); configure network hostname, interface eth0, and set to no dhcp; install the base system; make bootable from the hd and create a boot floppy; reboot; set the clock to GMT, US, Eastern; yes to MD5 and shadow passwords; set the root password and create a user account; remove pcmcia and ppp; run dselect to install packages.
Select individual packages during the installation. I attempt to install as few packages as possible on servers. None of the servers have X or NFS. Avoiding printing, telnetd, ftpd, fingerd, nfs-common, nfs-server, portmap, and other security-risk services unless they are necessary. In addition to the default packages, I regularly find the following useful: acct, bzip2, cracklib-runtime, cracklib2, fping, grub, grub-doc, libncurses5-dev, links-ssl, lynx-ssl, nmap, ntpdate, slay, smartsuite, super, syslog-ng, traceroute, and wget.
For copy and paste purposes:
apt-get remove --purge telnetd ftpd fingerd nfs-common nfs-server portmap apt-get install acct bzip2 cracklib-runtime cracklib2 dlocate fping grub apt-get install grub-doc libncurses5-dev links-ssl lynx-ssl nmap ntp ntpdate apt-get install ntp-simple slay smartsuite super syslog-ng traceroute wget
The most useful packages are documentation packages. man, info, apropos, and /usr/share/doc/ are your friends.
Always be thinking about security: secure passwords, encryption, running few services, tcpwrappers, logging, etc. You did say yes to shadow and MD5 passwords, didn't you?
Turn off unneeded services. You can comment out everything in /etc/inetd.conf, or you may wish to leave identd running on machines with users. (If you don't like modifying files, you can also run 'update-inetd --disable servicename' for each service.) To read in the new configuration, run '/etc/init.d/inetd reload'. See what processes are still running with 'lsof -i', 'nmap localhost', or 'netstat -ln'.
Startup scripts for each runlevel, including S for startup, are linked to from /etc/rc*.d/. S is for start, K is for kill, and the order is defined by the numbers. The actual scripts are in /etc/init.d/. To start and stop services manually, look at these scripts. The syntax is usually '/etc/init.d/servicename start' and '/etc/init.d/servicename stop'. The command 'update-rc.d -f servicename remove' can be used to remove /etc/rc*.d/ scripts for each service. You may have to kill some proccesses manually with the 'kill' command (such as portmap).
You'll want to recompile a kernel after setting up a new machine. Grab kernel source from a mirror. bunzip and untar it in /usr/src. cd to /usr/src/linux/ (or /usr/src/linux-*) and read the README. Use 'make menuconfig' to configure the kernel.
The configuration guidelines that follow are intended for the 2.4.x kernel series. Most likely, the following categories should have everything disabled: Code maturity level options, Memory Technology Devices, Parallel port support, Plug and Play configuration, Multi-device support, Telephony Support, I20 device support, Amateur Radio support, IrDA support, ISDN subsystem, Old CD-ROM drivers, Input core support, Multimedia devices, Sound, USB support (unless you are using USB), and Kernel hacking.
General: Some people like modules, and others don't. I try to compile my kernels with exactly what they need, but I leave loadable module support enabled just in case (maybe a bad idea given recent exploits). Select the appropriate processor family, /dev/cpu/*/msr, MTRR, and SMP (for multi processor systems). Under general setup, hot-pluggable devices are usually not needed. BSD Process Accounting is good. Select APM for hardware that supports it; otherwise use APCI. Under block devices, select loopback. On software RAID systems, select Multiple devices, RAID support, and RAID-1 from the RAID options.
Network: Under networking options, turn off multicast and select syncookies. Under network device support, select dummy net, 10/100 ethernet, and the appropriate ethernet card (3com -> 3c590 Vortex/Boomerang or EtherExpressPro/100). Turn off EISDA, VLB, and PCI.
HD: If you are sure there is no IDE, turn off ATA/IDE/MFM/RLL support. Otherwise, turn off everything except IDE/ATAPI CDROM and AMD Viper support (for the rackmount servers). Under SCSI, say no to enable extra checks, probe all LUNs, verbose reporting, and drugs (j/k about the last one). Select SCSI lowlevel and the driver for the SCSI card (such as Adaptec AIC7xxx or Adaptec I2O RAID support). Select tape and cdrom options when appropriate. Run 'lspci -v' do determine the IDE chipsset of the motherboard.
Other: Under character devices, select enhanced real time clock on multi-processor systems and unselect /dev/agpart and direct rendering manager. Under file system, select quota support on user systems, unselect automounter and virtual memory file system, as well as the options under Network file systems. Also select iptables and masquerading as needed.
After you have configured your kernel and saved the config with a reasonable filename, run 'make clean; make dep; make bzImage'. For kernels with modules, run 'make modules; make modules_install'. Backup your old kernel. Copy /usr/src/linux/arch/i386/boot/bzImage to boot/vmlinuz-version; copy /usr/src/linux/System.map to /boot as well. Make a boot disk for your new or old kernel if you want. Use 'mkboot' which defaults to /vmlinuz or specify the location of your kernel as an option.
Instructions for two boot loaders, lilo and grub, follow; pick one or the other. For lilo, use ln -s to link the new and old versions of your kernel to /vmlinuz and /vmlinuz.old respectively, and modify /etc/lilo.conf as follows:
prompt
delay=30
timeout=30
vga=normal
image=/vmlinuz
label=Linux
read-only
image=/vmlinuz.old
label=LinuxOLD
read-only
optional
Important: You must then run 'lilo' before rebooting with 'reboot'.
For grub, run 'grub-install /dev/sda' (assuming this is the right drive) to create the /boot/grub/ directory. This should contain device.map containing the line "(hd0) /dev/sda". Run 'grub-install '(hd0)'' and 'update-grub'. Modify /boot/grub/menu.lst so "hda" is replaced with "sda" for servers with scsi drives. (Note that "#" does not imply a comment in menu.list.) Re-run 'update-grub'. Note that after every new kernel, 'update-grub' must be run and menu.lst should be consulted to verify that the kernels are in the correct order, but menu.lst most likely will not need to be modified.
Have your boot disk/CD handy. Should you need it, you will likely need to 'mount -t ext2 /dev/yourdrive /mnt' and 'chroot /mnt lilo' or 'chroot /mnt update-grub'. If at first you don't succeed, try again.
Debian package management utilities include dpkg, dselect, apt-get, apt-cache, and aptitude. Know how to use these. The list of apt sources used is in /etc/apt/sources.list. Make sure you are getting security updates and run 'apt-get update && apt-get upgrade' immediately after installation. /etc/apt/sources.list should contain something like the following:
deb http://http.us.debian.org/debian/ stable main non-free contrib deb-src http://http.us.debian.org/debian/ stable main non-free contrib deb http://non-us.debian.org/debian-non-US stable/non-US main contrib non-free deb-src http://non-us.debian.org/debian-non-US stable/non-US main contrib non-free deb http://security.debian.org stable/updates main contrib non-free deb-src http://security.debian.org stable/updates main contrib non-free
Important: If you are running a mixed system with some packages from stable and others from testing or unstable, you must add the following line to /etc/apt.conf:
APT::Default-Release "stable";
'dselect' is useful for selecting and installing a bunch of packages at once, such as when setting up a new system. 'dpkg -l' lists all installed packages. 'dpkg -l 'pattern' will list information about all packages matching "pattern".
Run 'apt-get update && apt-get upgrade' religiously to update the list of packages and upgrade all Debian packages to the newest version. If you are uncertain whether an upgrade will break something, try 'apt-get -s upgrade' first. Subscribe to the debian-security-announce mailing list.
Use 'apt-get install foo' to install package "foo" (again the "-s" option is available) and 'apt-get remove foo' to remove it. To purge configuration files as well, follow with 'dpkg --purge foo' or just use 'apt-get --purge remove foo'.
Other useful commands include 'dpkg -L', dpkg -S', 'apt-cache show', 'apt-cache search', 'apt-cache showpkg', 'apt-cache depends', and 'apt-get clean'. To build a custom src deb, use 'apt-get source', 'apt-cache showsrc', 'apt-get build-dep', edit debian/rules, and run 'dpkg-buildpackage' and 'dpkg -i'.
For servers which have multiple admins, create an admin group wheel with 'addgroup wheel'. Admins are added to this and group adm (for reading logs) with 'usermod -G adm,wheel username' for each user. Also, consider super (a bit like sudo; configuration is in /etc/super.tab) to allow admins to run routine rootly tasks without root. Sample super.tab file for cut and paste:
:global_options \
logfile=/var/log/super.log # Log to this file... \
syslog=y \
env=EDITOR,VISUAL
:define SuperUsers foo,bar,baz
update '/usr/bin/apt-get update' uid=root $SuperUsers
ls '/bin/ls' uid=root $SuperUsers
quota '/usr/bin/quota' uid=root $SuperUsers
samba-restart '/etc/init.d/samba restart' uid=root $SuperUsers
I have written a number of custom scripts for use on production servers. (Scripts available upon request.) Many of these scripts should be placed in /usr/local/bin/ and /usr/local/sbin/. Cron jobs are run out of /etc/crontab and /etc/cron.*ly. I recommend weekly backups, logging load average every 10 minutes, checking that services are still running every 15 minutes, and reporting on ethernet collisions, disk fullitude ('df'), drive problems (smartsuite, requires bios support), and high load averages nightly.
For backups performed via scp to a central backup server, ssh keys are needed. To generate ssh keys for root, run 'ssh-keygen' (specifying type with -t). The public key which is generated in /root/.ssh/ should be appended to .ssh/authorized_keys in the backup users homedir on the backup server. The first backup should then be run manually.
I have recently switched to syslog-ng for logging. syslog-ng configuration is in /etc/syslog-ng/ and /etc/logrotate.d/. It may be appropriate to add the line "delaycompress" to the top of /etc/logrotate.d/syslog-ng.
For keeping accurate time, install the packages ntp, ntp-simple, and ntpdate. The server option in /etc/ntp.conf should be set to the IP of the NTP server. ntpdate can be run manually just after install with 'ntpdate -u ipaddr' to initially sync time.
To ensure that the servers are using a proxy for updates and other web traffic, add the following lines (suitably modified) to /etc/profile:
export http_proxy="http://proxyserver:port" export ftp_proxy="ftp://proxyserver:port"
User systems require LDAP for authentication. This involves configuration in /etc/ldap.conf and /etc/pam.d/ldap.settings, as well as symlinking appropriate files in the /etc/pam.d/ directory to ldap.settings (for example, netatalk, imap, pop3, ppp, ssh). The password and shadow lines in /etc/nsswitch.conf may need to be modified. /etc/ldap.conf should contain entries for host, base, uri, ldap_version, port, scope, pam_password, nss_base_passwd, nss_base_shadow, and nss_base_group.
For systems with (but not booting from) software RAID, install the raidtools2 package, read the howto in /usr/share/docs/raidtools2/ for instructions about modifying /etc/raidtab, run 'mkraid /dev/md0' (verify with 'cat /proc/mdstat'), format the RAID array ('mke2fs -j /dev/md0'), modify /etc/fstab, and mount.
For servers with hardware RAID, install alien and libstdc++2.9-glibc2.1, obtain storage manager software from Adaptec, and install the package dptapps with 'alien -i /usr/local/src/sm_linux_v314_install.prm'. The following changes then need to be made:
mknod -m 600 /dev/dpti0 c 151 0 ln -s /usr/dpt/raidutil /usr/local/sbin/raidutil ln -s /etc/rc.d/init.d/dpt /etc/init.d/dpt update-rc.d -f dpt defaults
Configuration changes may also be required to the .ini files in /etc/dpt/.
That's about it for vanilla servers. Next comes installing and configuring particular services and determining what ports they use (for specifying in the firewall rules). The final (and arguably most important step) is documentation. And after that, it's routine maintenance, which largely consists of reading log files (/var/log/*) and 'apt-get update && apt-get upgrade'.
Enjoy!