Moving var, tmp Off the Root in FreeBSD

One one of the first things I do on a newly installed FreeBSD system is to move /var and /tmp to under /usr. Since I usually allocate about 4Gb for the root slice and the rest of a disk—usually several hundred gigabytes—goes to /usr (well, there’s also the swap slice that takes few gigabytes) having /var and /tmp there is more comfortable as some log files, database files, or some temp files can sometimes grow to multi-gigabyte size and exhaust the root space.

Below is a simple procedure to move the /var to /usr/var and /tmp to /usr/var/tmp. This is best to do early on in a new system installation since many services tend to hook into /tmp and/or /var, and may thus lock files in those directories making the move more difficult. If you’re making this move on an established system, at least stop all the services that might interfere with the process (such as database services). It might even be a good idea to boot into a single user mode (if you do so, remember to correctly mount your disks before proceeding). I usually do this early in a new system install, before installing any major services, or at least before scripting them to run.

  1. Move /var to /usr/var
    mkdir /usr/var
    cd /var
    tar cvf - . | (cd /usr/var; tar xvf - )
    cd /
    chflags -R noschg /var
    rm -rf /var
    ln -s /usr/var /var
  2. Move /tmp to /usr/var/tmp
    mkdir /usr/var/tmp
    cd /tmp
    tar cvf - . | (cd /usr/var/tmp; tar xvf - )
    cd /
    chflags -R noschg /tmp
    rm -rf /tmp
    ln -s /usr/var/tmp /tmp
    chmod -h 777 /tmp
    chmod 1777 /usr/var/tmp

FreeBSD 7.0-RELEASE Kernel Optimization

Below is my FreeBSD 7.0 kernel configuration file.  I created it on my reference system, to be used on four production servers whose hardware configurations differ some.  For that reason there’re few options (indicated as “[OPTION]”) that are conditional for the configurations. I’ve also left in IPv6 options which are currently commented out, but that I may take into use later if/when IPv6 becomes more prevalent in the environment these servers operate.

# INERTIA -- Inertia kernel configuration file for FreeBSD/i386
# For more information on this file, please read the handbook section on
# Kernel Configuration Files:
# The handbook is also available locally in /usr/share/doc/handbook
# if you've installed the doc distribution, otherwise always see the
# FreeBSD World Wide Web server ( for the
# latest information.
# An exhaustive list of options and more detailed explanations of the
# device lines is also present in the ../../conf/NOTES and NOTES files.
# If you are in doubt as to the purpose or necessity of a line, check first
# in NOTES.
# Based on
# $FreeBSD: src/sys/i386/conf/GENERIC,v 1.474. 2008/02/06 03:24:28 scottl Exp $

ident		INERTIA
machine		i386
cpu		I686_CPU

options		SMP 			# Symmetric MultiProcessor Kernel (`device apic' is also required for multiprocessor use)
options 	SCHED_4BSD		# 4BSD scheduler
options 	PREEMPTION		# Enable kernel thread preemption
options 	INET			# InterNETworking
#options 	INET6			# IPv6 communications protocols
options 	FFS			# Berkeley Fast Filesystem
options 	SOFTUPDATES		# Enable FFS soft updates support
options 	UFS_ACL			# Support for access control lists
options 	UFS_DIRHASH		# Improve performance on big dirs
options 	CD9660			# ISO 9660 Filesystem
options 	PROCFS			# Process filesystem (requires PSEUDOFS)
options 	PSEUDOFS		# Pseudo-filesystem framework
options		MSDOSFS			# MSDOS filesystem support (for floppies)
options 	COMPAT_43		# Compatible with BSD 4.3 (required)
options 	COMPAT_FREEBSD4		# Compatible with FreeBSD4
options 	COMPAT_FREEBSD5		# Compatible with FreeBSD5
options 	COMPAT_FREEBSD6		# Compatible with FreeBSD6
options 	SCSI_DELAY=15000	# Delay (in ms) before probing SCSI (TWA/TWE issue)
options 	KTRACE			# ktrace(1) support
options 	SYSVSHM			# SYSV-style shared memory
options 	SYSVMSG			# SYSV-style message queues
options 	SYSVSEM			# SYSV-style semaphores
options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev, may be needed to hot-plug USB keyboards
options 	AHC_REG_PRETTY_PRINT	# Print register bitfields in debug output
options 	AHD_REG_PRETTY_PRINT	# Print register bitfields in debug output
options 	ADAPTIVE_GIANT		# Giant mutex is adaptive
options		ACCEPT_FILTER_HTTP	# Must be here or AcceptFilter won't work w/Apache2
options		SC_DISABLE_REBOOT	# Disable Ctrl-Alt-Del reboot (this is a server)

device		apic			# I/O APIC (required)
device		npx   			# The Numeric Processing eXtension driver (required)

device		pci
device		isa			# Required by npx

device		fdc			# Floppy drives

device		ata			# ATA and ATAPI devices
device		atadisk			# ATA disk drives
device		ataraid			# ATA RAID drives [OPTION]
device		atapicd			# ATAPI CDROM drives
options 	ATA_STATIC_ID		# Static device numbering

device		scbus			# SCSI bus (required for SCSI, ALSO REQ'D FOR SATA-RAID, USB/umass)
device		da			# Direct Access (disks)
device		pass			# Passthrough device (direct SCSI access)

#device		twe			# 3ware ATA RAID [OPTION]
#device		twa			# 3ware 9000 series PATA/SATA RAID [OPTION]
#options	TWA_DEBUG		# 0-10; 10 prints the most messages; enable for twa debug only

device		atkbdc			# AT keyboard controller
device		atkbd			# AT keyboard
device		kbdmux			# keyboard multiplexer

device		vga			# VGA video card driver
device		sc			# syscons, the default console driver
device		sio			# 8250, 16[45]50 based serial ports

device		ppc			# Parallel port
device		ppbus			# Parallel port bus (required)
device		lpt			# Printer
device		ppi			# Parallel port interface device

#device		miibus			# MII bus support (required by some NICs) [OPTION]
#device		fxp			# Intel EtherExpress PRO/100B (82557, 82558); requires miibus [OPTION]
device		em			# Intel PRO/1000 adapter Gigabit Ethernet Card [OPTION]

options		DEVICE_POLLING		# Imporoves network driver performance

device		coretemp		# On-die temperature sensor on Intel Core and newer CPUs [OPTION]

device		loop			# Network loopback
device		random			# Entropy device
device		ether			# Ethernet support
device		pty			# Pseudo-ttys (telnet etc)
#device		gif			# IPv6 and IPv4 tunneling
#device		faith			# IPv6-to-IPv4 relaying (translation)
device		bpf			# Berkeley packet filter

# USB support [OPTION]
device		uhci			# USB support / UHCI PCI->USB interface
device		ohci			# USB support / OHCI PCI->USB interface
device		ehci			# USB support / EHCI PCI->USB interface (USB 2.0)
device		usb			# USB support / USB Bus (required)
device		ugen			# USB support / Generic
device		uhid			# USB support / "Human Interface Devices"
device		ukbd			# USB support / Keyboard
device		umass			# USB support / Disks/Mass storage - Requires scbus and da

options		COMPAT_LINUX		# Linux compat / Enable Linux ABI emulation
options		COMPAT_AOUT		# Linux compat / Enable i386 a.out binary support
options		LINPROCFS		# Linux compat / Enable procfs support (COMPAT_LINUX / PSEUDOFS)

FreeBSD 7.0 New Install

I’ve been running various versions of FreeBSD since 2001, and over that time the installation procedure has changed several times as new versions of the operating system have been released. Since I’m jack of all trades (or at least many trades), often several months or more passes without significant work in the UNIX environment, and my memory fades as I’m engrossed in something completely different such as software development work. I’m still running FreeBSD 6.1 in production environments, but want to make the move to 7.0 soon. Before doing so, however, I decided to set up a reference system and document the setup process to avoid major surprises (or my own memory glitches) as I reinstall the OS on the production systems — and hopefully have as a result minimum downtime possible.

Step one… FreeBSD installation using the Custom install. I post my notes below; perhaps someone will find this useful. The system is being set up as a web/db/mail server that is administered remotely; no X11 is needed or desired.

  1. Use custom install w/defaults except..
    Skip PCCARD: YES
    Media Type: CDROM
  2. Set keymap if not ASCII (I use Finnish keyboard so I selected “Finnish ISO”) and timezone in post install.
  3. Old mount point & label info is ok (from the older installations). I use “4069M” for boot, “6100M” for swap, and rest for application data (generally a largish RAID-5 or RAID-6 array is used), the respective mount point labels are “/”, swap, and “/usr”. I use standard boot record since no other operating systems are installed on the system. Boot drive is made bootable, softupdates is enabled on the data slice.
  4. Configure network, system name, keyboard map (sysinstall should’ve created delta for it) in /etc/rc.conf.  Much more should and will go into it, but the basics that will get the system online are:

    keymap="finnish.iso" #obviously optional :)
    ifconfig_em0="inet netmask" #currently installing behind a firewall
    fsck_y_enable="yes" #this is good to set in case your system crashes during setup without orderly shutdown.. you don't have to press "yes" a million times

    Note that you may have a different kind of network interface and you might have to adjust the “ifconfig_em0 accordingly.
  5. Configure DNSes in /etc/resolv.conf. I use OpenDNS servers, like so:

  6. If you’re basing this installation on an older install, import your old .cshrc or the equivalent alias/setting file of your favorite shell (makes the life easier as aliases work, etc).
  7. If you’re not using X11, enter `WITHOUT_X11′ in /etc/make.conf so you don’t have to set it in the environment every time.

    WITHOUT_X11=yes #don't compile GUI to ports apps
    CPUTYPE=i686 #set this for modern Intel CPUs
    OPENSSLBASE=/usr/local #obviously if you use OpenSSL
  8. Build /usr/ports/net/cvsup with WITHOUT_X11 set in make.conf (as above) or in the environment, or use /usr/ports/net/cvsup-without-gui/ and update the ports tree.First create /usr/local/etc/cvsup/supfile.ports containing:

    *default base=/usr/local/etc/cvsup
    *default prefix=/usr
    *default release=cvs tag=.
    *default delete use-rel-suffix

    If you want to use the fastest cvs server available, install /usr/ports/sysutils/fastest_cvsup/ and run with fastest_cvsup -c us (replace ‘us’ with your local country code if you’re not in the U.S. :-)), then use the cvs server indicated as the default host. Then update ports with
    /usr/local/bin/cvsup -g -L 2 /usr/local/etc/cvsup/supfile.ports

    You can use an optional `-d 100′ to limit file deletions to 100 initially to make sure update is working and the entire ports tree won’t be wiped out. Then remove it for full run (intial run *will* need to delete more than 100 files, but they’re not all in sequence)

  9. Build & install /usr/ports/editors/joe (or whatever your favorite editor might be); this makes life easier as configuration progresses.
  10. Build & install /usr/ports/security/openssh-portable. Use defaults + select `Enable CHROOT support’ (for later use)
  11. Create a non-root user for remote login. vipw is an easy way to manage users.
  12. Configure OpenSSH daemon in /usr/local/etc/ssh/sshd_config & make sure sshd starts (set openssh_enable=”yes” in /etc/rc.conf, and make sure the start file is called /usr/local/etc/rc.d/; reboot may be required to create the necessary server keys & start the service (confirm with ps -waux | grep “ssh”).sshd_config params of note (for initial access) are..

    AllowUsers root MyUserName
    PermitRootLogin without-password #allow root login only with a RSA-key
    PasswordAuthentication yes
    UsePAM no
    UseDNS no

    ** remote login should be possible at this point **
  13. Create /usr/local/etc/cvsup/supfile.sources with the below content, then update sources with /usr/local/bin/cvsup -g -L 2 /usr/local/etc/cvsup/supfile.sources

    *default base=/usr/local/etc/cvsup
    *default prefix=/usr
    *default release=cvs tag=RELENG_7_0
    *default delete use-rel-suffix
  14. Review kernel configuration at /usr/src/sys/i386/conf (see my kernel defaults in the next post).
  15. Build & install new world if any deltas were applied in source update:
    cd /usr/src && make buildworld.
    If there are problems, try the following, then run buildworld again.

    cd /usr/obj
    chflags -R noschg *
    rm -rf *

    When buildworld completes, reboot the system, select option 4, or interrupt the reboot (option 6) and type boot -s to boot into single user mode; accept /bin/sh as the shell, then continue with the following commands to install new world:

    mount -u /
    mount -a -t ufs
    swapon -a

    cd /usr/src
    make installworld

    exit (goes multi-user)

  16. Build custom kernel & install with below commands:

    cd /usr/src
    make cleandir
    make buildkernel
    if this fails, try cd /usr/src/usr.sbin/config/ && make depend all install clean and also check your kernel configuration file for problems, then start the above build process again.
  17. Make a backup copy of the old kernel and install the new:

    cp -Rp /boot/kernel /boot/kernel.recent
    make installkernel
  18. Reboot & confirm that the latest kernel version is running with uname -a (or uname -rs).
  19. Make a copy of the functional kernel if boot is ok:chflags -R noschg /boot/kernel && cp -Rp /boot/kernel /boot/ && chflags -R schg /boot/kernel
  20. Set /boot/loader.conf parameters, like so:

    kern.ipc.nmbclusters=16384 # Set the number of mbuf clusters
    kern.ipc.maxsockets=16384 # Set the number of tcp sockets

Basic install & kernel setup is now complete.