Tag: enforcer

NFS Automount, The Fourth Iteration (the complete rewrite)

** Note: This post has been significantly altered on 18 July 2013 from the original, posted a few days earlier.

A few days ago I released the fourth iteration of the NFS Automount script, with some minor changes to the previous version from December 2011. The earlier versions were released May 2011 (first CentOS Linux version), and July 2010 (originally written for FreeBSD).

Upon releasing the fourth version I realized the script was becoming brittle, the logic was, well, somewhat illogical, and minor refactoring would not help. Hence this complete rewrite of the script, now called “nfs_automount”, was born. It is conceptually based on the older versions, and I also borrowed some ideas from AutoNFS script on Ubuntu’s Community Wiki.

Like the earlier version, the goal of this script is to provide static (i.e. /etc/fstab-like) NFS mounts, while at the same time supporting cross-mounts between servers.

The other non-fstab alternative is to lazy-mount NFS shares with autofs (where available), but with it NFS shares are not continually maintained. When a remote share is accessed, it takes a few moments for it to become accessible as autofs mounts the share on-demand. While autofs times out a mounted share after some time of inactivity, it does not unmount the share before the timeout has lapsed in the event the remote server becomes inaccessible. While on-demand mounting may save some bandwidth, it is not suitable for all applications. Furthermore, when a system has one or more active mounted shares off of a server that goes offline, unexpected behavior is often observed on the client server until the now-defunct NFS shares are unmounted, or the remote server becomes available once again.

nfs_automount offers a solution:

  • The NFS shares are not statically defined in /etc/fstab so that the system startup is not delayed even when the remote server is not available. As soon as the shares become available they’re automatically mounted. If multiple servers cross-mount NFS shares from each other, and the servers are turned on at the same time, nfs_automount ensures that all mounts are established as soon as the shares become available.

  • The shares are monitored at a frequency you define, for example, every 60 seconds. If a share has become dismounted, stale, or their exporting server has become inaccessible, nfs_automount takes action to correct the situation: dismounted and stale shares are attempted to be remounted (stale shares are first immediately unmounted), and shares whose remote NFS service has disappeared are unmounted to prevent impact on the client system stability. Once a remote NFS service returns online, or definition of a previously stale share is reinstated, any shares that were unmounted as a result of those conditions are remounted.

  • The script is intended to run as a daemon (an upstart job script is provided for Ubuntu), and it reads its configuration from /etc/nfs-automount.conf where you can conveniently define the shares to be mounted and monitored along with some other options. You can also set ‘RUNTYPE’ option to ‘cron’, and run the script from crontab if you so choose.

  • You can define the shares to be mounted either as Read/Write, or Read Only. Of course, a share will be Read Only regardless of this setting if it has been exported as Read Only on the remote server.

  • An option to define a remote check file is provided. If provided in the configuration for a share, its unreachability can alert of a problem on the exporting server, such as a failed filesystem mount, even when the NFS share is otherwise working correctly. You can easily expand this feature to add additional functionality.

  • Provides clear logging which provides alerts by default, and more informative detail if you turn ‘DEBUGLOG’ setting to ‘true’.

  • Written in bash script with modular and clear syntax.

  • Tested on Ubuntu 12.x (should also work on Debian) and CentOS 6.x (should also work on RedHat). The service installation instructions (available on GitHub) have been written for Ubuntu, so if you’re installing the script for CentOS/RedHat, you will need to alter the installation steps somewhat. FreeBSD is no longer explicitly supported, but I believe it should work with minor modifications. I have not tested with Solaris or other *NIX environments. If you try, please post comments here!

  • Can be easily run as a service (upstart script is provided), or from crontab; the script works with crontab with just a single configuration switch change.

  • Distributed under MIT license.

Rather than posting the code (now 400+ lines) here, I have created a repository on GitHub from where it is easy to download or clone.

Enjoy! 🙂

NFS automount evolves

** NOTE: This version is obsoleted! The latest version can be found here.

I’ve updated the NFS automount script that provides “self-healing” NFS mounts. The script now allows a mount to be defined as read-write or read-only, and then subsequently monitors that the share is mounted as R/W or R/O (of course, it can’t mount a share that has been shared as R/O as R/W). Both Linux (tested on CentOS 6.1) and FreeBSD versions are provided.

Since various systems can provide cross-mounts via NFS, and they may be started/rebooted at the same time, various shares may or may not be available at each system’s boot time. By utilizing this script the mounts become available soon after the respective share becomes available (simply adjust the run frequency in crontab to the needs of your specific application). Also, by not adding the NFS mount points in fstab the boot process is not delayed by a share that is not [yet] available.

First for CentOS/Linux:

#!/bin/sh

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin

# set mount/remount request flags
mount=false
remount=false

# remote system name
remotesystem="$1"

# rw/ro
if [ "$2" = "rw" ]; then
    mountmode="-w"
else
    mountmode="-r"
fi

# remote share name
remoteshare="$3"

# local mount point
mountpoint="$4"

# file to indicate local mount status
testfile=${mountpoint}/"$5" 

# rw test file
rw_testfile=${mountpoint}/nfs_enforcer_rw_testfile

# command locations
pingcmd=/bin/ping
showmountcmd=/usr/sbin/showmount
grepcmd=/bin/grep
mountcmd=/bin/mount
umountcmd=/bin/umount
statcmd=/usr/bin/stat
touchcmd=/bin/touch
rmcmd=/bin/rm

# --- end variables ---

# make sure the mountpoint is not stale
statresult=`${statcmd} ${mountpoint} 2>&1 | ${grepcmd} "Stale"`

if [ "${statresult}" != "" ]; then
   #result not empty: mountpoint is stale; remove it
   ${umountcmd} -f ${mountpoint}
fi

# ping the remote system (2 sec timeout)
${pingcmd} -w2 -c1 -q ${remotesystem} > /dev/null 2>&1

# make sure the remote system is reachable
if [ "$?" -eq "0" ]; then

   # query the availability of the remote share; not empty result indicates OK   
   offsiteshare=`${showmountcmd} -e ${remotesystem} | ${grepcmd} "${remoteshare}"`
   if [ "${offsiteshare}" != "" ] ; then

      # make sure the local mount point (directory) exists (so that [re-]mount, if necessary, is valid)
      if [ -d ${mountpoint} ] ; then

         localmount=`${mountcmd} | ${grepcmd} "${mountpoint}"`

         # make sure the share test file is _not_ present (to make sure the mountpoint is inactive)
         if [ ! -f ${testfile} ] ; then

            # make sure the local mountpoint is inactive (double checking)
            if [ "${localmount}" = "" ] ; then

               # all set to go; request mount
               mount=true
            fi
         
         else 
            
            # make sure the local mountpoint is active (double checking)
            if [ "${localmount}" != "" ] ; then

               # attempt to create a test file..
               ${touchcmd} ${rw_testfile} > /dev/null  2>&1

               # ..and test its existence; first handle RW mounted shares:
               if [ -f ${rw_testfile} ] ; then

                  # share was RO requested
                  if [ "$2" = "ro" ]; then
                     remount=true
                  fi

                  # Delete the testfile
                  ${rmcmd} ${rw_testfile}

               # hanle RO mounted shares:
               else

                  # share was RW requested
                  if [ "$2" = "rw" ]; then
                     remount=true
                  fi
               fi
            fi
         fi
      fi
   fi
fi

# perform remount (unmount, request mount)
if $remount ; then
   ${umountcmd} -f ${mountpoint}
   mount=true
fi

# perform mount when so requested
if $mount ; then
   ${mountcmd} ${mountmode} -t nfs ${remotesystem}:${remoteshare} ${mountpoint}
fi

exit 0

Then for FreeBSD/UNIX:

#!/bin/sh

SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/bin:/usr/bin:/usr/sbin:/usr/local/bin

# set mount/remount request flags
mount=false
remount=false

# remote system name
remotesystem="$1"

# rw/ro
if [ "$2" = "rw" ]; then
    mountmode="-w"
else
    mountmode="-r"
fi

# remote share name
remoteshare="$3"

# local mount point
mountpoint="$4"

# file to indicate local mount status
testfile=${mountpoint}/"$5" 

# rw test file
rw_testfile=${mountpoint}/nfs_enforcer_rw_testfile

# command locations
pingcmd=/sbin/ping
showmountcmd=/usr/bin/showmount
grepcmd=/usr/bin/grep
mountcmd=/sbin/mount
umountcmd=/sbin/umount
statcmd=stat
touchcmd=/usr/bin/touch
rmcmd=/bin/rm

# --- end variables ---

# make sure the mountpoint is not stale
statresult=`${statcmd} ${mountpoint} 2>&1 | ${grepcmd} "Stale"`

if [ "${statresult}" != "" ]; then
   #result not empty: mountpoint is stale; remove it
   ${umountcmd} -f ${mountpoint}
fi

# ping the remote system (2 sec timeout)
remoteping=`${pingcmd} -c1 -o -q -t2 ${remotesystem} | grep " 0.0%"`

# make sure the remote system is reachable
if [ "${remoteping}" != "" ] ; then
   
   # query the availability of the remote share; not empty result indicates OK   
   offsiteshare=`${showmountcmd} -e ${remotesystem} | ${grepcmd} "${remoteshare}"`
   if [ "${offsiteshare}" != "" ] ; then
   
      # make sure the local mount point (directory) exists (so that [re-]mount, if necessary, is valid)
      if [ -d ${mountpoint} ] ; then

         localmount=`${mountcmd} | ${grepcmd} "${mountpoint}"`
      
         # make sure the share test file is _not_ present (to make sure the mountpoint is inactive)
         if [ ! -f ${testfile} ] ; then
        
            # make sure the local mountpoint is inactive (double checking)
            if [ "${localmount}" = "" ] ; then

               # all set to go; request mount
               mount=true
            fi
               
         else

            # make sure the local mountpoint is active (double checking)
            if [ "${localmount}" != "" ] ; then

               # attempt to create a test file..
               ${touchcmd} ${rw_testfile} > /dev/null  2>&1

               # ..and test its existence; first handle RW mounted shares:
               if [ -f ${rw_testfile} ] ; then

                  # share was RO requested
                  if [ "$2" = "ro" ]; then
                     remount=true
                  fi

                  # Delete the testfile
                  ${rmcmd} ${rw_testfile}

               # hanle RO mounted shares:
               else

                  # share was RW requested
                  if [ "$2" = "rw" ]; then
                     remount=true
                  fi
               fi
            fi
         fi
      fi
   fi
fi

# perform remount (unmount, request mount)
if $remount ; then
   ${umountcmd} -f ${mountpoint}
   mount=true
fi

# perform mount when so requested
if $mount ; then
   ${mountcmd} ${mountmode} -t nfs ${remotesystem}:${remoteshare} ${mountpoint}
fi

exit 0

You should run the automount script from a runfile, like so:

#!/bin/sh

NFS_ENFORCE=/usr/local/sbin/nfs_enforcer

# Separate the following parameters with spaces:
#
# - nfs enforcer command (set above)
# - remote system name (must be resolvable)
# - read/write (rw) or read-only (ro); NOTE: share may be read-only regardless of how this is set
# - remote share name (from remote's /etc/exports)
# - local mount point (existing local directory)
# - share test file (an immutable file on the share)

# e.g.
# $NFS_ENFORCE dbsysvm rw /nfs4shares/conduit /mnt/dbsys_conduit .conduit@dbsysvm
# or (for local remount read-only)
# $NFS_ENFORCE localhost ro /var/web/projects/repository /mnt/rorepo .repository@localhost

$NFS_ENFORCE localhost ro /var/web/projects/repository /mnt/rorepo .repository@localhost

exit 0

..and call the the above runfile from crontab:

*/10  *  *  *  *  root  /usr/local/sbin/nfs_enforcer.batch > /dev/null