• My Links at Diigo
  • Ergonomics Resource

RANDOM.AC/CESS

Musings of a Web Technologist

Home Archives for enforcer

NFS Automount, The Fourth Iteration (the complete rewrite)

Posted on 13 July 2013 by Ville 7 Comments

** Note: This post has been significantly altered on 18 July 2013 from the original, posted a few days earlier.

A few days ago I released the fourth iteration of the NFS Automount script, with some minor changes to the previous version from December 2011. The earlier versions were released May 2011 (first CentOS Linux version), and July 2010 (originally written for FreeBSD).

Upon releasing the fourth version I realized the script was becoming brittle, the logic was, well, somewhat illogical, and minor refactoring would not help. Hence this complete rewrite of the script, now called “nfs_automount”, was born. It is conceptually based on the older versions, and I also borrowed some ideas from AutoNFS script on Ubuntu’s Community Wiki.

Like the earlier version, the goal of this script is to provide static (i.e. /etc/fstab-like) NFS mounts, while at the same time supporting cross-mounts between servers.

The other non-fstab alternative is to lazy-mount NFS shares with autofs (where available), but with it NFS shares are not continually maintained. When a remote share is accessed, it takes a few moments for it to become accessible as autofs mounts the share on-demand. While autofs times out a mounted share after some time of inactivity, it does not unmount the share before the timeout has lapsed in the event the remote server becomes inaccessible. While on-demand mounting may save some bandwidth, it is not suitable for all applications. Furthermore, when a system has one or more active mounted shares off of a server that goes offline, unexpected behavior is often observed on the client server until the now-defunct NFS shares are unmounted, or the remote server becomes available once again.

nfs_automount offers a solution:

  • The NFS shares are not statically defined in /etc/fstab so that the system startup is not delayed even when the remote server is not available. As soon as the shares become available they’re automatically mounted. If multiple servers cross-mount NFS shares from each other, and the servers are turned on at the same time, nfs_automount ensures that all mounts are established as soon as the shares become available.
  • The shares are monitored at a frequency you define, for example, every 60 seconds. If a share has become dismounted, stale, or their exporting server has become inaccessible, nfs_automount takes action to correct the situation: dismounted and stale shares are attempted to be remounted (stale shares are first immediately unmounted), and shares whose remote NFS service has disappeared are unmounted to prevent impact on the client system stability. Once a remote NFS service returns online, or definition of a previously stale share is reinstated, any shares that were unmounted as a result of those conditions are remounted.
  • The script is intended to run as a daemon (an upstart job script is provided for Ubuntu), and it reads its configuration from /etc/nfs-automount.conf where you can conveniently define the shares to be mounted and monitored along with some other options. You can also set ‘RUNTYPE’ option to ‘cron’, and run the script from crontab if you so choose.
  • You can define the shares to be mounted either as Read/Write, or Read Only. Of course, a share will be Read Only regardless of this setting if it has been exported as Read Only on the remote server.
  • An option to define a remote check file is provided. If provided in the configuration for a share, its unreachability can alert of a problem on the exporting server, such as a failed filesystem mount, even when the NFS share is otherwise working correctly. You can easily expand this feature to add additional functionality.
  • Provides clear logging which provides alerts by default, and more informative detail if you turn ‘DEBUGLOG’ setting to ‘true’.
  • Written in bash script with modular and clear syntax.
  • Tested on Ubuntu 12.x (should also work on Debian) and CentOS 6.x (should also work on RedHat). The service installation instructions (available on GitHub) have been written for Ubuntu, so if you’re installing the script for CentOS/RedHat, you will need to alter the installation steps somewhat. FreeBSD is no longer explicitly supported, but I believe it should work with minor modifications. I have not tested with Solaris or other *NIX environments. If you try, please post comments here!
  • Can be easily run as a service (upstart script is provided), or from crontab; the script works with crontab with just a single configuration switch change.
  • Distributed under MIT license.

Rather than posting the code (now 400+ lines) here, I have created a repository on GitHub from where it is easy to download or clone.

Enjoy! 🙂

Filed Under: Networking, Technical, UNIX Tagged With: automount, enforcer, Linux, monitoring, nfs, self-healing, unix

NFS automount evolves

Posted on 19 December 2011 by Ville Leave a Comment

** NOTE: This version is obsoleted! The latest version can be found here.

I’ve updated the NFS automount script that provides “self-healing” NFS mounts. The script now allows a mount to be defined as read-write or read-only, and then subsequently monitors that the share is mounted as R/W or R/O (of course, it can’t mount a share that has been shared as R/O as R/W). Both Linux (tested on CentOS 6.1) and FreeBSD versions are provided.

Since various systems can provide cross-mounts via NFS, and they may be started/rebooted at the same time, various shares may or may not be available at each system’s boot time. By utilizing this script the mounts become available soon after the respective share becomes available (simply adjust the run frequency in crontab to the needs of your specific application). Also, by not adding the NFS mount points in fstab the boot process is not delayed by a share that is not [yet] available.

First for CentOS/Linux:

Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
#!/bin/sh
 
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
 
# set mount/remount request flags
mount=false
remount=false
 
# remote system name
remotesystem="$1"
 
# rw/ro
if [ "$2" = "rw" ]; then
    mountmode="-w"
else
    mountmode="-r"
fi
 
# remote share name
remoteshare="$3"
 
# local mount point
mountpoint="$4"
 
# file to indicate local mount status
testfile=${mountpoint}/"$5"
 
# rw test file
rw_testfile=${mountpoint}/nfs_enforcer_rw_testfile
 
# command locations
pingcmd=/bin/ping
showmountcmd=/usr/sbin/showmount
grepcmd=/bin/grep
mountcmd=/bin/mount
umountcmd=/bin/umount
statcmd=/usr/bin/stat
touchcmd=/bin/touch
rmcmd=/bin/rm
 
# --- end variables ---
 
# make sure the mountpoint is not stale
statresult=`${statcmd} ${mountpoint} 2>&1 | ${grepcmd} "Stale"`
 
if [ "${statresult}" != "" ]; then
   #result not empty: mountpoint is stale; remove it
   ${umountcmd} -f ${mountpoint}
fi
 
# ping the remote system (2 sec timeout)
${pingcmd} -w2 -c1 -q ${remotesystem} > /dev/null 2>&1
 
# make sure the remote system is reachable
if [ "$?" -eq "0" ]; then
 
   # query the availability of the remote share; not empty result indicates OK  
   offsiteshare=`${showmountcmd} -e ${remotesystem} | ${grepcmd} "${remoteshare}"`
   if [ "${offsiteshare}" != "" ] ; then
 
      # make sure the local mount point (directory) exists (so that [re-]mount, if necessary, is valid)
      if [ -d ${mountpoint} ] ; then
 
         localmount=`${mountcmd} | ${grepcmd} "${mountpoint}"`
 
         # make sure the share test file is _not_ present (to make sure the mountpoint is inactive)
         if [ ! -f ${testfile} ] ; then
 
            # make sure the local mountpoint is inactive (double checking)
            if [ "${localmount}" = "" ] ; then
 
               # all set to go; request mount
               mount=true
            fi
        
         else
            
            # make sure the local mountpoint is active (double checking)
            if [ "${localmount}" != "" ] ; then
 
               # attempt to create a test file..
               ${touchcmd} ${rw_testfile} > /dev/null  2>&1
 
               # ..and test its existence; first handle RW mounted shares:
               if [ -f ${rw_testfile} ] ; then
 
                  # share was RO requested
                  if [ "$2" = "ro" ]; then
                     remount=true
                  fi
 
                  # Delete the testfile
                  ${rmcmd} ${rw_testfile}
 
               # hanle RO mounted shares:
               else
 
                  # share was RW requested
                  if [ "$2" = "rw" ]; then
                     remount=true
                  fi
               fi
            fi
         fi
      fi
   fi
fi
 
# perform remount (unmount, request mount)
if $remount ; then
   ${umountcmd} -f ${mountpoint}
   mount=true
fi
 
# perform mount when so requested
if $mount ; then
   ${mountcmd} ${mountmode} -t nfs ${remotesystem}:${remoteshare} ${mountpoint}
fi
 
exit 0

Then for FreeBSD/UNIX:

Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
#!/bin/sh
 
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/bin:/usr/bin:/usr/sbin:/usr/local/bin
 
# set mount/remount request flags
mount=false
remount=false
 
# remote system name
remotesystem="$1"
 
# rw/ro
if [ "$2" = "rw" ]; then
    mountmode="-w"
else
    mountmode="-r"
fi
 
# remote share name
remoteshare="$3"
 
# local mount point
mountpoint="$4"
 
# file to indicate local mount status
testfile=${mountpoint}/"$5"
 
# rw test file
rw_testfile=${mountpoint}/nfs_enforcer_rw_testfile
 
# command locations
pingcmd=/sbin/ping
showmountcmd=/usr/bin/showmount
grepcmd=/usr/bin/grep
mountcmd=/sbin/mount
umountcmd=/sbin/umount
statcmd=stat
touchcmd=/usr/bin/touch
rmcmd=/bin/rm
 
# --- end variables ---
 
# make sure the mountpoint is not stale
statresult=`${statcmd} ${mountpoint} 2>&1 | ${grepcmd} "Stale"`
 
if [ "${statresult}" != "" ]; then
   #result not empty: mountpoint is stale; remove it
   ${umountcmd} -f ${mountpoint}
fi
 
# ping the remote system (2 sec timeout)
remoteping=`${pingcmd} -c1 -o -q -t2 ${remotesystem} | grep " 0.0%"`
 
# make sure the remote system is reachable
if [ "${remoteping}" != "" ] ; then
  
   # query the availability of the remote share; not empty result indicates OK  
   offsiteshare=`${showmountcmd} -e ${remotesystem} | ${grepcmd} "${remoteshare}"`
   if [ "${offsiteshare}" != "" ] ; then
  
      # make sure the local mount point (directory) exists (so that [re-]mount, if necessary, is valid)
      if [ -d ${mountpoint} ] ; then
 
         localmount=`${mountcmd} | ${grepcmd} "${mountpoint}"`
      
         # make sure the share test file is _not_ present (to make sure the mountpoint is inactive)
         if [ ! -f ${testfile} ] ; then
        
            # make sure the local mountpoint is inactive (double checking)
            if [ "${localmount}" = "" ] ; then
 
               # all set to go; request mount
               mount=true
            fi
              
         else
 
            # make sure the local mountpoint is active (double checking)
            if [ "${localmount}" != "" ] ; then
 
               # attempt to create a test file..
               ${touchcmd} ${rw_testfile} > /dev/null  2>&1
 
               # ..and test its existence; first handle RW mounted shares:
               if [ -f ${rw_testfile} ] ; then
 
                  # share was RO requested
                  if [ "$2" = "ro" ]; then
                     remount=true
                  fi
 
                  # Delete the testfile
                  ${rmcmd} ${rw_testfile}
 
               # hanle RO mounted shares:
               else
 
                  # share was RW requested
                  if [ "$2" = "rw" ]; then
                     remount=true
                  fi
               fi
            fi
         fi
      fi
   fi
fi
 
# perform remount (unmount, request mount)
if $remount ; then
   ${umountcmd} -f ${mountpoint}
   mount=true
fi
 
# perform mount when so requested
if $mount ; then
   ${mountcmd} ${mountmode} -t nfs ${remotesystem}:${remoteshare} ${mountpoint}
fi
 
exit 0

You should run the automount script from a runfile, like so:

Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#!/bin/sh
 
NFS_ENFORCE=/usr/local/sbin/nfs_enforcer
 
# Separate the following parameters with spaces:
#
# - nfs enforcer command (set above)
# - remote system name (must be resolvable)
# - read/write (rw) or read-only (ro); NOTE: share may be read-only regardless of how this is set
# - remote share name (from remote's /etc/exports)
# - local mount point (existing local directory)
# - share test file (an immutable file on the share)
 
# e.g.
# $NFS_ENFORCE dbsysvm rw /nfs4shares/conduit /mnt/dbsys_conduit .conduit@dbsysvm
# or (for local remount read-only)
# $NFS_ENFORCE localhost ro /var/web/projects/repository /mnt/rorepo .repository@localhost
 
$NFS_ENFORCE localhost ro /var/web/projects/repository /mnt/rorepo .repository@localhost
 
exit 0

..and call the the above runfile from crontab:

Shell
1
*/10  *  *  *  *  root  /usr/local/sbin/nfs_enforcer.batch > /dev/null

Filed Under: Technical, UNIX Tagged With: automount, enforcer, freebsd, Linux, monitoring, nfs, self-healing, unix

Blog Author, Ville Walveranta

Information Architect, Application Developer, Web Technologist

Social

Follow me on:

StackExchange

profile for Ville on Stack Exchange, a network of free, community-driven Q&A sites

Recent Posts

  • macOS: ‘dig’ vs. ‘dscacheutil’ while using split DNS with Viscosity VPN client
  • Remove DRM Easily (?) from Your Audible Purchases
  • Exploring GitHub Flavored Markdown local preview
  • Interactive AWS CLI Query Filtering with JSONPath
  • Easy MFA and Profile Switching in AWS CLI

Tags

2fa automatic automount aws bash beyond compare boot centos co-lo co-location comparison diff DNS enforcer esxi freebsd ftp fusemail Hardware iam install key Linux mailtrust microsoft monitoring multi-factor nfs RELEASE-7.0 rotation script security self-healing shell software sublime tbe trackball ubuntu unix vista vmware Windows windows 7 workflow

Blog archive

April 2018
M T W T F S S
« Jan    
 1
2345678
9101112131415
16171819202122
23242526272829
30  

PGP/GPG

Keybase: vwal
(PGP Key ID 2E99 86D7 7ED9 9C13)

Copyright © Ville Walveranta 2018 - All Rights Reserved · Powered by Wordpress and Genesis Framework