• My Links at Diigo
  • Ergonomics Resource

RANDOM.AC/CESS

Musings of a Web Technologist

Home Archives for shell

Finding public IP on Linux command line

Posted on 24 August 2012 by Ville Leave a Comment

Here’s a handy command to display the Internet facing IP on a Unix/Linux command line. This is particularly useful on systems where lynx is not available, and where the system might be behind a firewall so that the public IP cannot be discerned from ifconfig output.

Shell
1
curl -s myip.dk | grep 'IP Address' | egrep -o '[0-9.]{3,}+'

Filed Under: Networking, Technical, UNIX Tagged With: bash, ip, Linux, shell, unix

FreeBSD Full / Incremental Filesystem Dump Shell Script

Posted on 17 July 2009 by Ville 12 Comments

I wanted to automate filesystem dumps on my servers running FreeBSD 7.2. After some searching I came across Vivek Gite’s FreeBSD Full / Incremental Tape Backup Shell Script which gave me a lot of ideas. Since I’m not using tape as the backup target I wanted to make a script specifically for that purpose while at the same time improve handling of some error conditions (such as, most importantly, checking for a missing level 0 dump before proceeding with an incremental dump) and add some new features such as autoincrement the dump level so that the dump level is not tied to specific day of the week.

Here’s my version of the script. While it bears some resemblance to Vivek’s script, it is largely rewritten. Read the script header for more information.

NOTE! In his comment James pointed out a possible bug in the script. The displayed script indeed had a problem: it was missing a backslash in front of the first dollar sign at:

eval “local fspath=$${fsname}path”

This was caused by the script display plugin in WordPress that treated the backslash as an escape character (this has now been fixed). To be on the safe side, please download the script as a tarball. To further validate the integrity of the tarball, it should produce a md5 hash of 732ac44f11ba4484be4568e84929bb6a.

Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
#!/bin/sh
 
# Autodump 1.5a (released 01 August 2009)
# Copyright (c) 2009 Ville Walveranta
#
# A FreeBSD shell script to dump filesystems with full, and automatically
# incremented incremental backups to a given directory location; this script
# was written with the intent of saving the filesystem dumps not onto a tape
# device but on another hard drive such as a different filesystem on the same
# computer. The resulting dump files can be copied offsite with a separate
# cron job.
#
# This script creates the necessary directory structure below the defined
# 'BASEDIR' as well as the necessary log file. This script also ensures that
# the level 0 dump exists before creating an incremental dump; if it doesn't
# the script automatically erases the incremental files for the current week
# (if any exist) and starts over with a level 0 dump. This way you can start
# using the script on any day of the week and level 0 dump is automatically
# created on the first run.
#
# When ran daily (such as from a cron job), the script creates level 0 dump
# on every Monday (beginning the ISO week), or Sunday (beginning of the U.S.
# week) and an incremental dump on all the other days of each week. The dumps
# are compressed with gzip and saved below the 'BASEDIR' to an automatically
# created directory whose name is derived from the list given in 'FSNAMES'.
# Each week's dumps are organized into subfolders with name YYYY-WW ('WW'
# being the current week). By default three most recent weekly dumps
# (level 0 + incrementals) are retained.
#
# The script maintains each weekly folder's date at the _beginning_ date
# of the dump (i.e. Monday or Sunday of the current week) at 00:00, not
# at the most recent incremental's date/time.
#
# By default the root (/) and usr (/usr) filesystems are dumped. To add more  
# add a "friendly name" to the 'FSNAMES' list (it is used for the weekly folder
# names, for dump filenames, and to reference the corresponding mount point
# variable); then add the corresponding mount point variable (i.e. if you
# add "var" to 'FSNAMES', then add a variable varpath=/var). The "path"
# ending of the mount point variable name is required.
#
# Since the number of incremental dumps is limited to nine (level 0 +
# incremental levels 1-9), the script will allow maximum of one dump
# to be created per day. However, since the level incrementing is dynamic
# you can start the script on any day of the week, and run it on any
# number of days during the rest of the week and you'll always get
# level 0 plus the incremental dumps in sequential order. However, The
# new weekly folder is always created on Monday or Sunday (as chosen by
# you). Note that the script determines whether "today's" dump exists
# based on the modification date stamp of the most recent dump. Hence
# it is a good idea to run this script in the early hours of each day
# rather than in the very end of each day. Running the script, for
# example, at 23:50 has the potential to push longer dump processes
# over the midnight and so potentially cause the next day's dump to
# be skipped.
#
# Written for FreeBSD 7.2 but should work on most BSD and *NIX systems with
# minor modifications.
# -------------------------------------------------------------------------
# Copyright (c) 2009 Ville Walveranta
# <http://my.galagzee.com/2009/07/17/freebsd-dump-filesystem-shell-script>
# This script is licensed under GNU GPL version 2.0 or above, and is provided
# 'as-is' with no warranty which is to say that I'm not liable if it wipes out
# your hard drive clean or doesn't back up your precious data. However, to the
# best or my knowledge it is working as expected -- I'm using it myself. :-)
# -------------------------------------------------------------------------
# This script was inspired by
# FreeBSD Full / Incremental Tape Backup Shell Script
# by nixCraft project / Vivek Gite
# at <http://bash.cyberciti.biz/backup/freebsd-dump-filesystem-shell-script/>
# -------------------------------------------------------------------------
 
 
#### GLOBAL VARIABLES ###############################################
 
WEEKSTARTS=Mon      # Accepted values are "Mon" (ISO standard) or "Sun" (U.S.)
KEEPDUMPS=30        # in days; this is evaluated on the weekly level per start
                    # of the week, so '30' keeps 3-4 weekly dumps
BASEDIR=/bak/dumps
GLOBALDUMPOPTS=Lua  # add 'n' for wall notifications
LOGFILE=/var/log/dump.log
 
# to add more filesystems to be dumped add the dump name in 'FSNAMES'
# and add the corresponding mount point variable (dumpname+path=mountpoint)
FSNAMES="root usr"  # this is used for dump directory name
                    # and to ID the path from a variable below
rootpath=/
usrpath=/usr
 
#####################################################################
 
DUMP=/sbin/dump
GZIP=/usr/bin/gzip
LOGGER=/usr/bin/logger
 
WEEKDAY=$(date +"%a")
DATE=$(date +"%Y%m%d")
HUMANDATE=$(date +"%d-%b-%Y")
HUMANDATE=`echo $HUMANDATE | tr '[:lower:]' '[:upper:]'`
HUMANTIME=$(date +"%H:%M (%Z)")
TODAYYR=$(date +"%Y")
TODAYMO=$(date +"%m")
TODAYDT=$(date +"%d")
 
# datestamp at midnight today
TODAYSTARTSTAMP=$(date -j +%s "${TODAYYR}${TODAYMO}${TODAYDT}0000")
 
# default lastdump to midnight today; it will be checked
# and and adjusted later
LASTDUMP=$TODAYSTARTSTAMP
 
# do not crete world-readable dumps!
umask 117
 
# make sure the logfile exists
if [ ! -e $LOGFILE ] ; then
   touch $LOGFILE
   chmod 660 $LOGFILE
fi
 
# make sure that entire week's incremental dumps are deposted
# in the same directory, even when a week spans new year
# NOTE: When the ending year has a partial 53rd week, there
# won't be a dump folder for the first week of the new year.
# The incremental dumps instead complete the 53rd week folder,
# even when the 1st week of the new year begins mid-week.
# However, the dates of the incremental dump files in the
# 53rd week folder correctly reflect the dates of the
# beginning year.
adjust_date(){
   local dateoffset=$1
   local epochnow=$(date +%s)
   local offsetsecs=`expr $dateoffset "*" 86400`
   local newepoch=`expr $epochnow "-" $offsetsecs`
   local year=`date -r $newepoch +"%Y"`
  
   if [ "$WEEKSTARTS" = "Mon" ] ; then
      local week=`date -r $newepoch +"%W"`
   else
      local week=`date -r $newepoch +"%U"`
   fi
   NEWEPOCHISO=`date -r $newepoch +"%Y%m%d0000"`
 
   #system week starts from `0', there is no calendar week `0'
   week=`expr $week "+" 1`
   YWEEK=${year}-${week}
}
 
# determines the 'distance' from the level 0 dump in days
if [ "$WEEKSTARTS" = "Mon" ] ; then
   case $WEEKDAY in
      Mon) adjust_date 0;;
      Tue) adjust_date 1;;
      Wed) adjust_date 2;;
      Thu) adjust_date 3;;
      Fri) adjust_date 4;;
      Sat) adjust_date 5;;
      Sun) adjust_date 6;;
      *) ;;
   esac
else
   case $WEEKDAY in
      Sun) adjust_date 0;;
      Mon) adjust_date 1;;
      Tue) adjust_date 2;;
      Wed) adjust_date 3;;
      Thu) adjust_date 4;;
      Fri) adjust_date 5;;
      Sat) adjust_date 6;;
      *) ;;
   esac
fi
 
mk_auto_dump(){
 
   local fsname=$1
  
   # get the current filesystem's path
   # as defined in the corresponding variable
   eval "local fspath=$${fsname}path"
 
   # composite the dump path
   local dumppath=${BASEDIR}/${fsname}/${YWEEK}
 
   # make sure the dump directory for this week exists;
   # this automatically creates a new dump directory on
   # every Monday or Sunday (as selected by 'WEEKSTARTS')
   [ ! -d $dumppath ] && mkdir -p $dumppath
 
   # get name of the last file in the current dump directory
   local lastfile=`ls -ltr $dumppath | grep -v "^d" | tail -n 1 | awk '{ print $9 }'`
 
   # assume that the 'lastfile', if it exists, was not created today
   local dumped_today=false
  
   # if a file exists, check its modification date;
   # if it is at or after 00:00 today, set a flag to skip the dump
   if [ "$lastfile" != "" ] ; then
      local fq_lastfile=${dumppath}/$lastfile
      if [ -e $fq_lastfile ] ; then
         # get the last modification time for the most recently created dumpfile
         LASTDUMP=`stat -f %m $fq_lastfile`
         if [ $LASTDUMP -ge $TODAYSTARTSTAMP ] ; then
            local dumped_today=true
         fi
      fi
      
      # get the first and the last dump level for this directory
      local levelcommand="ls $dumppath | sed -e 's/^[[:digit:]]*_//' | sed -e 's/..*$//'"
      local firstlevel=`eval $levelcommand | head -n 1`
      local lastlevel=`eval $levelcommand | tail -n 1`
 
      # make sure level zero dump exists;
      # if it doesn't, start over
      if [ "$firstlevel" != "0" ] ; then
         # it doesn't matter if a previous dump exists from today
         # since we're starting over as level 0 dump is missing
         local dumped_today=false
         local dumplevel=0
         rm -f $dumppath/*.gz
      else
         # otherwise just increment the dump level
         # for levels 1-6, i.e. normally Tuesday thru Sunday
         local dumplevel=`expr $lastlevel "+" 1`
      fi
   else
      # no dump exists in this week's folder; reset level to '0'
      local dumplevel=0
   fi
 
   # skip the entire dump process if a dumpfile has
   # already been created for this filesystem today
   if [ "$dumped_today" = "false" ] ; then  
  
      # define the dump filename
      local dumpfn=${DATE}_${dumplevel}
 
      echo ---------------- >> $LOGFILE
      echo >> $LOGFILE
      echo BEGINNING LEVEL $dumplevel DUMP OF '$fsname' (${fspath}) FILESYSTEM ON $HUMANDATE AT $HUMANTIME >> $LOGFILE
      echo >> $LOGFILE
      echo Creating a snapshot of '$fspath'.. >> $LOGFILE
      # execute the dump
      $DUMP -$dumplevel -$GLOBALDUMPOPTS -f ${dumppath}/${dumpfn} $fspath >> $LOGFILE 2>&1
      local dumpresult=$?
  
      if [ "$dumpresult" != "0" ] ; then
         # log the dump result to syslog
         $LOGGER "$DUMP LEVEL $dumplevel DUMP OF $fsname (${fspath}) FAILED!"
 
         echo "*** DUMP FAILED - LEVEL $dumplevel DUMP of $fsname (${fspath}) ***" >> $LOGFILE
         echo >> $LOGFILE
      else
         # log the dump result to syslog
         $LOGGER "LEVEL $dumplevel DUMP of $fsname (${fspath}) COMPLETED SUCCESSFULLY!"
 
         echo >> $LOGFILE
         # compress the dump
         echo Compressing the dumpfile '${dumpfn}'.. >> $LOGFILE
         $GZIP -v ${dumppath}/${dumpfn} >> $LOGFILE 2>&1
         echo DONE >> $LOGFILE
         echo >> $LOGFILE
 
         # make sure dumps are not world readable (security risk!)
         echo Updating dumpfile '${dumpfn}.gz' permissions.. >> $LOGFILE
         chmod -v -v 440 ${dumppath}/${dumpfn}.gz >> $LOGFILE 2>&1
         echo DONE >> $LOGFILE
         echo >> $LOGFILE
  
         # reset current dump dir's timestamp to that of the level 0 dump
         touch -t ${NEWEPOCHISO} ${dumppath}
 
         # delete old dumps
         echo Deleting old '$fsname' dumpfiles.. >> $LOGFILE
         find $BASEDIR/$fsname -mtime +$KEEPDUMPS -maxdepth 1 -print -exec rm -rf {} ; >> $LOGFILE 2>&1
         echo DONE >> $LOGFILE
         echo >> $LOGFILE
      fi
   else
      local lastdump_readable=`date -j -r $LASTDUMP +"%H:%M"`
      local lastdump_readableZ=`date -j -r $LASTDUMP +"%Z"`
      local lastdumpmsg="Autodump for filesystem '$fsname' ($fspath) has already been executed today at $lastdump_readable ($lastdump_readableZ)."
      echo $lastdumpmsg
      $LOGGER $lastdumpmsg
   fi
}
 
 
# Dump filesystems defined in 'FSNAMES'
#
# Monday or Sunday (as selected by 'WEEKSTARTS') starts with
# the level 0 dump, with incrementals created through the rest of
# the week (autoincremented). If the level 0 dump is missing in
# the current week's folder for filesystem currently being backed
# up, it is created automatically instead of an incremental dump,
# no matter what day of the week it is.
for f in $FSNAMES
do
   mk_auto_dump $f
done

Filed Under: Technical, UNIX Tagged With: automatic, dump, filesystem, freebsd, incremental, script, shell

Unix Shell: find files by a date range

Posted on 23 February 2009 by Ville 14 Comments

I needed to restore some files from an archive on UNIX, but only the files of a particular date-range were needed.  It took a few moments to find and figure out how I could easily extract files older than a particular date, or files from a particular date-range. This is how:

  1. Create a perimeter file, like so:
    touch -t yyyymmddHHMM marker_date

  2. List files older than the marker_date:
    find . -type f ! -newer marker_date -ls
    Of course, instead of -ls' parameter (to list), you can use -print’ and a pipe to xargs to, for example, delete the selected files, etc.

Likewise, for a range of dates:

  1. Create the perimeter files:
    touch -t yyyymmddHHMM range_start
    touch -t yyyymmddHHMM range_end

  2. List the files between the range dates:
    find . -type f -newer range_start ! -newer range_end -ls

NOTE: For an even easier way to accomplish the same, see comment by Hampus below!

Filed Under: Technical, UNIX Tagged With: files, shell, unix

Blog Author, Ville Walveranta

Information Architect, Application Developer, Web Technologist

Social

Follow me on:

StackExchange

profile for Ville on Stack Exchange, a network of free, community-driven Q&A sites

Recent Posts

  • macOS: ‘dig’ vs. ‘dscacheutil’ while using split DNS with Viscosity VPN client
  • Remove DRM Easily (?) from Your Audible Purchases
  • Exploring GitHub Flavored Markdown local preview
  • Interactive AWS CLI Query Filtering with JSONPath
  • Easy MFA and Profile Switching in AWS CLI

Tags

2fa automatic automount aws bash beyond compare boot centos co-lo co-location comparison diff DNS enforcer esxi freebsd ftp fusemail Hardware iam install key Linux mailtrust microsoft monitoring multi-factor nfs RELEASE-7.0 rotation script security self-healing shell software sublime tbe trackball ubuntu unix vista vmware Windows windows 7 workflow

Blog archive

February 2019
M T W T F S S
« Apr    
 123
45678910
11121314151617
18192021222324
25262728  

PGP/GPG

Keybase: vwal
(PGP Key ID 2E99 86D7 7ED9 9C13)

Copyright © Ville Walveranta 2019 - All Rights Reserved · Powered by Wordpress and Genesis Framework