1 I 44 Lbfpsvivzz 8 Zvqs 9

Unix_tricks.
txt Page 1 of 271

Printed: Tuesday, May 27, 2008 3:20:00 PM
++ Basic UNIX Tutorials

Quick Unix Tutorial (http://unlser1.unl.csi.cuny.edu/tutorials/QuickUnixTutorial.html)
UNIX Tutorial for Beginners (http://www.ee.surrey.ac.uk/Teaching/Unix/index.html)
Basic Introduction to UNIX/linux
(http://rcsg-gsir.imsb-dsgi.nrc-cnrc.gc.ca/documents/basic/basic.html)

++ Intermediate UNIX Tutorials
UNIXhelp for Users (http://www.ee.surrey.ac.uk/Docs/Unixhelp/TOP_.html)
The CERN Unix User Guide (http://consult.cern.ch/writeup/unixguide/unix_2.html)

++ Advanced UNIX Tutorials
Advanced UNIX Tutorial (http://www.engin.umich.edu/caen/technotes/advancedunix.pdf) (150 KB PDF
file)
Unix System Administration Independent Learning (http://www.uwsg.iu.edu/usail/index.html)
Unix System Administration (http://wks.uts.ohio-state.edu/sysadm_course/html/sysadm-1.html)
Administrating Peripherals (http://www.uwsg.iu.edu/usail/peripherals/)

++ C/C++ Programming Tutorials
Introduction to C Programming (http://www.le.ac.uk/cc/tutorials/c/)
Programming in C/UNIX System Calls and Subroutines using C
(http://www.cs.cf.ac.uk/Dave/C/CE.html)
C++ tutorial for C users (http://www.4p8.com/eric.brasseur/cppcen.html)
Compiling "C" And "C++" Programs On Unix Systems - gcc/g++
(http://users.actcom.co.il/~choo/lupg/tutorials/c-on-unix/c-on-unix.html)
Building And Using Static And Shared "C" Libraries
(http://users.actcom.co.il/~choo/lupg/tutorials/libraries/unix-c-libraries.html)
Debugging "C" And "C++" Programs Using "gdb"
(http://users.actcom.co.il/~choo/lupg/tutorials/debugging/debugging-with-gdb.html)
How do I use gdb? (http://www.unknownroad.com/rtfm/gdbtut/gdbuse.html")
RMS's gdb Tutorial (http://www.unknownroad.com/rtfm/gdbtut/gdbtoc.html)
Automating Program Compilation - Writing Makefiles
(http://users.actcom.co.il/~choo/lupg/tutorials/writing-makefiles/writing-makefiles.html)
ANSI C for Programmers on UNIX Systems
(http://www-h.eng.cam.ac.uk/help/tpl/languages/C/teaching_C/teaching_C.html)
Debugging with GDB (http://www-md.fsl.noaa.gov/eft/developer/gdb/gdb.html)

++ Advanced Programming Tutorials
UNIX Systems Programming I & II (http://www.hiraeth.com/alan/tutorials/courses/unixprog.html)
Network programming under Unix systems
(http://users.actcom.co.il/~choo/lupg/tutorials/internetworking/internet-theory.html)
Beej's Guide to Network Programming (http://beej.us/guide/bgnet/)
Getting Started With Posix Threads (http://www.thinkbrown.com/programming/pthreads.pdf)
Multi-Threaded Programming With POSIX Threads
(http://users.actcom.co.il/~choo/lupg/tutorials/multi-thread/multi-thread.html)
Serial Programming Guide for POSIX Operating Systems
(http://www.easysw.com/~mike/serial/serial.html)
Guide to Realtime Programming
(http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/HTML/APS33DTE_html/TOC.HTM)
Which is better, static or dynamic linking?
(http://sunsite.uakom.sk/sunworldonline/swol-02-1996/swol-02-perf.html)
Building And Using Static And Shared "C" Libraries
(http://users.actcom.co.il/~choo/lupg/tutorials/libraries/unix-c-libraries.html)

++ Very Advanced Programming Tutorials
Linux Assembly HOWTO (http://www.tldp.org/HOWTO/Assembly-HOWTO/index.html)
Linux Assembly Tutorial Step-by-Step Guide (http://docs.cs.up.ac.za/programming/asm/derick_tut/)
adb Tutorial (http://docs.hp.com/en/92432-90006/ch01.html)
Writing Device Drivers: Tutorial
(http://www.cs.arizona.edu/computer.help/policy/DIGITAL_unix/AA-PUBVD-TE_html/TITLE.html)
(Digital Unix)
Unix_tricks.txt Page 2 of 271
Device Driver Tutorial (http://docs.sun.com/app/docs/doc/817-5789) (SunOS)

A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux
(http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html)

++ Shell Scripting Tutorials
Sh - the Bourne Shell (http://www.grymoire.com/Unix/Sh.html)
Csh (http://www.grymoire.com/Unix/Csh.html)
Regular Expressions (http://www.grymoire.com/Unix/Regular.html)
Shell Programming (http://www.scit.wlv.ac.uk/~jphb/spos/notes/shell/shell.index.html)
Linux Shell Scripting Tutorial v1.05r3 (http://www.freeos.com/guides/lsst/)
The UNIX Bourne Shell (http://www.csci.csusb.edu/dick/samples/unix.syntax.html)
Bash Reference Manual (http://cnswww.cns.cwru.edu/~chet/bash/bashref.html)
Korn Shell (ksh) Programming (http://www.bolthole.com/solaris/ksh.html)

++ Advanced Shell Scripting Tutorials
UNIX shell scripting with sh/ksh (http://www.dartmouth.edu/~rc/classes/ksh/index.html)
Linux I/O Redirection (http://www.cpqlinux.com/redirect.html)
Advanced Bash-Scripting Guide (http://www.tldp.org/LDP/abs/html/index.html)
Common Desktop Environment: Desktop KornShell User's Guide
(http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/HTML/Dtksh/dtksh_1.html)

++ Perl Tutorials
Perl in 20 pages -- A guide to Perl 5 for C/C++, awk, and shell programmers
(http://quong.best.vwh.net/perlin20/)
Perl Tutorial (http://www.comp.leeds.ac.uk/Perl/start.html) (Perl 4)
Introduction to Perl (http://www.cclabs.missouri.edu/things/instruction/perl/perlcourse.html)
The Perl You Need to Know (http://www.wdvl.com/Authoring/Languages/Perl/PerlfortheWeb/toc.html)
(Perl 4, aimed towards web/cgi)
introduction to Perl 5 for web developers (http://www.extropia.com/tutorials/perl5/toc.html)
An Introduction to Perl's XML::XSLT module
(http://www.linuxfocus.org/English/July2000/article156.shtml)

++ Version Control Systems
Small tutorial on SCCS (http://parand.com/docs/sccs.html)
SCCS Tutorial (http://www.eng.mu.edu/corlissg/229.S02/how_sccs.html)
Very Quick RCS Tutorial (http://galton.uchicago.edu/~gosset/Compdocs/rcs.html)
Revision Control System (http://www.csc.calpoly.edu/~dbutler/tutorials/winter96/rcs/)
Sys Admin File Revision Control with RCS (http://www.samag.com/documents/s=1184/sam9812a/)
GNU RCS Tutorial (http://www.uvm.edu/~ashawley/rcs/tutorial.html)
Concurrent Versions System (http://www.csc.calpoly.edu/~dbutler/tutorials/winter96/cvs/)
CVS Tutorial (http://wwwasd.web.cern.ch/wwwasd/cvs/tutorial/cvs_tutorial_toc.html)
CVS Tutorial (http://www.star.bnl.gov/STAR/sds_l/ssd_l/cvs_tutorial.html) (nearly identical to
above link)
Open Source Development with CVS, 3rd Edition (http://cvsbook.red-bean.com/cvsbook.html) (An
on-line book)

++ Individual Command Tutorials
Mastering the VI editor (http://www.eng.hawaii.edu/Tutor/vi.html)
List of Commands for vi - An Unix Editor (http://www.chem.brown.edu/instructions/vi.html)
Using find to locate files
(http://www.kingcomputerservices.com/unix_101/using_find_to_locate_files.htm)
Find (http://www.grymoire.com/Unix/Find.html)
Using grep, fgrep, and egrep to search for strings of words
(http://www.kingcomputerservices.com/unix_101/grep_this.htm)
Getting started with awk (http://www.cs.hmc.edu/tech_docs/qref/awk.html)
An Awk Tutorial (http://www.vectorsite.net/tsawk.html)
Awk (http://www.grymoire.com/Unix/Awk.html)
sed tutorial (http://www.selectorweb.com/sed_tutorial.html)
Sed - An Introduction (http://www.grymoire.com/Unix/Sed.html)
A collection of sed tutorials (http://sed.sourceforge.net/grabbag/tutorials/)
How to set up sar (http://thproxy.jinr.ru/file-archive/doc/sun/sarhowto.htm)

++ X Windows Tutorials
Introduction To The X-Window System (http://riceinfo.rice.edu/Computer/Documents/Unix/unix2.pdf)
(102 KB PDF file)
X Windows/Motif Programming (http://www.cs.cf.ac.uk/Dave/X_lecture/X_book_caller/)

++ Networking Tutorials
An Overview of TCP/IP Protocols and the Internet (http://www.garykessler.net/library/tcpip.html)
Networking Tutorial (http://www.comptechdoc.org/independent/networking/guide/)
Using Samba (http://www.faqs.org/docs/samba/toc.html)
HTML Code Tutorial (http://www.htmlcodetutorial.com/)
An Introduction To HTML (http://www.vectorsite.net/tshtml.html)

++ Other Tutorials
Computer Numbering Formats (http://www.vectorsite.net/tsfloat.html)
Using Java, 2nd Edition (http://www-sbras.ict.nsk.su/rus/docs/java/sej2e/httoc.htm)
SQL Tutorial (http://www.thinkbrown.com/programming/sql_tutorial.pdf)
Celeste's Tutorial on Solaris 2.x Modems and Terminals
(http://www.sunhelp.org/stokely/2xmodem.html)
thesitewizard.com -- Website design, promotion, programming and revenue making
(http://www.thesitewizard.com/)
ProgrammingTutorials (http://www.programmingtutorials.com/default.aspx)
Tutorial Maniacs (http://www.tutorialmaniacs.com/) (Mostly Windows tutorials)
Programming Texts and Tutorials (http://stommel.tamu.edu/~baum/programming.html) (The Mother
Lode)

locate a file

To quickly find a specific file, or group of files, use the 'locate' command.

Find all files on your Mac called 'my_letter', or which have 'my_letter' as a part of their name:

locate my_letter

Find all files with an extension of '.txt':

locate .txt

'locate' does not search the file system, but uses a database of file names so the search is very
quick.

update locate database

The 'locate' command searches a file name database that is updated at 4:30 am every Saturday
morning (if your Mac is switched on!). Files created more recently than the last update will not
be found. Deleted files will be erroneously found.

To manually update the data base issue the following command:

sudo /usr/libexec/locate.updatedb

giving your administrator password.

find a file

To search for a specific file use the 'find' command. 'find' does a live search on the file
system, unlike 'locate' which searches a database.

Find all files in your home directory called 'my_letter':

find ~ -name my_letter

To search the current directory instead, replace '~' with '.'. To search the entire file system
use '/'.

find a group of files

To search for a group of files with a common prefix or ending, use 'find' with wildcards in the
filename. Enclose the name in quotes.

Find all files with extension text in the current directory:

find . -name "*.txt"

Find all files starting with 'temp' searching the whole file system:

find / -name "temp*"

find big and small files

The 'find' command can be used to search for files equal to, greater than, or less than a given
size. The size is given as the number of 512 byte blocks. If preceded with '+' this means greater
than the given size, '-' means less than.

To find all files in your home directory bigger than 1 Meg use:

find ~ -size +2048

The size 2048 is obtained from: 1 Meg divided by the block size of 512.

To find all empty files in the current directory use:

find . -size -1

man basics

If you are not sure how to use a Unix command (such as 'ls') read its manual page using:

man ls

And to find out how to use 'man' itself...

man man

The manual contains other useful information besides commands - try these:

man hier
man ascii

search by keyword

If you need to do something but cannot think of a Unix command that might help, locate a suitable
command by keyword lookup. For example, commands that may change the owner of a file can be
located with either:

man -k owner
or
apropos owner

the 8 man sections

The Unix manual is divided into 8 sections:

* (1) user commands
* (8) system commands
* (5) configuration Files

less useful:

* (6) terminal games - empty :-(
* (7) miscellaneous
* (2,3,4) library calls for programmers

Find out about each section with:

man <section number> intro

Unfortunately, 'intro' is lacking detail in OS X. Instead, list section commands with (for
section 1):

man -k "(1)"

'files' and 'see also'

When reading a manual page, look at the 'FILES' and 'SEE ALSO' sections at the end. 'FILES' lists
configuration files associated with the command. 'SEE ALSO' lists similar commands and is a good
way to learn more Unix commands.

Often, information about the format of a file listed in 'FILES' can be got by:

man name-of-file
or
man 5 name-of-file

the man pager

The environment variable 'PAGER' tells 'man' the command it should use to display its output. The
default is 'more'. Read the manual page for 'more' in order to find out how to better control the
output from 'man'.

If you wish to use a different pager, such as 'less', set the environment variable with:

setenv PAGER less

To get 'man' to write its output to a file, and also remove the formatting, type:

man the-command | col -b > a-file

star and query

So what is globbing? Globbing is where you specify an ambiguous filename and the shell finds all
those files (in the given directory) that match it.

The globbing character '*' tells the shell to match 0 or more characters - any character will
match.
rm *

will remove ALL files in the current directory. Actually, that's not quite true. A leading '.'
must be matched explicitly, so filenames that begin with dot must be removed with 'rm .*'.
rm *.txt

will remove all files with the extension '.txt'

The character '?' tells the shell to match any one character.
rm test?

will remove 'test1', 'testa', and 'test$'. It will not remove 'test' or 'test12'.

selected characters

Enclose characters in '[ ]' to tell the shell to match any one of a specific list of characters.
ls b[aeiou]g

will list 'bag', 'beg', 'big', 'bog', and 'bug'; but not 'bfg' or 'bags'.

The shell processes matched filenames in alphanumeric order.

ranges

To match any one of a range of characters, separate a pair of characters with '-'.

[a-z] matches any lowercase letter.

[0-9] matches any digit.

[a-z0-9] matches any lowercase letter or digit.

glob patterns may be combined, as in:
ls [mt]*day[0-9]

Match filenames that start with 'm' or 't', have any number of intervening characters, and end in
'day' followed by one digit.

escaping

If you wish to pass any of the characters * ? [ ] to a command, you must prevent the shell from
globbing them first by escaping those characters. Use either quotes or backslash.

To pass '*' to the 'find' command, use:
find . -name "*.html"
find . -name '*.html'
find . -name \*.html

Escaping is also used to reference a filename that includes spaces.

pico "my file"
pico 'my file'
pico my\ file

tilde

The tilde character is also 'globbed' - into your home directory:

ls ~
is equivalent to
ls /Users/your-short-name

and for other users:

ls ~janice
is equivalent to
ls /Users/janice

cat

Use your feline friend to display the contents of a file to a Terminal window.
cat todo.txt

If 'todo.txt' is big, the upper portion will scroll off the top of the Terminal window. Use the
page up and page down keys, or the scroll bar, to view the whole of the file.

Paging like this makes use of the Terminal's scroll-back buffer. You can increase the size of the
scroll-back buffer in Terminal preferences.

If the file contains control and other non-displayable characters, use:
cat -v odd-file.txt

more

If the file you wish to view is bigger than a Terminal screen-full, use 'more' instead of 'cat'.
more todo.txt

The last line of output shows some inverse text giving the filename and how far you are through
viewing the file.

Press:

return to advance one line
d to advance one half page
space to advance a full page

nf to move forward n pages
nb to move back n pages
(miss out 'n' for a single page)

/pattern to skip through the file until 'pattern' is found

q to quit 'more'

less

Better than 'more' is 'less'. To view a file with less:
less todo.txt

The commands are as for 'more', plus many others. Most of the navigation commands from the text
editor 'vi' will work too.

Use the cursor (arrow) keys to move back and forth through a file, and to view a wide file.

Press:

control-g to display file status information

ng to go to line n
G to go to the end of the file
n% to position n% into the file

/pattern to search forward for 'pattern'
?pattern to search backwards for 'pattern'
n to repeat the last search

All occurrences of a match pattern are highlighted.

Place bookmarks and return to them at any time (exactly as for 'vi') with:

ma to mz to place a bookmark
'a to 'z to return to a bookmark

heads or tails

To view the first few lines of a file use 'head'.
head todo.txt

To view the last few lines of a file use 'tail':
tail todo.txt

To view the first or last 'n' lines of a file:
head -n 20 todo.txt
tail -n 15 todo.txt

chase your tail

To view a file live (as it is updating) use option '-f' with 'tail'.
tail -f todo.txt

will print the last few lines of 'todo.txt', then monitor the file printing out additional lines
as and when they are added. This is useful to monitor progress and log files.

The command:
tail -f /var/tmp/console.log

will watch the console log. The application Console in Utilities is simply a GUI front-end to
this command.

Press 'control-c' to quit 'tail'.

directory sizes

These tricks apply equally to 'disks' and 'discs' :-)

To report on how much disc space a directory occupies, use the disc usage command 'du'. To query
your home directory for example:
du ~

This will display the size of every directory from your home directory down. The size of a
directory includes the size of all directories within it. The last line displayed will be the
total size of your home directory.

The size is reported in file-system blocks. Each block is 512 bytes or 1/2k. Take the answer you
get and divide by 2 to get the size in k-bytes.

summary of directory sizes

To report on how much disc space a directory occupies, but without displaying that information
for every sub-directory, use the '-s' or summary option.
du -sk ~

Will print a single line displaying the total size of your home directory, and in k-bytes too
(requested by the '-k' option).

Specifying option '-x' tells 'du' to avoid mounted file systems.

summary of disc usage

To display how full each disc and partition is, use the 'df' commnd:
df -k

Option '-k' causes sizes to be displayed in k-bytes.

tar balls

A 'tar ball' is Unix-speak for a single file into which many other files have been bundled, or
archived - similar to an Aladdin 'stuffit' file or a disc image.

To archive 'Documents' use:
tar cvf Documents.tar Documents

'Documents' remains untouched. Documents.tar is created and is a single file that contains a copy
of all the directories and files in Documents.

To list the contents of an archive:
tar -tf Documents.tar

To expand an archive and reconstruct the original directory:
tar -xvf Documents.tar

zipping up files

To compress a file use 'gzip'. This is the GNU implementation of the Lempel-Ziv coding used in
the 'zip' family of commands.

To compress the archive we created on Thursday:
gzip Documents.tar

This will overwrite Documents.tar with Documents.tar.gz.

Documents.tar.gz is very much like an Aladdin stuffit archive. A single compressed file that can
be decompressed and expanded.

To decompress use:
gunzip Documents.tar.gz

This will overwrite Documents.tar.gz with Documents.tar.

Sometimes gunzip will give an error message something like:
'...more than one file...'

In this case try using 'unzip'.

'tar' can archive and compress, or decompress and expand, in one go. Use the 'z' option to get
'tar' to zip and unzip.
tar cvfz test.tgz test
tar xvfz test.tgz

Unix Calendar

To display a calendar for any month or year use the 'cal' command.

This month:
cal

This year:
cal -y

April 1900:
cal 4 1900

Year 2112:
cal 2112

Unix Calculator

To perform complex calculations use the calculator 'bc'. The results from 'bc' are shown in
italic in the following examples.

2^64
18446744073709551616

365*24*60*60
31536000

42*(6+9)
630

scale=5
1/3
.33333

scale=100
22/7
3.14285714285714285714285714285714\
2857142857142857142857142857142857\
1428571428571428571428571428571428

a=20
b=60
a*b
1200

++a*++b
1281

And this is just the tip of the 'bc' iceberg.

Unit Conversions

To find out how to convert from anything to anything use the Unix command 'units'. In the first
example I wish to convert between feet and metres (meters), and so I type 'feet' then 'metres'
(or 'meters').

units
You have: feet
You want: metres
* 0.3048
/ 3.2808399

You have: kilo-litres
You want: gallons
* 264.17205
/ 0.0037854118

You have: zetta-ergs
You want: joules
* 1e+14
/ 1e-14

You have: scruple
You want: grams
* 1.2959782
/ 0.77161792

control-c to exit.

Create Banners

Create a short banner with the Unix command 'banner'.

banner hello

# # ###### # # ####
# # # # # # #
###### ##### # # # #
# # # # # # #
# # # # # # #
# # ###### ###### ###### ####


Quiz Time

Today, a quiz.

How many days were there in September 1752? And why?

How many days into 2002 are we today?

How many pico-gallons in a bushel?

Copy to the Clipboard

Copy the output of a Unix command to the GUI clipboard (that to which one cuts, copies, and
pastes).

On the command line type (for example):

ls -al | pbcopy
or
cal | pbcopy

In a GUI app such as TextEdit select menu Edit->Paste, or command-v, to paste the output copied
from the Unix command.

August 2002
S M Tu W Th F S
1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

Paste from the Clipboard

Copy from the GUI clipboard into a Unix command.

Copy a page from TextEdit using command-c. Now paste it to the Unix command 'wc' ('wc' counts the
number of words in its input).

pbpaste | wc -w
348

Note that Mac GUI programs and Unix programs use a different end of line character, so the
clipboard will appear to contain one long line.

Drag and Drop

Drag and Drop onto the command line.

Change your current working directory to a folder you can see in the Finder without typing the
full pathname. Simply type:
cd <space>

Now select the destination folder and drag and drop in anywhere on the Terminal window, then
press return.

This works for files as well as folders.

Launch GUI Applications

Launch a GUI application, or open a document, from the command line.

To launch an application use 'open -a'.
open -a /Applications/Clock.app/Contents/MacOS/Clock

This can also be done remotely via ssh, as long as the 'ssh' user and the user logged into GUI on
the remote machine are the same. Guaranteed to surprise the user of the remote machine :-)

To open documents, for example all text files in ~/Documents, type:

cd ~/Documents
ls *.txt
test.txt test2.txt
open *.txt

Both files will open in the default application for '.txt' - probably TextEdit.

Double Click a Shell Script

Run a shell script by double-clicking it in the GUI.

Create a shell script as usual, and give it the extension '.command'. Simple as that. Now
double-click it and a Terminal window will open, which automatically runs the script.

Remember that the command must be executable. Use:
chmod +x script-name.command

Terminal Startup

When you start the Terminal, or open a new Terminal window, the new shell executes commands from
(among others) ~/.tcshrc.

To display a friendly and personalised greeting on startup, add the following line to '.tcshrc'
in your home directory.
echo Hello $user

Use an editor such as pico to create this file, or use the following command:
echo 'echo Hello $user' >> ~/.tcshrc

Also, have a look at /etc/motd.

Customised Prompt

Change the Terminal prompt.
set prompt="%B%c4%b %d %D %w %t %n@%M\n%# "

This gives a two-line prompt:
~/SitesTutorials Wed 21 Aug 5:37pm
%

Pick and mix these.

%B %c4 %b is bold on, display current directory up to 4 levels, bold off.
%d %D %w %t is day, date, month, time
%n %M \n %# is user name, host name, new line, standard prompt

Add your custom prompt to '.tcshrc' - see Monday's trick. Use an editor or issue the command:
echo 'set prompt="%B%c4%b %d %D %w %t %n@%M\n%# "' >> ~/.tcshrc

Customised Title Bar

Terminal preferences allow one to change what is written to the Title Bar of the Terminal window.
This can also be done on the command line, which offers more flexibility.

Issue the command:
echo "^[]0;Hello ${user}@${host}"

to change the Title Bar to 'Hello user-name@host-name' - for example:
Hello janice@melkor

Note that ^[ is obtained by pressing control-v followed by the 'esc' or escape key.
Alternatively, one can use printf as in:
printf "\033]0;Hello ${user}@${host}\n"

Add your custom prompt to '.tcshrc' - see Monday's trick. Use an editor or issue the command:
echo 'printf "\033]0; Hello ${user}@${host}\n"' >> ~/.tcshrc

Use ANSI Colours

Change the colour of your prompt.
set prompt="%{\033[31m%}%c3%{\033[0m%} %M %# "

This sets a prompt like:
~/Sites/Tutorials melkor %

The sequence
%{\033[#;#m%{

sets the ANSI colour or attribute specified by # (or several attributes can be specified by
#;#;#...).

Possible values for # are:

Attributes:
0 reset, 1 bold, 4 underline 5 flashing 7 inverse
Text:
30 black, 31 red, 32 green 33 yellow
34 blue, 35 magenta, 36 cyan, 37 white
90-97 are lighter variants of the above colours
Background:
40-47, colours as for text
100-107 are lighter variants

Enable Filename Correction

The shell auto-corrects mistyped command names. It can also do this for filenames as in the
following example.

ls LearningCanter
ls: LearningCanter: No such file or directory

set correct=all
ls LearningCanter

OK? ls LearningCenter? yes
AdvancedUnix UnixTutorials ...

Let's do the grep

Sherlock allows one to search the contents of files for a specific string. One can do this on the
command line too.

Use 'grep' to find all files in you home directory containing a given string.
grep -nis the-string-you-wish-to-search-for *

The option 'n' prints a line number, 'i' ignores case, and 's' suppresses warnings.

grep = Get Regular ExPression.

Let's do the Recursive grep

One can search a whole directory structure for a given string using recursive 'grep'

Use 'grep' to find all files in you home directory, and all directories within, by specifying
option '-r'.
grep -nisr the-string-you-wish-to-search-for *

Searching for Words

'grep' can be made to search for a whole word, rather than text that is contained within a word,
by specifying option '-w'.

For example:
grep -nisw day *

will find matches for 'day' as a whole word, but not 'day' as in 'today' or 'Wednesday'.

Searching for Patterns

Use grep to search files for text that matches a pattern rather than a fixed string.

To search for Saturday or Sunday but no other days use:
egrep -nis "(Satur|Sun)day" *

Note that we must use 'egrep', which is an extended version of grep, for this particular example.
If in doubt use egrep all the time.

(e)grep can do all sorts of fancy pattern matching, using what are termed 'regular expressions'
These are quite powerful and beyond the scope of this Trick, but will be covered in depth in the
Learning Center. If you speak 'vi' then you can use (mostly) the same regular expressions in
(e)grep.

Word Counting

Count the number of words (and characters and lines) in a document. The document should be plain
text.

wc letter.txt
16 138 882 letter.txt

This letter has 16 lines, 138 words, and 882 characters.

Substitute with sed

If you wish to replace a particular string with another throughout a file use 'sed'.

To replace the occurrence of 'colour' with 'color' from 'old.html' and write the output to
'new.html' use:
sed 's/colour/color/g' old.html > new.html

's' stands for substitute. The 'g' at the end of the substitute pattern says to substitute all
occurrences of colour on each line, not just the first.

To write the processed file back to the original file use:
sed 's/colour/color/g' file.html > tmp; mv tmp file.html

sed = Stream EDitor

Strip with sed

Use 'sed' to strip out unwanted lines in files.

To remove lines containing the text DELETE-ME:
sed '/DELETE-ME/d' x > x.new

To remove all blank lines:
sed '/^$/d' x > x.new

Strip with grep

Use 'grep' to strip lines from files.

By specifying option '-v' 'grep' will print all lines that don't match the specified expression.
Therefore to strip all lines that contain 'rem' use:
grep -vw "rem" x > x.new

Use option '-w' to match 'rem' as a whole word.

Translate with tr

If you have a file with Mac style end of line character <CR> that you want to look sensible in a
Unix editor, you must translate all occurrences of <CR> to the Unix end of line <NL>.

Use:
tr \\r \\n <x >x.new

to translate x and write the output to x.new.

Strip with col

Use the 'col' command to strip backspace characters from files. For example, capturing a man page
with:
man grep > grep.txt

will produce odd-looking output.

Use:
man grep | col -b > grep.txt

Hardware Profile

Get a hardware profile of your Mac with the command line profiler.

Type:
AppleSystemProfiler

What on What?

Obtain the Unix variant and processor name.

Type:
uname
machine

Network Id

Query the hostname of your Mac with:
hostname

And set it with:
hostname my-new-host-name

Display the domain name with:
domainname

(which may be blank.)

Display interface configuration with:
ifconfig -a

Keepy-Uppy

How long has your Mac been up and running? Find out with:
uptime

Let's Date

To query the current date and time use:
date

'date' can also be used to set the date and time with:
sudo date 200208222355.08

2002 August 22 23hrs 55mins 8 seconds

This command must be run as superuser and has the format: yyyymmddhhmm.ss

You may omit as much of the leading section as you wish.

The command 'time' does not display or set the time, it times execution of the command that
follows it.

Teleporting

Shell aliases are not the same as Finder aliases. A shell alias is shorthand for a command.

If you have a directory to which you often refer, make an alias of it.
alias cdut cd ~/Sites/Tutorials/LearningCenter/UnixTutorials

Now type:
cdut

to change directory to ~/Sites/Tutorials/LearningCenter/UnixTutorials

View Aliases

To view the aliases currently set type:
alias

To view a particular alias type:
alias alias-name-here

To remove an alias type:
unalias alias-name-here

When you create an alias, make sure it does not have the same name as an existing command with:
which your-new-name

Useful Aliases

When you start the Terminal, or open a new Terminal window, the new shell executes commands from
(among others) ~/.tcshrc.

To have aliases set automatically add them to '.tcshrc' in your home directory. Here are a couple
of examples:
alias xps "ps auxc | grep -i \!:1"

'xps' will now print the process id (PID) for a given command.

xps OmniWeb
melkor 965 0.0 7.0 157072 59540 ?? S 79:23.50 OmniWeb

For an easy grep:
alias xgrep "egrep -ilrs \!:1 *"

'xgrep' will search all files in the directory hierarchy for the given string, case insensitive,
printing a list of files that match the string.

xgrep osxfaq

~ melkor % xgrep osxfaq
Development/.FBCIndex
Library/Addresses/Address Book.addressbook
Library/Application Support/AOL Instant Messenger (SM)/Buddies Cache
Library/Application
Support/Chimera/Profiles/default/eb0lqzsx.slt/cookies.txt
Library/Application Support/Chimera/Profiles/default/eb0lqzsx.slt/cookperm.txt
...

Periodic Aliases

Run any command periodically.

The alias 'period' is run every 'tperiod' minutes. For example:
alias periodic date
set tperiod=30

will execute the 'date' command every 30 minutes.

Shell Special Aliases

Special aliases are executed by the shell at specific times.

'precmd' runs just before each prompt is printed.
'postcmd' runs before each command gets executed.
'shell' specifies the interpreter for executable scripts which do not themselves specify an
interpreter.

For example:
alias precmd 'printf "\033]0; `pwd` $host@$user xxx\007"'

will update the Terminal window title bar with the current directory and the host and user name
each time you issue a command (in particular 'cd').

Recently Modified Files

Find is able to search files dependant on their time-stamp.

To find all files in your home directory modified within the last 24 hours, for example to do a
daily backup, use:
find ~ -mtime 1

Find This or That

Find has a very powerful syntax and can combine criteria.

To find all HTML and WebSiphon (.ws) files in ~/Sites use:
find ~/Sites $ -name "*.ws" -or -name "*.html" $


Find and Exec

Use the power of find to root out the files you are interested in, and then apply a given command
to each file found.

For example, to search all HTML files in your Sites directory, use the -exec option to invoke
grep.
find . $ -name "*.htm" -or -name "*.html" $ -exec grep -il warning {} \;

Search

Make an alias of the find command to do common searches.

For example to search for strings in text files:
alias ftxt 'find . -name "*.txt" -exec grep -il \!:1 {} \;'

Then type:
ftxt hello

to find all .txt files containing hello.

Ensure that the aliases you create do not have the same name as existing commands.

Restore the 10.1 Shell Defaults

For whatever reason, the default shell initialization that came with Mac OS X 10.0 and 10.1 is
included but not activated in 10.2.

If you notice that your Terminal (or more correctly the shell) does not behave as it used to, and
you would like the behaviour restored, type:
cat /usr/share/tcsh/examples/README

and follow the instructions.

It is not necessary to follow this part:
mkdir ~/Library/init/tcsh

as you can add your own customisations to the files you have just created:
~/.tcshrc
~/.login
~/.logout

(Note that ~ or tilde is shorthand for your home directory.)

Some Useful Settings

If you do not wish to follow Monday's tip, you can restore a few of the more useful shell
features missing in 10.2 by adding these commands to the tcsh initialization file '~/.tcshrc'.
set autolist # List autocomplete alternatives when tab pressed
set correct = cmd # Correct misspelled commands
set savehist = 100 # Save command line history when Terminal closed

As an example of the first two settings:
cd ~
ls D <hit the tab key>
Desktop/ Development/ Documents/

mun ls
CORRECT>man ls (y|n|e|a)?
...

Type:
set

to see all the shell's variables.

Change History

Be default, the size of your command line history is 100 or 150 lines. You can increase this with
the commands:
set history = 2000 # History remembered is 2000 commands.
set savehist = (2000 merge) # Save and merge with existing saved commands.

These should be added to your tcsh initialization file '~/.tcshrc'.

This saves the last 2000 commands. Notice the merge option, this is very useful if you have more
than one Terminal window open at a time. Instead of the last Terminal session overwriting the
history saved by the others, history is merged in with previously saved history.

So Insensitive

By default, command and filename completion by the tab key is case sensitive. To make it non case
sensitive add the following to your tcsh initialization file '~/.tcshrc':
set complete = enhance # Complete case insensitive.

Enhance also considers '-' and '_' to be equivalent.

By way of an example:
cd ~
ls d <hit the tab key>
Desktop/ Development/ Documents/

The Path Home

You may not realise this, but typing, for example:
cd /
cd Desktop

will find the directory Desktop even though it is not in the current directory. 'cd'
automatically searches your home directory if it cannot find the specified directory. It does
this because (or if) the 'cdpath' variable is set. If this does not work for you try:

set cdpath = (~)

You can add more directories to this search path with:

set cdpath = (~ ~/Sites)

If 'cd' cannot find the given directory in the current directory, it will search your home
directory (~), then 'Sites' in your home directory.
cd /
cd data-base

goes to
~/Sites/data-base

Note that autocompletion does not follow the 'cd' search path.

Auto Complete in Color

The 'tcsh' shell has a built-in 'ls' that it uses for auto-completion. This can be made to list
in colour by setting the shell variable 'color'
set color

Now type:
ls <now hit control-d or tab if you have autolist enabled>

(You must type a space after the 'ls')

The auto completion will now be shown in colour.

If tab completion does not work for you, set the autolist variable:
set autolist

List in Color

Following on from the previous tip, one can use the 'tsch' built-in colour 'ls' instead of the
regular 'ls'. Remember to set the shell variable 'color'.

Type (with no spaces):
ls-F

You may want to make an alias to avoid typing the '-F' part:
alias ls ls-F

and put this line in your ~/.tcshrc file.

Changing Color

If you wish to change the colors that 'ls' and auto-completion use, set the shell environment
variable LS_COLORS.

As an example, add this, as one long line with no spaces, to your ~/.tcshrc file.
setenv LS_COLORS 'no=00:fi=00:di=00;33:ln=04:ex=32:
*.sh=32:*.tar=31:*.tgz=31:*.gz=31:*.sit=31:
*.jpg=36:*.jpeg=36:*.gif=36:*.psd=36:*.html=35:*.htm=35:*.css=35:'

The first line defines file type versus colours; the second and third file extensions versus
colours. You can customise this as you wish.

File types are:

no Normal (non-filename) text
fi Regular file
di Directory
ln Symbolic link
pi Named pipe (FIFO)
so Socket
bd Block device
cd Character device
ex Executable file
mi Missing file
or Orphaned symbolic link

Colours are:

0 default Terminal colours
1 brighter colour
4 underline
5 flashing
30 black foreground
31 red foreground
32 green foreground
33 dark yellow foreground
34 blue foreground
35 magenta foreground
36 cyan foreground
37 dark white (grey) foreground
40 black background
41 red background
42 green background
43 dark yellow background
44 blue background
45 magenta background
46 cyan background
47 dark white (grey) background

Shut Up!

If you use auto-completion frequently then the shell's constant beeping may annoy you, especially
if you have the sound turned up while listening to iTunes. Set the shell variable matchbeep to
stop this.
set matchbeep=never

Add this to your ~/.tcshrc file to make the change permanent.

An alternative is to set it to beep only when no match can be found:
set matchbeep=nomatch

Really Shut Up!

To tell the shell never to beep issue the command:
set nobeep

(This is not the same as the Terminal Window Settings --> Emulation - no beep option, which stops
all programs run within the Terminal from beeping.)

ditto

You may be aware that Unix commands do not understand Mac resource forks, and so commands like
'cp' cannot be used to copy files that have them.

The command 'ditto' does. It can do a recursive copy of an entire directory structure. It does
this as a 'carbon copy' of the directory, preserving owners, permissions, timestamps, etc. It's
an ideal way to backup the /Users directory.

The command:
sudo ditto -rsrc /Users/ /Volumes/Backup/2002.10.16/Users

will do just this, copying the /Users directory to a disc or partition called 'Backup'

If you have the developer tools installed, there is also a command called CpMac that does a
similar thing.

Setting HFS+ File Attributes

Unix commands do not understand all the attributes of HFS+ files. (HFS+ is the default Mac file
system. The native Unix file system is called UFS - no prizes for guessing what that stands for).

The command:
SetFile

will set the attributes, creator codes, timestamps, and type of a file on an HFS+ formatted disc.
This command is part of the Developer Tools so it won't be available unless you have installed
them.

Type:
man SetFile

for full details.

For example:
SetFile -a V file-name

will make a file called 'file-name' invisible in the Finder. Note that it is still visible to
'ls', but the Finder will hide it. The Finder will not update its information on the file
immediately; so don't expect the file to vanish instantly.
SetFile -a v file-name

will make it visible again.

Bless You!

To Change start up discs from the command line use 'bless', and run the command as root or with
'sudo'.

To set the start up disc to OS 9:
sudo bless -folder9 '/Volumes/Mac OS 9/System Folder' -setOF

And for OS X:
sudo bless -folder '/System/Library/CoreServices' -setOF

or to boot from another disc:
sudo bless -folder '/Volumes/volume-name-here/System/Library/CoreServices' -setOF

AppleTalk

Here are a few useful AppleTalk command line utilities.

appleping - like 'ping' but for AT rather than TCP/IP
atlookup - lists visible devices registered with AT
atstatus - give status information for At devices like printers
at_cho_prn - select a printer to send to via AT
atprint - print to an AT printer
appletalk - configure AT

Kill All!

'killall' is new in OS X 10.2, but not specific to OS X.

This will kill a process by name. The good old 'kill' command requires the PID (Process ID).

For example:
killall Clock

does the same job as:

ps -cx
PID TT STAT TIME COMMAND
170 ?? Ss 2:50.85 ATSServer
182 ?? Ss 19:37.36 Window Manager
...
1493 ?? S 0:00.49 Clock
813 std S 0:01.05 tcsh

kill 1493

Remember that Unix is case-sensitive, so 'killall clock' will not work.

User ID and Group ID

To display a user's name and primary group, followed by all the groups to which that user
belongs, use 'id'. This displays both the numeric ID and textual name.

Use:
id
uid=501(saruman) gid=20(staff) groups=20(staff), 0(wheel), 80(admin)

for the currently logged in user.

Use:
id root
uid=0(root) gid=0(wheel) groups=0(wheel) 1(daemon) 2(kmem) 3(sys) 4(tty) 5(operator) 20(staff)
31(guest) 80(admin)

for a given user, eg root.

Note that admin users pre 10.2 will belong to groups as above. Admin users created in 10.2 do not
belong to group wheel

Use:
groups
staff wheel admin

to list just the names of groups to which a user belongs.

Or:
groups root
wheel daemon kmem sys tty operator staff guest admin

for a given user.

List Current Users

To list the names of all users currently logged in, user 'users'. Just to check no one is
sneaking onto your Mac.
users
saruman evil-cracker

You will only see genuine logins, not ftp users, or users started with 'su'.

Hopefully you will be the only user listed.

Who is Logged in?

Use 'who' to determine who is logged into your Mac. This differs from 'users' in that it lists
each Terminal connection, not each user. A user connected to three Terminal sessions will be
listed three times by 'who', but only once by 'users'.

who
saruman console Oct 13 16:36
saruman ttyp1 Oct 27 21:38
saruman ttyp2 Oct 27 23:33

saruman is connected 3 times, once as the original login when the machine boots up (console), and
twice in the Terminal (ttyp#).

You can also use the command 'w', which saves a whole two characters of typing and gives a little
extra information.

w
12:01AM up 14 days, 8:26, 2 users, load averages: 1.49, 1.19, 1.03
USER TTY FROM LOGIN@ IDLE WHAT
saruman co - 13Oct02 14days -
saruman p2 - Sun11PM 0 -

It also gives the information provided by 'uptime', telling how long your machine has been up and
running between re-boots.

Who Am I?

Severe case of amnesia?
whoami
saruman

Useful if you 'su' to other users.
whoami
saruman
su janice
Password:
whoami
janice

To get a history of logins since quite a long time ago (the last time your Mac tidied its log
files) use:

last

saruman ttyp1 Sun Oct 27 15:26 - 15:26 still logged in

saruman ttyp1 Sat Oct 26 19:54 - 15:26 (20:32)


saruman ttyp1 Thu Oct 24 17:17 - 19:54 (2+02:36)

saruman ttyp1 Thu Oct 24 17:17 - 17:17 (00:00)

saruman ttyp1 Tue Oct 22 21:39 - 17:17 (1+19:37)

saruman ttyp1 Tue Oct 22 21:39 - 21:39 (00:00)

saruman ttyp1 Sun Oct 20 15:51 - 21:39 (2+05:48)

saruman ttyp1 Sun Oct 20 15:51 - 15:51 (00:00)





saruman ttyp2 Wed Oct 16 23:02 - 15:38 (10+17:35)

.....

Screen Capture

The Mac OS X Screen Capture program 'Grab' can be called from the command line with:
screencapture ~/Pictures/sc1.pdf

to capture the whole screen to the given file, in pdf format. The option '-c' is available to
force the screen capture to go to the clipboard instead of a file.

Alternatively:
screencapture -i ~/Pictures/sc2.pdf

will start in interactive mode where:

control key - causes screen shot to go to clipboard
space key - toggle between mouse and window selection modes
escape key - cancels interactive screen shot

Apple System Profiler

The Apple System Profiler in /Applications/Utilities can be called from the command line to write
a profile report to the Terminal.

For example:
AppleSystemProfiler > ~/profile.txt

will write a profile report to the given file.

Note: this worked well in 10.1 but seems to be flaky in 10.2.

Software Update

The Software Update utility can be called from the command line. It will list the updates
available and allow one to select an update for installation.

For example:
softwareupdate

Software Update Tool
Copyright 2002 Apple Computer, Inc.

Software Update found the following new or updated software:

- 3280
AirPort Software (2.1.1), 4350K - restart required
- 3283
Internet Explorer 5.2 Security Update (5.2.2), 9330K
- 3359
QuickTime (6.0.2), 19620K - restart required
To install an update, run this tool with the item name as an
argument.
e.g. 'softwareupdate <item> ...'

Energy Saver

The settings controlled by the Energy Saver system preference pane can also be set from the
command line.

Use pmset (power management set) to control:

- the screen dim time (dim)
- the disc spin-down time (spin)
- the sleep time (sleep)

including the individual battery (-b) and charger (-c) settings of an iBook or PowerBook, or all
(-a).

For example:
pmset -a dim 5 spin 5 sleep 20 womp 1

will set the screen dim, disc spin-down, and sleep time for all (battery and charger) settings.

The 'womp 1' parameter says to 'Wake on Magic Packet' and is the equivalent of the 'Wake for
network administrator access' check box in the Energy Saver preference pane.

Preferences

Preference can be set from the command line. Applications present preference panes in which the
user can set preferences. These are recorded in ~/Library/Preferences/pref-file-name.plist.

These preferences can be viewed and set from the command line using 'defaults'.

For example, to view the Terminal settings:
defaults read com.apple.terminal

{
AppleSavePanelExpanded = NO;
AutoFocus = NO;
Autowrap = YES;
Backwrap = YES;
Bell = YES;
BlinkCursor = YES;
CleanCommands = "rlogin;telnet;ssh;slogin";
Columns = 80;
CursorShape = 0;
...
TerminalOpaqueness = 1;
... etc

Setting can be changed with:
defaults write com.apple.terminal TerminalOpaqueness 0.5

You will often find many more settings in these files than appear in the preference panes. For
example the Terminal Opaqueness setting has always been available, but only in 10.2 did it appear
in the preference panes.

Note that such options may not work correctly so experiment carefully.

The command:
defaults read

will display all settings for all programs.

Top of the Hogs

The 'top' command displays system usage statistics including cpu and memory usage. It displays a
list of running processes and the resources they are currently using.

For example:
top -u

lists processes in order of cpu usage (the top cpu hogs).

and:
top -e

lists event information.

Top Logging

Top can be used to periodically log usage statistics over a prolonged period using the '-l' and
'-s' options.

For example:
top -s60 -l120 > top.log &

will log every 60 seconds for 2 hours (120 logs) The output will be written to file 'top.log' and
the process will run in the background to free up the command line (putting '&' at the end of a
command causes it to run in the background).

Running Processes

The 'ps' command lists running processes; by default just those of the current user that are
running within a Terminal window.
ps xc

will display a list of all your processes; and:
ps axc

will display a list of all processes for all users.

The options 'l' and 'u' display more information for each process, and the option 'ww' avoids
truncation of long lines.

A Useful Alias for 'ps'

Create an alias to 'ps' a given application by name.

Type this:
echo "alias xps 'ps aucx | grep -i \\!:1'" >> ~/.tcshrc

to add the alias to your shell startup script, then start a new shell or Terminal window.

To get information for 'Clock' type:
xps clock

saruman 3677 0.0 0.5 72676 3552 ?? S
7:05PM 0:03.30 Clock

Kill(all)

To (force)quit an errant process use the kill command. This takes the process id (or PID) as a
parameter

For example, to kill the clock application:
open /Applications/Clock.app/
ps cx

PID TT STAT TIME COMMAND
178 ?? Ss 7:19.41 ATSServer
...
3587 ?? S 0:00.58 Clock
3359 p4 S 0:00.60 tcsh
3377 std S 0:00.52 tcsh

kill 3587

If the Clock refuses to lay down and die:
kill -KILL 3587

'-KILL' says to force quit the application instead requesting it to quit.

OS X 10.2 includes the 'killall' command which can kill processes by name.
killall Clock

(This is case sensitive.)


What is Whatis?

The command 'whatis' displays a brief one-line summary of the given command.

% whatis ls

ls(1) - list directory contents

'whatis' actually searches a database containing a one-line summary of all commands taken from
the man pages. It displays any lines that contain the search term as a complete word.

To update this database after you have added new commands do this:
sudo /usr/libexec/makewhatis man-path

where man-path is the directory containing the man pages. Most likely this will be
/usr/local/share/man for commands you have added yourself.

Is Apropos Appropriate?

The command 'apropos' gives a brief one-line summary for a given command.

% apropos cd

cd9660.util(8) - helps mount, probe, and unmount cd9660 filesystems
gluQuadricDrawStyle(3) - specify the draw style desired for quadrics
mount_cd9660(8) - mount an ISO-9660 filesystem
mount_cddafs(8) - mount an Audio CD
...

This is the same as Monday's 'whatis' except that the search term does not have to be a complete
word, as illustrated for 'cd' above. It searches the same database.

Note that 'man -k' is equivalent to 'apropos'.

Which Which?

'which' informs you which alternative of a given command would be executed if it were typed at
the command line.

For example, I have two versions of 'ls', one in /usr/bin and one in /usr/local/bin. I also have
an alias called 'ls'.

% which ls
ls: aliased to ls --color=tty

tells me that the alias will be run, and also what the alias will be expanded to.

Now try this:
~ % which which
which: shell built-in command.

This tells me which 'which' I executed - and it was in fact the shell's built-in 'which' - hence
its knowledge of aliases.

Escaping From Aliases

This is a handy trick if you wish to run a command - and you want to run the actual executable -
but it has been aliased or the shell has a built-in version.

For example, Wednesday's tip told us that 'ls' was aliased. In order to run the original 'ls' I
could give the full pathname (if I knew it):
% /bin/ls

Alternatively I can escape the command with '\':
% \ls

Which Which is Which?

Following on from Wednesday's and Thursday's tips, I now wish to know which 'ls' I am running
when I type:
% \ls

(ie the non-aliased 'ls').

I can't simply type:

% which ls
ls: aliased to ls --color=tt

because this just tells me about the alias 'ls'. Remember that from:
~ % which which

'which' is also built-in, which is why it considers aliases.

To tell us about non-built-in commands we can do one of two things.

From this weeks tips it should be possible to guess at the commands.

Or scroll down for the answer.....
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

either 1)
~ % which \ls
/usr/local/bin/ls

or 2)
% \which ls
/usr/local/bin/ls

And to determine which 'which' is which:
% \which which
/usr/bin/which

or:
% which \which
/usr/bin/which

I hope you followed that! :-)

The Path to Execution

When you execute a command the shell first looks for that command by following a path from one
'bin' directory to another (after first resolving aliases and checking if the command is built in
to the shell itself).

The path is described by the environment variable PATH. You can check the path with:
echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin

A Longer Path

If you wish to put commands in a new bin directory such as:
/Users/you/bin

add this directory to your path so commands will be found automatically, without typing the full
pathname.
% echo $PATH

% setenv PATH /Users/saruman/bin:$PATH

% echo $PATH
/Users/saruman/bin:usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin

(You could instead use ~/bin.)

Add the 'setenv' command to ~/.login to have the path set automatically when you login.

paths and PATHs

Two Paths?
% echo $path
/Users/saruman/bin /usr/local/bin /usr/bin /bin /usr/local/sbin /usr/sbin /sbin

% echo $PATH
/Users/saruman/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin

What is the difference between 'path' and 'PATH'?

'path' is a shell variable and 'PATH' is an environment variable. You may set either, and
whichever is changed the shell will synchronise the other.


Dotty Paths

If you write a shell script and wish to execute it, simply typing:
% my_script
my_script: Command not found.

will not work even though my_script is in the current directory. Why? Is the shell blind? Yes, it
is blind to all exectuables except those on the PATH.

You must type
% ./my_script
script runs....

By giving a pathname the shell does not need to use the search path.

To avoid doing this add the current directory '.' to the search path:
% echo $PATH

% setenv PATH "${PATH}:."

% echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:.

Warning: adding '.' to your path has security implications.

Rehash the Cache

Ok, previously I said that the shell searches PATH to find commands. Well, it doesn't really.
When the shell first starts, or when the PATH is changed, it follows PATH and remembers (caches)
all the commands in each directory.

When you execute a command the shell simply looks in its cache, which is much quicker than
searching PATH. Consequently when you add a new command or script it will not be found.
rehash

will cause the shell to recreate its cache.

Note that the 'bash' shell rehashes automatically.

If you think about it: adding '.' to the path as in Thursday's tip, the shell must treat it as a
special case.

About disktool

'disktool' will unmount, mount, rename, or eject drives and partitions.
disktool

will list the options. It does not have a manual page.

You must supply device names to 'disktool', not disk or volume names.
/dev/disk0 is a disk
/dev/disk0s9 is a partition (or slice) of disk0.

Type:
disktool -l

to determine which volume names (also called mount points) correspond to which /dev/... device
names.

Refresh Disk Icons with disktool

If you mount a server or drive using the command line 'mount_afp' or 'mount_hfs', the Finder may
not show the icon in 'Computers'. Use:
disktool -r

to refresh the Finder's view.

Eject with disktool

Use 'disktools' to eject a CD that seems stuck.
disktool -l

will tell you which disk represents the CD.

disktool -l
*** Unrecognized disk appeared on disk1 ***
*** Unrecognized disk appeared on disk1s1s1 ***
***Disk Appeared ('disk1s1s2',Mountpoint =
'/Volumes/Return to Zork', fsType = 'hfs', volName =
'Return to Zork')
...

To eject 'Return to Zork' (disk1) use:

disktool -e disk1
disk1 device will attempt to be ejected ...
***Notifications Complete for type 1
***Disk Unmounted('disk1')
***Disk Unmounted('disk1s1s1')
***Disk Unmounted('disk1s1')
***Responding yes to unmount - disk1s1s2
***Disk Unmounted('disk1s1s2')
***Responding yes to eject - disk1
***Responding yes to eject - disk1s1s1
***Responding yes to eject - disk1s1s2
***Responding yes to eject - disk1s1
***Disk Ejected('disk1')

Unmount with disktool

You can use 'disktool' to unmount whole disks or individual partitons. For example, I wish to
unmount the partition /Volumes/Backup.

First use 'mount' or 'disktool -l' to determine the device:

mount
/dev/disk0s9 on / (local)
devfs on /dev (local)
fdesc on /dev (union)
<volfs> on /.vol (read-only)
/dev/disk0s10 on /Volumes/Grima (local)
/dev/disk0s11 on /Users (local)

automount -fstab [400] on /Network/Servers (automounted)
automount -static [400] on /automount (automounted)
/dev/disk2s2 on /Volumes/Font Tools (local, nodev, nosuid, read-only)
/dev/disk3s11 on /Volumes/Saruman (local)
/dev/disk3s10 on /Volumes/Sauron (local)
/dev/disk3s9 on /Volumes/Melkor (local)
/dev/disk3s12 on /Volumes/Backup (local)

Then unmount using the '-p' option of disktool:

disktool -p disk3s12
disk3s12 partition will attempt to be unmounted ...
***Notifications Complete for type 1
***Responding yes to unmount - disk3s12
***Disk Unmounted('disk3s12')

pdisk

You might wonder why your disk partitions are named as they are. For example I have three
partitions on my startup disk, as shown by 'mount'.

mount
/dev/disk0s9 on / (local)
/dev/disk0s10 on /Volumes/Grima (local)
/dev/disk0s11 on /Users (local)

Numbered 9, 10, 11. To find out what all the other partitions (0 to 8) are, use: 'pdisk'

sudo pdisk
Password:
Top level command (? for help): l /dev/disk0
/dev/disk0 map block size=512
#: type name length base ( size )
1: Apple_partition_map Apple 63 @ 1
2: Apple_Driver43*Macintosh 56 @ 64
3: Apple_Driver43*Macintosh 56 @ 120
4: Apple_Driver_ATA*Macintosh 56 @ 176
5: Apple_Driver_ATA*Macintosh 56 @ 232
6: Apple_FWDriver Macintosh 512 @ 288
7: Apple_Driver_IOKit Macintosh 512 @ 800
8: Apple_Patches Patch Partition 512 @ 1312
9: Apple_HFS Curunir 21118960 @ 1824 ( 10.1G)
10: Apple_HFS Grima 5732288 @ 21120784 ( 2.7G)
11: Apple_HFS Isengard 51287080 @ 26853072 ( 24.5G)
12: Apple_Free 0+@ 78140152

What's the Code for London?

Q. What is the international dialling code for London?

A. grep -i London /usr/share/misc/inter.phone

What's the ASCII Code For...?

Q. What is the ASCII code for 'A'?

A. man ascii

What's with /var?

Q. What is the Unix directory /var used for?

A. man hier

When was Apple Founded?

Q. When was Apple Computer founded?

A. cat /usr/share/calendar/* | grep Apple | grep founded

(In case you are wondering, using cat and grep gives a cleaner output than grep alone when one
passes a wildcard to cati.)

Wanna Cheat at Crosswords?

Q. 12 Across : t _ _ _ k _ _ _r

A. cat /usr/share/dict/words/ | grep -wi 't...k...r'

(You didn't get this from me, right!)

One Column Output

To list a directory one file per line (without a long listing using ls -l) use:
ls -1
Desktop
Development
Documents
Library
...

(-1 is the number one, not the letter ell).

Alternatively use:
ls | cat

Directory Details

To list the details of a file one would use:
ls -l letter.txt
-rw-r--r-- 1 saruman staff 0 Nov 22 17:32 letter.txt

If you wish to do the equivalent for a directory, 'ls -l' does not work as it lists the directory
contents. To list details of the directory itself use the '-d' option.
ls -ld Documents/
drwx------ 16 saruman staff 544 Nov 22 13:43 Documents

List Long /etc

If you've ever tried to do a long list (ls -l) on the directory /etc:
ls -l /etc
lrwxrwxr-t 1 root admin 11 Nov 22 14:03 /etc -> private/etc

you will have found that it does not list the directory contents. This is because /etc is a link
(to /private/etc/).

To list the contents use either:
ls -l /private/etc/

or better, use the '-L' option:
ls -lL /etc
total 1256
-rw-r--r-- 1 root wheel 753 Jul 15 04:03 6to4.conf
-rw-r--r-- 1 root wheel 515 Jul 14 07:20 afpovertcp.cfg
-rw-r--r-- 1 root wheel 6 Nov 22 15:22 appletalk.cfg
-rw-r--r-- 1 root wheel 256 Nov 22 15:23 appletalk.nvram.en1
.......

The Quickest way to List a Directory

Q. How can you list the current directory with one keystroke (ls takes 3 including the return)?

A. Simply hit the tab key on an empty command line.

(This tip assumes you have 'autolist' set. If not enable it by adding the line
set autolist

to your ~/.tcshrc file.)

Distinguish File Types

If you don't use colour 'ls', then a simple:
ls

will not give any indication of what a file is - plain file, directory, link, executable, etc.

Use option '-F'
ls -F
directory/ file1 file2 link1@ my_script*

* @ = link
* / = directory
* * = executable

The next tip will be on Monday 30th. Until then have a good:
grep 12/25 /usr/share/calendar/*

and happy birthday Sir Isaac Newton!

Kill A Command Line

I hope Santa delivered that new Mac you asked for :-).

If you have started to type a command line and wish to abandon it, rather than back-deleting a
character at a time, type control-u to delete the whole line.

(Also, use control-k will delete from the cursor to the end of the line.)

Deja Vu

We all know that pressing the up arrow key moves back through the command history. But this is
tedious if the desired command is a long way back in the history sequence.

To jump straight back to a command, type the first part of it then hit esc-p (the escape or 'esc'
key followed by 'p'). Another esp-p will jump back further to a previous match.

Have a happy:
grep 01/01 /usr/share/calendar/*

and happy birthday the Internet, 20 years old!

Glob Now

If I type:
rm fil?*

how can I check exactly which files will be removed before I hit return?

Either precede the command with:
ls fil?*

or expand the globbing ('*' '?' or '[]' wildcard characters).

Typing control-x followed by '*' will expand 'fil?*' onto the command line:
rm fil?* (hit control-x followed by *)

You will end up with an expanded command line like:
rm file1 file2

Hit return if this is ok, otherwise hit control-u to delete the command line (instead of the
files!).

List Globbing

I hope the new year's hangover has subsided and you are no longer seeing double!

Similar to Wednesday's tip, to list an expanded globbing rather than expanding it to the command
line, hit control-x followed by G (upper case G).
ls fil?* (hit control-x followed by G)
file1 file2
ls fil?*

Similar to Wednesday's tip, to list an expanded globbing rather than expanding it to the command
line, hit control-x followed by G (upper case G).
ls fil?* (hit control-x followed by G)
file1 file2
ls fil?*

Which Command?

As in a previous tip, the command:
which ls
ls: aliased to ls --color=tty

says which variant of 'ls' will be executed.

Following a command with escape then '?' will also do this:
ls (hit escape followed by ?)
/usr/local/bin/ls --color=tty
ls

What's the difference?

Use the "diff" command to compare two files. "diff" will report on all lines that differ, have
been added, or have been deleted. Diff assumes the first file is the original and the second file
is the newer.
cat f1.txt
This line in both
This line has File One
This line in both
This Line not in File Two
Completely different in each file

cat f2.txt
This line not in file one
This line in both
This line has File Two
This line in both
Not at all the same as in file one

diff f1.txt f2.txt
0a1
>This line not in file one
2c3
< This line has File One
---
> This line has File Two
4,5c5
< This Line not in File Two
< Completely different in each file
---
> Not at all the same as in file one

The "< and "> characters indicate which file. Imagine each as an arrow pointing to the position
of the file name on the "diff" command line.

"0a1" means line 1 of f2.txt is an addition compared to f1.txt.

The "2c3" indicates line 2 of f1.txt has changed in f2.txt, and is line 3 because of the
additional line as above.

What's a Little White Space Between Friends

Sometimes files may differ in insignificant ways. To ignore only differences in white space
(spaces and tabs) use option '-b'. To ignore differences due to blank lines use option '-B'.
diff -bB f1.txt f2.txt

The '-q' option can also be useful.
diff -bBq f1.txt f2.txt
Files f1.txt and f2.txt differ

Side by Side

Diff can compare two files by arranging them side by side via the '-y' opton:
cat f1.txt
This line in both
This line in both

cat f2.txt
This line in both
This line in both

diff -y f1.txt f2.txt > diff.txt

It is easier to direct output to a file and view it with an editor - preferable in a wide window.

This line in both This line in both
This line has File One | This line has File Two
This line in both This line in both
This Line not in File Two <

(This may not format correctly in your email.)

diff diff diff

'diff3' can compare three files at once. It will compare the difference between file1 and file3
relative to file2. The report treats file2 as the oldest file, and shows how the updates to file1
differ from those to file3.
cat f1.txt f2.txt f3.txt
This line in both
This line in both

This line in both
This line in both

This line in both
This line has File Three
This line in both

diff3 f1.txt f2.txt f3.txt
====
1:2c
This line has the words File One
2:2c
This line has the words File Two
3:2c
This line has the words File Three
====2
1:4c
3:4c

2:3a

diff Directories

'diff' can compare whole directories, comparing like-named files, and reporting files that are in
one directory but not the other.
diff test test2

diff test/f1.txt test2/f1.txt
3d2
< This line in both
Only in test: test.info
Only in test2: test2.info

Here, f1.txt in test2 has had a line removed, and each directory has a self-named .info file
added.

The recursive option '-r' tells diff to look into and compare like-named nested directories too.
Otherwise directories are ignored.

Editing Hidden Files in GUI Apps

When you choose 'File -> Open...' in an editor such as TextEdit, hidden Unix files (those
starting with a dot) are not shown. If you wish to edit one either:

type the full name into the Go To box at the bottom of the open dialog window:
~/.tcshrc

or navigate to the directory containing the file and type its name into the Go To box:
.tcshrc

TextEdit understands Unix end of line characters.

Editing Hidden Files in OmniWeb

Enter the location:
file:///

into the address field.

OmniWeb can be used as a file browser, showing hidden files too.

Double click directories to navigate them. Double click a text file to open it. Hit 'View Source'
in OmniWeb's toolbar and you can edit a text file too.

Edit With 'echo'

If you need to add a line to a Unix text file, an easy way is to use echo. For example to active
the shell's autolist feature I need to add:
set autolist

to my ~/.tcshrc file.

I can do this with the command:
echo "set autolist" >> ~/.tcshrc

NOTE: Use a >> not a single >. The double redirect appends to a file, the single redirect will
start a new file overwriting any existing file of that name.

Edit With 'cat'

Use the 'cat' command to create a new file, or add to the end of an existing file.
cat > newfile.txt
Type your text
here until you have
finished then
type control-d

To add to the end of an existing file use double redirect:
cat >> newfile.txt
type here...

Edit with 'printf'

Here is a neat trick using printf.
ls
file1 file2 file3 file4 file5

( printf "<pre>\n" ; printf "<tt>%s</tt>\n" `ls` ; printf "</pre>\n" ) > ~/list.html

cat ~/list.html

<pre>
<tt>file1</tt>
<tt>file2</tt>
<tt>file3</tt>
<tt>file4</tt>
<tt>file5</tt>
</pre>

'sort'

Command 'sort' will sort the lines of a text file alphabetically (or alphanumerically).

For example:

cat to-sort.txt

ls 3 list files

cd 2 change directory

pwd 4 print working directory

grep 1 search files for regular expression

sort to-sort.txt



ls 3 list files


The sorted output is written to the Terminal. Use redirection or option '-o' to write to a file.

DON'T write the output back to the input file or you will end up with an empty file.

Mac OS X uses GNU 'sort'.

Sort on Another Field

Command 'sort' will sort on any field, where fields are separated by white space.

For example, to make the second field the sort key use:

cat to-sort.txt

ls 3 list files




sort -k2 to-sort.txt



ls 3 list files


In this example the files were separated by tab characters.

Changing the Field Separator

Tuesday's example sorted a file using the second field of each line, and 'sort' assumed that
fields were separated by white space.

Q. What if fields are separated by a character other than white space?
A. We can specify a custom separator with option '-t', as in:
cat to-sort.html
<a href="../ls">list files
<a href="../cd">change directory
<a href="../pwd">print working directory
<a href="../grep">search files for regular expression

sort -t\> -k2 to-sort.html
<a href="../cd">change directory
<a href="../ls">list files
<a href="../pwd">print working directory
<a href="../grep">search files for regular expression

Now the file is sorted according to the description rather than the URL.

(Note that '<' is escaped to prevent the shell from interpreting it as a redirection character -
see Tutorial 7.)

Space Separated Fields

You may have a file where the fields are separated by multiple spaces, rather than tabs as in
Tuesday's example. Using 'sort' results in rather odd behaviour. This example attempts to sort by
field 2, the sequence number.

cat to-sort.txt

ls 3 list files




sort -k2 to-sort.txt


ls 3 list files



Use option '-b' to ignore leading blanks when finding sort keys:

sort -b -k2 to-sort.txt



ls 3 list files


Sorting and Merging Files

Command 'sort' can be used to sort and merge two or more files.
cat s1.txt
9
7
1
4

cat s2.txt
2
3
8
6
5

sort s1.txt s2.txt
1
2
3
4
5
6
7
8
9

Recall a Command by Name

If you wish to repeat a previously issued command, then type '!' followed by enough of the
command to make the recall unambiguous.

For example:
!pri

will recall, for me currently, the last issued 'printenv' call:
!printenv

Recall a Command by Number

Issuing the command 'history' will list a numbered history of commands. You can recall any
command with '!n' where 'n' is the command number given by the history list.

history

1 09:31 cd ~

...

95 23:04 printenv

96 23:07 echo "Hello"

97 23:07 ls -al

98 23:07 vim index.ws

99 23:13 history

100 23:13 ls


!96

echo "Hello"

Hello

The last command is recalled with:
!!

and the ones before that with:
!-2
!-3

etc.

If your history is long, type
history n

to display just the last n commands.

Recall and Tabbed Completion

Following on from Monday's tip, if you want to be sure of the command to be executed by:
!pri

hit the tab key and tcsh will complete the command and wait for you to hit return to execute it,
or control-u to dismiss it.

Search and Recall

A command may be recalled by giving a search string. The last issued command that matches the
string will be executed.

For example:

history

...

2044 23:26 ls -al

2045 23:26 vim log.txt

2046 23:26 ls

2047 23:26 cat log.txt

2048 23:27 cat index.ws

2049 23:27 history

!?.txt?

cat log.txt <-- this command will be found and issued

Recalling Command Arguments

The individual arguments of a previously issued command may be recalled.

For example:
less a-very-long-filename-to-type-again
vi !!:1
vi a-very-long-filename-to-type-again <-- this command will be issued

!!:0 is the command itself, and !!:1 is the first argument, !!:2 the second, and so on.

It is also possible to specify arguments of other commands with !n:n and !-n:n (see Tuesday's
tip).

The 'cd' Dash

Suppose you change directory to issue a few commands, and then wish to jump back to the original
directory. To save retyping the original directory name one can make use of the shell's special
'old working directory' shortcut '-'.

Start here:
pwd
/Users/saruman/Documents/My Setup

Change directory temporarily:
cd /System/Library/Displays/Overrides/Contents/Resources/English.lproj/
pwd
/System/Library/Displays/Overrides/Contents/Resources/English.lproj
...

Now jump back to the original directory:
cd -
pwd

The Old Working Directory

The shell keeps a record of the current and old working directories in variables 'cdw' and 'owd'.
cd ~
cd Documents/My\ Setup/

echo $cwd
echo $owd
/Users/saruman

Monday's trick is equivalent to:
cd $owd

Back and Forth

Following on from Monday's tip, when you issue 'cd -' the current and old working directories are
effectively swapped. This makes it possible to jump back and forth between two directories.
cd ~
cd Documents/My\ Setup/

pwd
cd -
pwd
/Users/saruman
cd -
pwd
cd -
pwd
/Users/saruman

Implicit CD

Setting the shell variable 'implicitcd' causes the shell to treat a directory name typed without
'cd' as a request to change to that directory.
cd ~
ls
Desktop Documents Movies Pictures Sites bin osxfaq
Development Library Music Public Temporary Items
Sites
Sites: Permission denied.
set implicitcd
Sites
pwd
/Users/saruman/Sites

Naturally, if a command of the same name exists, the command is executed in preference to
changing directory.

cdpath

How does the 'cd Sites' command below work?
cd /System
ls
Library
cd Sites
ls
Tips Tutorials archive areas css frames images index.html
pwd
/Users/saruman/Sites

The command 'cd Sites' should not work, but it has managed to change directory to 'Sites' in my
home directory. This behaviour is controlled by the shell variable 'cdpath'
set cdpath= ( ~ ; ~/Sites )

Tells 'cd' that if it can't find the requested directory in the current directory, search my home
directory (~), and then ~/Sites

So Insensitive

(See also Weeks 1 and 13)

As Mac OS X is case insensitive, it makes sense to search for files in a case insensitive way.
Instead of using 'find -name' use 'find -iname'.
find . -name "file*"

./.vim/filetype.vim
./Documents/test/file1

find . -iname "file*"
./.vim/filetype.vim
./Documents/test/file1
./Documents/test/FILE2
./Documents/test/File3
./Library/Preferences/BBEdit Preferences/File Filters

Deleting Files


'find' has a built-in delete option that can be used to delete all found files. Use this with
caution, running a test find first.

For example, to delete all .tmp files from the home directory use:
cd ~
find . -iname "*.tmp"
./Documents/Letters/letter.tmp
./Documents/Miscellaneous/list.tmp
./osxfaq/templates/template.TMP
./Sites/Tips/unix-tricks/schedule.tmp
./Sites/Tips/unix-tricks/tip.tmp

Check the list of files - now it's safe to actually delete:
find . -iname "*.tmp" -delete

find . -iname "*.tmp"

This command can be used to remove all '.DS_Store' files, though you may have to 'sudo' it.

Selective Paths


'find' can search a whole directory structure and return just those files that are in a given
range of directories.

For example, to search my home directory for all the weekly tips, I can use:
find ~ -path "*week*"
/Users/saruman/osxfaq/templates/tips-week-template
/Users/saruman/osxfaq/templates/tips-week-template/friday.ws
/Users/saruman/osxfaq/templates/tips-week-template/monday.ws
/Users/saruman/osxfaq/templates/tips-week-template/thursday.ws
/Users/saruman/osxfaq/templates/tips-week-template/tuesday.ws
/Users/saruman/osxfaq/templates/tips-week-template/wednesday.ws
/Users/saruman/Sites/Tips/unix-tricks/week1
/Users/saruman/Sites/Tips/unix-tricks/week1/friday.ws
/Users/saruman/Sites/Tips/unix-tricks/week1/monday.ws
/Users/saruman/Sites/Tips/unix-tricks/week1/thursday.ws
/Users/saruman/Sites/Tips/unix-tricks/week1/tuesday.ws
/Users/saruman/Sites/Tips/unix-tricks/week1/wednesday.ws
...

The option '-ipath' is also available.

Multiple Filters


Following on from Wednesday's tip, suppose I wish to find the weekly tips for each Monday. I can
apply a second filter to 'find' as in:
find ~ -path "*week*" -name "monday.ws"
/Users/saruman/osxfaq/templates/tips-week-template/monday.ws
...

All of the files that pass the '-path' filter are filtered again by the '-name" filter. Only
files that pass both filters will be displayed.

Locate All Empty Files and Directories

find . -empty

will do just that for the current directory.

To remove them use:
find . -empty -delete

The delete option will remove files and directories. If a directory contains just empty files,
the files will be deleted and then the directory, which has suddenly become empty, will also be
deleted.

Note that:
find . -size 0

will not detect empty directories as they have a non-zero size.

Pushing and Popping

If you issue 'cd' many times and then wish to retrace your steps back to the original directory,
use the shell's directory stack.

To change directory use:
pushd

To return use:
popd

Each command echoes the current directory stack.

For example:
pwd
/Users/saruman

pushd /System/Library/Extensions/
/System/Library/Extensions ~

pushd /Library/StartupItems/Tablet/
/Library/StartupItems/Tablet /System/Library/Extensions ~

pushd /etc
/etc /Library/StartupItems/Tablet /System/Library/Extensions ~

popd
/Library/StartupItems/Tablet /System/Library/Extensions ~
pwd
/Library/StartupItems/Tablet

popd
/System/Library/Extensions ~
pwd
/System/Library/Extensions

popd
~
pwd
/Users/saruman

Silencing pushd and popd

Following on from Monday's tip, if you don't want pushd and popd to echo the current directory
stack, set the shell variable 'pushdsilent'.
set pushdsilent

Viewing the Push/Pop Stack

If you wish to view the current directory stack, useful if 'pushd' and 'popd' are silenced, issue
the command:
dirs
/Library/PreferencePanes /System/Library /Developer/Applications ~

If the stack is deep, use:
dirs -v
0 /Library/PreferencePanes
1 /System/Library
2 /Developer/Applications
3 ~

Saving the Directory Stack

The directory stack may be saved between login sessions by setting the shell variable 'savedirs'
set savedirs

It may also be saved and loaded manually with the commands:
dirs -S

and
dirs -L

For example:
dirs
dirs -S
dirs -c
dirs
/Library/PreferencePanes
dirs -L
dirs

Nore that 'dirs -c' was used to clear the directory stack.

Jumping to a Stacked Directory

'pushd' will jump to any directory in the stack by specifying '+n' where 'n' is the position in
the stack.

For example:

dirs -v

0 ~/Sites/Tips/unix-tricks


2 /System/Library


4 ~

pwd

/Users/saruman/Sites/Tips/unix-tricks

pushd +3

pwd

/Developer/Applications

dirs -v


1 ~/Sites/Tips/unix-tricks


3 /System/Library

4 ~

System Cron

The Unix cron daemon runs continually looking at /etc/crontab (the cron table). This file lists
commands against the time at which they should be run.

Add your own scheduled commands to this file (which must be edited as the super-user).

The '*' means 'on every'. Each of the fields are as commented in /etc/crontab.

User Crontabs

Create a user-specific crontab using the command 'crontab'

Create a file with each line of the form:
minute hour month-date month day-of-week command and argumants

Issue the command:
crontab

to enter the commands into your crontab.

To list your crontab:
crontab -l

and to remove it:
crontab -r

Note that the format of user crontabs is the same as that for the system crontab except that the
'who' field is omitted. Commands are run as the user who created the crontab file.

Stopping 'cron' From Spamming

'cron' will mail you the results of running you crontab commands. To prevent this (especially if
you do not have sendmail running) add:
MAILTO="

to the top of your crontab file. Remove the old crontab and enter the new one (see Tuesday's
tip).

crontab Tricks

The time fields in your crontab can be more complex than simple numbers. To run 'command' at
11am, 4pm, and 8pm, use:
0 11,16,20 * * * command

And to run every 10 minutes:
*/10 * * * * command

Time Shortcuts

Instead of the five number fields, crontab can take special strings:

@reboot run at startup.

@hourly once an hour == "0 * * * *"

@daily once a day == "0 0 * * *"

@weekly once a week == "0 0 * * 0"

@monthly once a month == "0 0 1 * *"

@yearly once a year == "0 0 1 1 *"

List the Current Bindings

The command:
bindkey

will list all the shell's key bindings.

From the following extract....
...
"^[p" -> history-search-backward
"^X*" -> expand-glob
"^XG" -> list-glob
...

we can determine that 'escape-p' ('^[' signifies the escape key) searches back in the command
line history, and 'control-X *' expands globbing (see Week 25).

Roll Your Own

The shell allows one to change or add key bindings.

For example:
bindkey ^[a list-glob

causes the key sequence 'escape-a' to list globbing.

To find out what functions can be bound to keys, issue the command:
bindkey -l

Emulating Your Favourite Editor

The command line editing functions of the shell can be bound to keystrokes that emulate the vi
editor.
bindkey -v

Or, if you're of a different religion:
bindkey -e

for GNU Emacs.

Binding to Strings

'bindkey' normally binds a key sequence to one of the built-in functions, shown by 'bindkey -l'.
To bind keys to an arbitrary string use the '-s' option.
bindkey -s ^xa "Adrian Mayo"

Whenever I type the key sequence 'control-x a', it is replaced with my name.

Binding to Commands

'bindkey' can bind a key to a command with the '-c' option.

The command:
bindkey -c ^xe "ls -alL /etc"

will bind the key sequence 'control-x e' to the ls command specified. Note that it is not
necessary to press return after the key sequence.

Filtering Repeated Lines

The command 'uniq' will report or filter out repeated lines in a file. It reads and writes the
standard streams, so file redirection is required to read and write files. The -c option reports
the repeat count.

'uniq' is useful for trimming log files where the same message is repeated many times. Multiple
blank lines are also filtered.

For example, to filter repeated lines in f3
cat f3
Line 1
Line 1
Line 2
Line 3
Line 4
Line 4
Line 4
Line 4
Line 4
Line 4
Line 5
Line 1
Line 1
Line 1

uniq -c
2 Line 1
1 Line 2
1 Line 3
6 Line 4
1 Line 5
3 Line 1
1

Comparing Files

The command 'comm' will report on lines that are common to two files. The files must be sorted
lexically which can be achieved with 'sort' (see Week 28).

For example:
cat f1 f2
A line
B line
D line
F line

A line
B line
C line
E line
F line

comm f1 f2
A line
B line
C line
D line
E line
F line

The command 'diff' also compares files.

Extracting Common and Difference Lines

Using command 'comm' with options '-1', '-2', and '-3', one can extract common and difference
lines between two files.

Continuing from Tuesday's example, to extract the common lines between f1 and f2:
comm -12 f1 f2
A line
B line
F line

or:
comm -12 f1 f2 >f3

to write back to a new file. The options specify which columns to cut out of the report.

For the difference lines, use opton '-23' or '-13'

Splitting a Large File

Use the command 'split' to break a large file into multiple smaller ones.
split -b5k big-file

will split big-file into multiple 5k byte files.

And:
split -l200 big-file split_

will split big-file into multiple 200-line files, and the generated filenames will be prefixed
with 'split_'.

Merge Files with Paste

Paste will merge two files, line by line.

As an example:
cat f1 f2
Line 1
Line 2
Line 3
Line 4

rest of line 1
rest of line 2
rest of line 3
rest of line 4

paste f1 f2
Line 1 rest of line 1

creating and extracting tar balls

Just as Aladdin Systems' StuffIt allows one to 'stuff' many files into a single '.sit' file, the
Unix 'tar' command allows one to create an archive (often called a tar ball). 'tar' was
originally used to archive to magnetic tape (tape archiver), but now a days it is more often used
to archive to a named file.

To archive the whole of the ~/Sites directory into a single file called sites.tar, use:
cd ~
tar cvf sites.tar Sites

The options 'cvf' do the following:
c - create a new archive
v - verbose, name each file as it is added to the archive
f - archive to a named file, in this case sites.tar

The original directory is untouched.

To decompress the archive use the command:
tar xvf sites.tar

This extracts the original files into a directory called 'Sites', which is created if it does not
already exist.

The option 'x' says to extract from existing archive.

zipping tar balls

'tar' is able to compress and decompress archives using Lempel-Ziv coding (LZ77) - aka zip.
Simply add the option 'z' when compressing and extracting.
tar czf sites.tgz Sites

and
tar xzf sites.tgz

Don't use capital Z as this compresses using 'compress', not zip.

adding to and listing archives

One can add files to an existing tar ball using option 'r' (replace?) instead of 'c'.

The command:
tar rf sites.tar bin

will add all the files in directory 'bin' to an existing archive called 'sites'tar'.

To examine an archive use option 't' (table of contents).
tar -tf sites.tar


path names

The command:
tar cvf sites.tar Sites

differs from:
tar cvf sites.tar /Users/saruman/Sites

The first variant creates a tar file that extracts into 'Sites' in the current directory.

The second variant uses the whole path name, but strips the leading /, and creates a tar file
that extracts into Users/saruman/Sites in the current directory.

It's a good idea to always try:
tar -tf sites.tar

before actually extracting, so as you know where the extracted files are going to be placed.

absolute paths

The command:
tar cvfP sites.tar /Users/saruman/Sites

creates an archive that can be extract into /Users/saruman/Sites, independent of the current
directory. Compare this with Thursday's tip.

The option -'P' prevents the leading '/' from being stripped when the archive is made. However,
it will still be stripped when the archive is extracted unless the extract command also uses
option '-P'.
tar xvfP sites.tar

Zipping

I guess the zip compression utility needs no introduction. OS X comes with the GNU version, gzip.

To compress (zip) a file, use:
gzip filename

This will replace the file with a compressed version with the extension '.gz'

To unzip:
gunzip filename

or
gzip -d filename

The default level of compression (which is 6) can be changed. 1 (--fast) is the lowest level of
compression but the fastest algorithm, 9 (--best) is the highest level of compression but slowest
algorithm.
gzip --best filename

Zipping Several files

To compress more than one file into a single archive, use the '-c' option.
gzip -c filename1 filename2 > two-files.gz

This technique can be used to avoid replacing the original file:

gzip -c filename > filename.gz

The '-c' option is actually sending the compressed file to the terminal.

Too Many Members

If gzip gives you an error whilst trying to decompress a file, complaining about more than one
member, it probably means that the original files was compressed with 'zip'. To decompress such a
file use 'unzip' (without the 'g').

Displaying Without Unzipping

If you wish to display a compressed file without the hassle of decompressing it and then having
to compressing it again, use the '-c' option with gunzip, or use zcat.

For example, to peek at old system log files:
zcat /var/log/system.log.0.gz

There are many more 'z' variants such as zmore, zdif, and (not standard OS X, but part of the
fink base package) zless.

Other Compression

bzip2 is the new black.

bzip2 compresses better that gzip, and usage is similar to gzip. A number of 'bz' commands are
also available, such as bzcat, bzless, bzgrep, and bzip2recover for damaged compressions.
man bzip2

will tell you all.

'compress' is an older variant, and uses the extension '.Z'. gunzip will decompress '.Z' files.

The Dictionary

Visit:
/usr/share/dict

This has a couple of dictionaries, plus connectives and proper nouns.

From the README:

Welcome to web2 (Webster's Second International) all 234,936 words worth.
The 1934 copyright has elapsed, according to the supplier. The supplemental 'web2a' list contains
hyphenated terms as well as assorted noun and adverbial phrases. The wordlist makes a dandy
'grep' victim.

Log Files

Visit:
/var/log

for most of the log files kept by OS X, including the system log, netinfo, lookupd, mail, ftp,
and apache.

Keep an eye on log files with (for example the apache access log):
% tail -f /var/log/httpd/access_log 217.207.57.220 - - [31/Mar/2003:14:22:15 +0100] "GET
/default.ida?XXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%u9090%u6858%ucbd
3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090
%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a
HTTP/1.0" 200 809 217.155.44.175 - - [31/Mar/2003:15:01:18 +0100] "GET /default.ida?XXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXX%u9090%u6858%ucbd3%u7801
%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090
%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a
HTTP/1.0" 200 809

This will update each time an entry is added to the end of the log. Control-c to stop. In this
case we see attacks by a Code Red variant, which is back with us at the moment. Of course,
running apache we are immune to this attack.

Learn About Samba

Issue the command:
% open /usr/share/swat/using_samba/index.html

Learn About Cups

Issue the command:
% open /usr/share/doc/cups/documentation.html

Separating Filename Components I

Use the following technique in tcsh scripts to extract components of a full pathname from a
variable or parameter.

The full pathname:
% set fn="/Users/saruman/Documents/Essentials/todo.rtf"
% echo $fn
/Users/saruman/Documents/Essentials/todo.rtf

Extract the pathname (head):
% echo $fn:h
/Users/saruman/Documents/Essentials

Extract the filename (tail):
% echo $fn:t
todo.rtf

Extract the extension:
% echo $fn:e
rtf

Remove the extension:

% echo $fn:r
/Users/saruman/Documents/Essentials/todo

Separating Filename Components II

Following on from Monday's tip, the ':' modifiers can be combined. To extract just the filename
without the extension combine t and r:
% echo $fn

% echo $fn:t:r
todo

You can change a filename using substitution:
% echo $fn

% echo $fn:t:s/.rtf/.txt/
todo.txt

Batch Rename

Mac OS X does not have a rename command capable of renaming multiple files. Use the techniques
from Monday and Tuesday to write a script to rename.
% cat rename
#!/bin/tcsh
set fn_new={$1:r}.$2
echo rename $1 $fn_new
mv $1 $fn_new

Combine with 'find' from weeks 1 and 13 to batch rename:
% ls
a.htm b.htm c.htm

% find . -name "*.htm" -exec rename {} html \;
rename ./a.htm ./a.html
rename ./b.htm ./b.html
rename ./c.htm ./c.html

% ls
a.html b.html c.html

Extraction on the Command Line

The techniques from Monday and Tuesday can be applied to tcsh commands issued on the command
line. In particular they can be combined with the tips from Week 29, Friday:
% ls *.html

% ls !*:r
ls *
a.html b.html c.html rename rename2

and
% ls *.htm
ls: No match.

% ls !*:r.html
ls *.html

I'll leave you to figure out what is happening.

basename and dirname

The commands basename and dirname extract the respective parts from a full pathname. These are
commands and can therefore be used in any script, unlike the tricks at the beginning of the week
that are specific to tcsh.
% echo $fn

% echo `basename $fn`
todo.rtf

% echo `dirname $fn`
/Users/saruman/Documents/Essentials

Extended 'find'

Sometimes one needs to apply the same command to many files, and using 'find' is a good approach.
You can write the following script to make this simpler.

% cat xfind

#!/bin/sh
if [ "$1" = "" ]; then
echo "Usage: `basename $0` filetype [command to -exec]"
exit
fi

if [ "$2" = "" ]; then
find . -name "$1"
else
find . -name "$1" -exec $2 {} \;
fi

This script is written in /bin/sh, rather than /bin/tcsh. Don't forget to make the script file
executable.
% xfind "*.html"

will list all matching files in the current directory and subdirectories.
% xfind "*.html" less

will display them.

Mac to Unix

Here is a script to change a file with traditional Mac end of line characters to have Unix end of
line characters.

% cat unix2mac

#!/bin/sh

if [ "$1" = "" ]; then
echo "Usage: `basename $0` filename"
exit
fi

tr \\r \\n < $1 > $1.tmp ; mv $1.tmp $1

This script can be combined with the xfind script from Monday's tip:
% xfind "*.html" unix2mac

Unix to Mac

To complement Tuesday's script, here's the reverse:

% cat mac2unix

#!/bin/sh

if [ "$1" = "" ]; then
echo "Usage: `basename $0` filename"
exit
fi

tr \\n \\r < $1 > $1.tmp ; mv $1.tmp $1

Batch Rename

Here is a script to rename files in the current directory. It can be adapted to perform other
functions.

% cat rename

#!/bin/sh
#I'll leave the error checking to you....

for fn in *.$1
do
mv $fn `basename $fn $1`$2
done

The command:
% rename txt rtf

will change *.txt to *.rtf in the current directory.

Use Backquotes

Remember that enclosing a command in backquotes means we can pass its output as arguments to
another command.

For example, to edit all html files in a directory and its subdirectories (using Monday's 'xfind'
script):
% vi `xfind "*.html"`

(or vim, emacs, pico, bbedit...)

Enabling 'at'

The Unix command 'at' can be used to execute a given command at a given time. (But not on a given
date in OS X). However, a few tweaks must be made to OS X before it will work. These must be done
as root.

1. Uncomment (remove the # from) this line in /etc/crontab:
#*/5 * * * * root /usr/libexec/atrun

to check the queued 'at' jobs every 5 minutes.

2. Create the following directory if it does not already exist:
sudo mkdir /var/at/spool

3. Create an empty 'deny' file to allow all users to run 'at'
sudo touch /var/at/at.deny

(Or create 'at.allow' and add a list of the allowed users.)

(See week 33 for more scheduling commands.)

Using 'at'

Use the 'at' command to schedule a (sequence of) command(s) to be executed later. The following
example schedules the Clock application to be launched at 21:30 today. The commands to run can be
supplied either directly on the command line, or read from a file with the '-f' option.
% cat at-file
open /Applications/Clock.app

% at -f at-file 21:30
Job a010b400a.000 will be executed using /bin/sh

At 21:30 the Os X Clock application will be launched.

'atq' and 'atrm'

One can examine the list of jobs on the current user's 'at' queue with:

% atq
Date Owner Queue Job#
21:26:00 04/20/03 saruman a a010b400a.000

and delete jobs with:
% atrm a010b400a.000

passing the rather ugly job number.


More on 'at'

When the list of commands given by 'at' are executed, they are done so by the /bin/sh shell. The
environment variables and the current directory are re-created to match those from when the 'at'
command was issued.

If commands need to refer to your home directory use:
$HOME

Times can be specified relative to now with:
at -f at-file now + 10 minutes
at -f at-file now + 1 hour

The Unix Calendar

Use the 'calendar' command to query appointments and events. Create a plain text file (default
name 'calendar') containing a month, date, and event on each line.
% cat calendar

10 April Thursday's job
20 April Sunday's job
26 April Saturday's job

04/01 April Fool
04/20 Sunday's job
04/29 Dentist

Print the events for a given date (specified as MMDD) with the following commands:
% calendar -d 0410
10 April Thursday's job

% calendar -d 0429
04/29 Dentist

For today (20 April in this example), you can miss out the date specification:
% calendar
20 April Sunday's job
04/20 Sunday's job

If the file is other than 'calendar' in the current directory use the '-f' option:
% calendar -f /path/name/file-name -d 0429
04/29 Dentist

Unix has some pre-written calendar files in:
/usr/share/calendar

For example:

% calendar -f /usr/share/calendar/calendar.holiday -d 0421
04/21 San Jacinto Day in Texas
04/22 Arbor Day in Nebraska & Delaware
04/22 Oklahoma Day in Oklahoma
04/21 Tiradentes in Brazil

ping

We all know of the venerable 'ping' utility.

- check if a host (eg apple.com) is alive
- check if your network connection is in good order
- report on the general responsiveness of all things en route to a particular host (the 'time='
field)
- find the IP address of a named host

For example:
% ping apple.com
PING apple.com (17.254.3.183): 56 data bytes
64 bytes from 17.254.3.183: icmp_seq=0 ttl=45 time=165.922 ms
64 bytes from 17.254.3.183: icmp_seq=1 ttl=45 time=165.541 ms

The '-i' option can be used to change the interval between ping transmissions, and the '-c'
option used to send a specified number of pings. For example, to ping Apple once a minute
(specified as 60 seconds) for 30 minutes:
ping -i 60 -c 30 apple.com

(The name 'ping' comes from the sound sonar makes.)

ping tricks

Try pinging all the hosts on your local network to see which are running Rendezvous:
% ping 224.0.0.251

Ping the broadcast address of your local subnet to discover all the hosts on it, eg:
% ping 10.0.1.255

or
% ping 192.168.1.255

See if your router responds to:
% ping 0

and who responds to:
% ping 255.255.255.255

You can ping an AppleTalk host using 'appleping'.

traceroute

The traceroute utility shows the path or route to remote hosts by reporting on each 'hop' a
packet makes on its way through the Internet.

It's a long way to OSXFAQ.com.....

% traceroute osxfaq.com
traceroute to osxfaq.com (64.84.37.30), 30 hops max, 40 byte packets
1 valinor.zen (217.155.168.150) 21.229 ms 0.819 ms 0.353 ms
2 gauss-dsl1.wh.zen.net.uk (62.3.83.2) 19.78 ms 19.17 ms 20.999 ms
3 chrysippus-ve-131.wh.zen.net.uk (62.3.83.78) 22.385 ms 21.946 ms 22.521 ms
4 deleuze-ge-0-2-0.hq.zen.net.uk (62.3.80.81) 23.122 ms 22.709 ms 22.613 ms
5 suarez-so-0-0-0.te.zen.net.uk (62.3.80.62) 29.521 ms 29.589 ms 29.233 ms
6 lndnuk1icx1.wcg.net (195.66.224.105) 29.292 ms 29.049 ms 29.994 ms
7 nycmny2wcx2-oc12.wcg.net (64.200.87.149) 100.266 ms 100.063 ms 99.222 ms
8 nycmny2wcx3-oc48.wcg.net (64.200.87.78) 169.294 ms 169.843 ms 169.936 ms
9 chcgil1wcx3-oc48.wcg.net (64.200.240.37) 169.212 ms 170.859 ms 169.893 ms
10 dnvrco1wcx2-pos10-0.wcg.net (64.200.210.66) 169.951 ms 170.472 ms 170.171 ms

11 snfcca1wcx3-pos11-0.wcg.net (64.200.240.94) 170.758 ms 171.265 ms 171.403 ms
12 snfcca1wce1-pos3-0.wcg.net (64.200.199.238) 169.128 ms 170.197 ms 167.675 ms
13 snfcca1wce1-naciosystems-se.wcg.net (64.200.198.18) 168.242 ms 168.977 ms 169.301 ms
14 is10-0-0m.nov55c-75-003.nacio.com (64.84.1.9) 172.06 ms 171.154 ms 169.978 ms
15 if0-0-0m.nov55c-75-002.nacio.com (64.84.0.195) 172.116 ms 172.63 ms 171.409 ms
16 is11-1-1m.sfob-02.nacio.com (167.160.242.253) 173.154 ms 174.414 ms 173.645 ms
17 ife0-0m.sfoc-01.nacio.com (167.160.239.177) 173.634 ms 172.496 ms 173.64 ms
18 gw.scotthaneda.m-l.net (167.160.239.194) 178.804 ms 178.785 ms 177.821 ms
19 osxfaq.com (64.84.37.30) 179.822 ms 181.029 ms 182.053 ms

If traceroute fails, try the '-p' option with a big port number, eg:
% traceroute -v -p 35353 osxfaq.com

visual traceroute

Try this cool web site for a more graphical version of traceroute:
http://visualroute.visualware.co.uk/

netstat

netstat gives you lots of vital statistics on your network connections, routing tables, and such
like. It is useful to debug you connection, or just to probe around.

Some examples:

To list current connections, including hostnames and local port numbers:

% netstat
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 286 saruman.zen.55040 osxfaq.com.http ESTABLISHED
tcp4 0 284 saruman.zen.55039 osxfaq.com.http ESTABLISHED
tcp4 0 0 saruman.zen.55038 osxfaq.com.http CLOSE_WAIT
tcp4 0 0 localhost.1033 localhost.1006 ESTABLISHED
tcp4 0 0 localhost.1033 *.* LISTEN
udp4 0 0 saruman.zen.ntp *.*
udp4 0 0 localhost.ntp *.*
udp4 0 0 localhost.49155 localhost.1002
udp4 0 0 localhost.49154 localhost.1002
udp4 0 0 localhost.1033 *.*
Active LOCAL (UNIX) domain sockets
Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr
25b2540 stream 4782 0 0 0 0 0
25b23b8 stream 0 0 0 25b24d0 0..
......
......
......

To list your machine's routing tables:

% netstat -r
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire

default valinor.zen UGSc 36 433 en0
localhost localhost UH 8 26887 lo0
169.254 link#4 UCS 0 0 en0
217.155.168.144/29 link#4 UCS 1 0 en0
saruman.zen localhost UHS 0 96 lo0
valinor.zen 0:50:7f:6:82:34 UHLW 36 201 en0 1072

Internet6:
Destination Gateway Flags Netif Expire
UH lo0
fe80::%lo0 Uc lo0
link#1 UHL lo0
fe80::%en0 link#4 UC en0
0:3:93:b2:d6:4 UHL lo0
fe80::%en1 link#5 UC en1
0:30:65:6:59:b2 UHL lo0
ff01:: U lo0
ff02::%lo0 UC lo0
ff02::%en0 link#4 UC en0
ff02::%en1 link#5 UC en1

(Use option '-n' to display IP addresses instead of hostnames.)

You should see the default router, your router name/IP address and its MAC address, plus the
various names and addresses assigned to your own machine.

The command:
arp -a

shows you all the hosts on the local subnet for which your machine has a MAC to IP address
mapping. When connected to a router you will probably just see the details of your router.

Query Host Information with 'host'

To find the IP address of a host use:
% host mayo-family.com.
mayo-family.com has address 64.84.37.15
mayo-family.com mail is handled (pri=10) by mail.hostwizard.com

This command also returns the mail server for the domain.

It is advisable to end the domain name with a '.' (as in the above example) to ensure that it is
not interpreted relative to your own domain.

To do the reverse, i.e. lookup a hostname from an IP address, simply give the IP address:
host 17.254.3.183
183.3.254.17.IN-ADDR.ARPA domain name pointer apple.com

'host' Tricks

'host' has a whole host of options. The most useful is '-a' to grab all records and display in
verbose mode.

% host -a mayo-family.com
Trying null domain
rcode = 0 (Success), ancount=5
The following answer is not authoritative:

The following answer is not verified as authentic by the server:
mayo-family.com 79546 IN MX 10 mail.hostwizard.com
mayo-family.com 79546 IN A 64.84.37.15
mayo-family.com 79546 IN NS ns1.hostwizard.com
mayo-family.com 79546 IN NS dns2.intermag.com
For authoritative answers, see:
mayo-family.com 79546 IN NS ns1.hostwizard.com
Additional information:
mail.hostwizard.com 65327 IN A 64.84.37.6
ns1.hostwizard.com 80083 IN A 64.84.37.14
dns2.intermag.com 882 IN A 216.218.240.104
dns3.intermag.com 883 IN A 198.144.200.118

Note that:

* A = IP Address
* MX = mail server
* NS = name server

'dig' and 'nslookup'

The commands 'dig' and 'nslookup' are alternatives to 'host' and may be better suited to your
needs.

% dig apple.com.

; <> DiG 8.3 <> apple.com.
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->HEADER<- opcode: QUERY, status: NOERROR, id: 2
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 4
;; QUERY SECTION:
;; apple.com, type = A, class = IN

;; ANSWER SECTION:
apple.com. 26m56s IN A 17.254.3.183

;; AUTHORITY SECTION:
apple.com. 1d23h43m9s IN NS nserver.asia.apple.com.
apple.com. 1d23h43m9s IN NS nserver.euro.apple.com.
apple.com. 1d23h43m9s IN NS nserver.apple.com.
apple.com. 1d23h43m9s IN NS nserver2.apple.com.

;; ADDITIONAL SECTION:
nserver.asia.apple.com. 1d8h48m29s IN A 203.120.14.5
nserver.euro.apple.com. 23h13m18s IN A 17.72.133.64
nserver3.apple.com. 4d22h14m31s IN A 17.112.144.50
nserver4.apple.com. 4d22h14m31s IN A 17.112.144.59

;; Total query time: 47 msec
;; FROM: saruman.mayo-family.com to SERVER: default -- 212.23.8.1
;; WHEN: Sun May 4 20:47:50 2003
;; MSG SIZE sent: 27 rcvd: 252

% nslookup apple.com.
Server: ns0.zen.co.uk
Address: 212.23.8.1

Non-authoritative answer:
Name: apple.com
Address: 17.254.3.183

As ever, use the 'man' command to discover more.

'whois'

The 'whois' command queries whois.internic.net to discover the registrar, owner, and name servers
for a domain.

% whois mayo-family.com

Whois Server Version 1.3

...

Domain Name: MAYO-FAMILY.COM
Registrar: BULKREGISTER.COM, INC.
Whois Server: whois.bulkregister.com
Referral URL: http://www.bulkregister.com
Name Server: DNS2.INTERMAG.COM
Name Server: NS1.HOSTWIZARD.COM
Name Server: DNS3.INTERMAG.COM
Status: ACTIVE
Updated Date: 27-feb-2003
Creation Date: 19-jul-2001
Expiration Date: 19-jul-2003
...

If the domain was not registered by internic, a second whois is necessary. In this case a query
to the registrar's whois server is required using option '-h'.

% whois -h whois.bulkregister.com mayo-family.com
Adrian Mayo
54 Greenland Mills
Bradford-on-Avon, England BA15 1BL
UK


Domain Name: MAYO-FAMILY.COM

Administrative Contact:
Carlton Thomas domreg@gifford.co.uk
Gifford Internet Services
14 Tyrrel Way, Stoke Gifford
Bristol, England BS34 8UY
UK
Phone: +44 117 9397722
Fax: +44 117 9397733
Technical Contact:
Carlton Thomas domreg@gifford.co.uk
Gifford Internet Services
14 Tyrrel Way, Stoke Gifford
Bristol, England BS34 8UY
UK
Phone: +44 117 9397722
Fax: +44 117 9397733

Record updated on 2001-07-19 11:28:07
Record created on 2001-07-19
Record expires on 2003-07-19
Database last updated on 2003-05-04 15:58:47 EST

Domain servers in listed order:

DNS2.INTERMAG.COM 216.218.240.104
DNS3.INTERMAG.COM 198.144.200.118
NS1.HOSTWIZARD.COM 64.84.37.14

'whois' for non TLD

To do a 'whois' query on a non Top Level Domain, for example '.co.uk', one has to use another
'whois' service.

% whois -h whois.nic.uk bbc.co.uk

Domain Name:
bbc.co.uk

Registrant:
British Broadcasting Corporation

Registrant's Agent:
The British Broadcasting Corporation t/a BBC Technology [Tag = BBC]

Relevant Dates:
Last updated: 13-Mar-2001


Name servers listed in order:
ns.bbc.co.uk 132.185.132.21
ns1.thny.bbc.co.uk 38.160.150.21
ns1.thdo.bbc.co.uk 212.58.224.21
ns1.bbc.co.uk 132.185.132.11

WHOIS database last updated at 21:20:01 04-May-2003

And if anyone knows of a reliable way to determine the whois address for the various TLD and
country domains, let me know....

Sub Shells

To execute a command, or commands, in a new shell, bracket the commands.

To execute several commands on one line, separate them with a semicolon.

For example, compare the following two commands that make use of both of the above.

% (cd /; pwd); pwd
/
/Users/saruman

% cd / ; pwd ; pwd
/
/

Redirection

This applies to the 'tcsh' shell.

To redirect standard output use '>'. Eg:
ls -al * > list

Output and error messages write to different streams. To redirect both standard output and
standard error use '>&". Eg:
ls -al kfhsdhdjhsjkhf >& list

Redirection 2

The 'tcsh' shell does not provide an obvious way to redirect standard output and error to
different files. Use the following trick to do so:
(ls -al filename > list ) >& list.err

Redirection 3

The 'tcsh' does not provide an obvious way to redirect standard error, but not standard out. Use
the following trick to do so:
(ls -al filename > /dev/tty) >& list.err

No Clobber

If you wish to avoid accidentally overwriting files with redirection, set the 'tcsh' 'noclobber'
variable. For example:

% set noclobber
% ls
.....list list.err ....

% ls > list
list: File exists.
% ls >! list
%

Notice the use of '!' to temporarily override 'noclobber'.

Killing Time

To kill an application (or process) from the command line, we first need to find its Process ID
or PID. For example:

% ps -axww | grep -i clock
15048 ?? S 0:00.65 /Applications/Clock.app/Contents/MacOS/Clock
15054 std R+ 0:00.00 grep -i clock

and then issue the 'kill' command:
% kill 15048

Note the use of option 'ww' which ensures that the output from 'ps' is not truncated losing the
name we are grepping for.

A PID of -1 broadcasts the signal to all process for which you have the necessary permissions to
kill.

Alternatively, we can use:
killall Clock

which kills by (case sensitive) name. Be careful as this command kills all process that match the
given name.

The '-m' option of 'killall' takes a regular expression as the process name, and kills all those
processes that match it.

Many Ways to Kill

The 'kill' and 'killall' commands send a 'TERM' signal to the process, which is a request to that
process to terminate.

Other signals are available, including

-KILL - make the process an offer it can't refuse. This always kills the process.

-HUP - tell the process to restart. This is useful to restart demons (like configd and lookupd),
as they will re-read their configuration files on receipt of a -hup signal.

The command:
% kill -l

lists all the available signals, and:
% man sigaction

gives more detail.

The Killing Script

Here is a handy script to kill processes by name:
% cat killer
#!/bin/sh
kill $(ps auxww | grep -i $1 | grep -v $1 | awk '{print $2}')

% killer clock
*gone!*

Remember to make the script executable.

Find Your Victim

Many Unix demons kindly leave their address. Check out:

% ls /var/run/
AppleFileServer.pid cron.pid mDNSResponder.pid ntp.drift sshd.pid xinetd.pid
StartupItems davlocks named.pid ntpd.pid sudo
autodiskmount.pid httpd.pid ndc pppconfd syslog
automount.pid inetd.pid netinfo_local.pid proxy syslog.pid
configd.pid lookupd.pid niconfig_local.xml resolv.conf utmp

These should be independent of the name of the demon. For example, www servers should report
their pid at 'httpd.pid'.

You can make a handy alias to restart a demon, such as:
% alias nfs-mount 'sudo kill -HUP `cat /var/run/automount.pid`'
% alias nfs-export 'sudo kill -HUP `cat /var/run/mountd.pid`'

which will restart the NFS client and server respectively after new shares and connections have
been added.

The Hanging Script

Here is a handy script to restart a demon:

% cat hanger
#!/bin/sh
if [ "$1" = "" ]; then
echo "List of demons:"
cd /var/run
ls -1 *.pid | sed -e 's/\.pid//'
else
echo "restarting demon $1"
sudo kill -HUP `cat /var/run/$1.pid`
fi

It will list the demons:

~ % hanger
List of demons:
AppleFileServer
autodiskmount
automount
configd
cron
httpd
inetd
lookupd
mDNSResponder
named
netinfo_local
ntpd
sshd
syslog
xinetd

and kill them:
% hanger lookupd
restarting demon lookupd

Remember to make the script executable.

Start and Stop Apache

To start and stop apache from the command line use the Apache control utility:
% sudo apachectl start
/usr/sbin/apachectl start: httpd started

% sudo apachectl stop
/usr/sbin/apachectl stop: httpd stopped

To cause Apache to be started when your Mac boots, edit the file:
/etc/hostconfig

as root, and change
WEBSERVER=-NO-

to
WEBSERVER=-YES-

Restart Apache

If you change Apache's configuration files, you must restart Apache to cause it to re-read them.
To do this gracefully (without disturbing existing connections on a live server) use:
% sudo apachectl graceful
/usr/sbin/apachectl graceful: httpd gracefully restarted

Change WebServer Root

By default, Apache on Mac Os X servers from /Library/WebServer/Documents. If you wish to change
this to another directory, like ~/Sites, edit Apache's configuration file:
% sudo pico /etc/httpd/httpd.conf

Replace this line (line 363?):
DocumentRoot "/Library/WebServer/Documents"

with:
DocumentRoot "/Users/your-user-name-here/Sites"

and this line (line 388?):
<Directory "/Library/WebServer/Documents">

with:
<Directory "/Users/your-user-name-here/Sites">

The Apache Manual
Access the Apache on-line manual with the URL: http://localhost/manual/

The Apache Log Files

Apache writes logging information to:
/private/var/log/httpd/error_log

and:
/private/var/log/httpd/access_log

Examine the error log if you experience problems in setting up the server. Examine the access log
to see who is viewing your site.

You can change the amount of logging information written to these files. For example, to add the
referrer (the site which linked to your site) and the user agent (the browser) to the log file,
do the following.

Edit the apache configuration file:
% sudo pico /etc/httpd/httpd.conf

Search for:
CustomLog "/private/var/log/httpd/access_log" common

and replace with:
CustomLog "/private/var/log/httpd/access_log" combined

List the Shell Built-ins

The tcsh shell has many useful built-in commands (commands that are part of the shell and do not
live as separate executables in /bin etc).

To list all builtins:

% builtins
: @ alias alloc bg bindkey
break breaksw builtins case cd chdir
...

If the same command exists as both a separate executable and a shell built-in, the shell built-in
will be executed. To override this, precede the command with '\'.
% which which

% which \which
/usr/bin/which

and try this:
% \which which

The Command Hash Table

When the shell is given a command to execute, it will search its own built-in commands first,
then all the directories specified in the PATH or path environment/shell variables. To see the
path, issue either of:
% echo $PATH
% echo $path

In order to speed up this search process, the shell searches these directories when it first
starts, and caches the information in memory. If you add a new command or shell script to one of
these directories, the shell will not recognize it. To get the shell to refresh its cache issue
the built-in command 'rehash'
% rehash

No response means all is ok.

Try this:
% hashstat

umask

The umask built-in command tells the file system the default permissions to apply to newly
created files and directories. The umask can be set in your tcsh startup file, or applied just to
a partiaular Terminal session.

Without a 'umask' directories are set to:
drwxrwxrwx 2 saruman staff 68 Jun 9 11:16 test-dir

and files to:
-rw-rw-rw- 1 saruman staff 0 Jun 9 11:16 test-file

The umask is usually set to 022:

% umask 022
% touch test-file
% ls -al test-file
-rw-r--r-- 1 saruman staff 0 Jun 9 11:22 test-file

Give write access to your group:

% rm test-file
% umask 002
% touch test-file
% ls -al test-file
-rw-rw-r-- 1 saruman staff 0 Jun 9 11:23 test-file

For shared files:

% rm test-file
% umask 000
% touch test-file
% ls -al test-file
-rw-rw-rw- 1 saruman staff 0 Jun 9 11:23 test-file

For private files:

% rm test-file
% umask 077
% touch test-file
% ls -al test-file
-rw------- 1 saruman staff 0 Jun 9 11:23 test-file

Be Nice

The 'nice' shell built-in is used to execute a command with a priority greater or less than the
default priority. A negative nice value ensures that command gets a greater share of the CPU
(high prioriry), and a positive value ensures the command see less of it (low priority).

Normally, a command is issued with a nice value of 4:

% ./test.sh

% ps -axlhww | grep test.sh
UID PID PPID CPU PRI NI ...
501 9997 3457 0 27 4 ...

showing a priority (PRI) of 31-4=27 and a nice (NI) of 4

To issue a low-priority command use:

% nice +20 ./test.sh
501 10036 3457 0 11 20 ...

showing a priority of 31-20=11 (low) and a nice of 20 as specified

To increase the priority, one must be a super-user:

% sudo -s
# nice -10 ./test.sh
0 10142 10130 0 41 -10 ...

Check the (not built-in) command 'renice', which adjusts the priority of running processes:
% man renice

exec

The built-in command 'exec' allows one to run a command in place of the shell. The shell starts
the command and exits.

This can be useful to start a command that runs as a daemon and does not require its parent
shell.
% exec my-daemon

Shell Startup

First off, apologies for the lack of tips last week - I'm fit and well again for this week.

When the 'tcsh' shell starts up it reads and executes several script files. These you are free to
change (or create) if you wish to customise your own command line environment. They are:
/etc/csh.cshrc
/etc/csh.login

~/.tcshrc
~/.login

You may add Unix and 'tcsh' shell commands to these files. The first two files are global to all
users, and the second two are per-user.

tcshrc and cshrc

For backward compatibility with the 'csh' shell (from which the 'tcsh' shell was derived), the
script file:
~/.cshrc

may or may not be executed when the 'tcsh' shell starts up.

In the absence of '~/.tcshrc', '~/.cshrc' is executed. Once you create '~/.tcshrc', '~/.cshrc' is
no longer executed. Therefore, never rely on having both of these files executed. Choose one,
preferably '~/.tcshrc', and stick to it even if you are advised to add lines to the other. The
two are interchangeable.

Login and Interactive Shells

What are login and interactive shells?

Opening a Terminal window starts an interactive login shell.

Executing a shell script starts a non-interactive non-login shell.

Typing 'tcsh' at the command line spawns an interactive non-login shell.

Login shells differ in that they read and execute:
/etc/csh.login
~/.login

whereas non-login shells do not.

Interactive shells are attached to a Terminal window so you can type commands into them.
Non-interactive shells take their commands from a script.

Some terminals (such as xterm that runs in X11) by default do not start a login shell. To get
xterm to do so, and therefore behave more like Apple's Terminal, add this to ~/.Xdefaults:
# xterm
XTerm*.LoginShell: True

Login-only Commands

Optimize the shell startup sequence by using the login-shell-only startup files:
/etc/csh.login
~/.login

In these files set up aspects of the environment that persist, such as environment variables.
This applies especially to commands like:

setenv PATH ${PATH}:xxx

which add to an environment variable. A command like this must not be executed more than once
otherwise your $PATH will grow and grow.

Interactive-only Commands

Following on from Wednesday's tip...

Optimize the shell startup sequence. Many commands in ~/.tcshrc are not necessary for
non-interactive shells. Most in fact, such as setting shell variables and aliases, and the
bindkey command. Filter these commands out for non-interactive shells.

Include this statement near the top of ~/.tcshrc
if ($?prompt) set interactive

Then surround all interactive-only commands with:

if ($?interactive) then

... interactive only commands are here...

endif

'xargs'

Previous tips (from weeks 1, 13, and 31) have dealt with 'find'. This weeks tips expose more
tricks using this powerful command in combination with the 'xargs' command.

Week 13 gave an example using the '-exec' option of 'find' to execute a command on each file
found:
% find . -iname "*.ws" -exec grep -il warning {} \;
./week13/thursday.ws
./week5/friday.ws
./week5/thursday.ws
./week9/monday.ws

The same effect can be achieved by piping to 'xargs':
% find . -iname "*.ws" | xargs grep -il warning
./week5/friday.ws
./week5/thursday.ws
./week9/monday.ws

'xargs' is a Unix command in its own right and is not specific to 'find'. It forms a command from
its parameters, which will include the output from 'find', thus...
grep -il warning file1 file2 file3 file4 .......

'xargs' vs '-exec'

Referring back to Monday's tip...
% find . -iname "*.ws" -exec grep -il warning {} \;

versus:

...some may tell you that piping to 'xargs' is less efficient than using the built-in '-exec'
option. This is not so, and in fact the latter example will run at least ten times as fast as the
former.

Why?

Because '-exec' will execute the 'grep' command once for each file found . In this example, over
200 files matched "*.ws". 'xargs' on the other hand absorbs all the arguments and passes them to
'grep' in one go.

'xarg' option

Monday's example...

...works because 'grep' can take a long list of files to process. If a command takes only one
file at a time, use option '-n1'.
% find . -iname "*.ws" | xargs -n1 my-simple-script

Option '-t' is useful (trace) - it and causes 'xargs' to echo to standard output the command it
has formed and will execute.
% find . -iname "*.ws" | xargs grep -t -il warning
grep -il warning ./index.ws ./week1/friday.ws ./week1/monday.ws
./week1/thursday.ws ./week1/tuesday.ws ./week1/wednesday.ws
./week10/friday.ws ./week10/monday.ws ./week10/thursday.ws
...
./week8/friday.ws ./week8/monday.ws ./week8/thursday.ws
./week8/tuesday.ws ./week8/wednesday.ws ./week9/friday.ws
./week9/monday.ws ./week9/thursday.ws ./week9/tuesday.ws

./week9/wednesday.ws
./week5/friday.ws
./week5/thursday.ws
./week9/monday.ws

Too Many Files?

If 'find' finds many, many files, the resulting command line formed by 'xargs' may be too long.
Limit this by using option '-n' which states the maximum number of parameters that 'xargs' will
pass to the command (in this example to 'grep')

% find . -iname "*.ws" | xargs -n100 grep -il warning

...causes 'xargs' to call 'grep' once it has 100 filenames. If 350 filenames were found, 'xargs'
would call 'grep' four times.

Filenames With Spaces

If filenames have spaces, the 'find | xargs' combination will fail...
% ls -1
a file name.doc
b file name.doc

% find . -iname "*.doc" | xargs grep -il "Dear Sirs"
grep: ./a: No such file or directory
grep: file: No such file or directory
grep: name.doc: No such file or directory
grep: ./b: No such file or directory
grep: file: No such file or directory
grep: name.doc: No such file or directory

Use option '-print0' with 'find'. 'find' will print the pathname of the current file, followed by
an ASCII NUL character (character code 0). Next, use option '-0' with 'xargs' so it will use NUL
(``\0'') instead of whitespace as the argument separator.
% find . -iname "*.doc" -print0 | xargs -0 grep -il "Dear Sirs"
./a file name.doc

Get Vim

Get vim. Mac OS X ships with 'vi'. Vim is an improved version of 'vi', and is available in three
flavours: pure Terminal, X11, or Carbon GUI.

The main vim homepage is: http://www.vim.org/

The Mac downloads page is: http://www.vim.org/download.php#mac

For the Carbon GUI: http://macvim.swdev.org/OSX

Marc Liyanage maintains command line versions for pure Terminal and X11:
http://www.entropy.ch/software/macosx/welcome.html#vim

This week's hints are not a vim tutorial for the absolute beginner, but some handy tips for those
who have got beyond the 'esc' and 'i' keys.

Cursor Movement in Vim

Learn basic cursor movement keystrokes. The most useful are:
{ back to start of paragraph
( back to start of sentence
- back to start of previous line
^ back to start of line
b back to start of word
w forward to start of word
$ forward to end of line
+ forward to start of next line
) forward to start of next sentence
} forward to start of next paragraph
G to end of file

(plus the obvious cursor keys).

Cut, Copy, and Paste in Vim

Use cut, copy, and paste:

d deletes (cut)
y yanks (copy)
p pastes (below the current line, and P pastes above it)

y and d must be followed by a cursor movement key (see Tuesday's tip), and will copy or delete
from the cursor to the move target.

Examples:
d$ deletes to the end of the line
d) deletes to the end of the sentence
(y) copies the entire sentence, no mater where the cursor is within the sentence

Shortcuts:
dd is short for delete current line
yy is short for yank current line

Use the 26 buffers in Vim

Vim has 26 named buffers (a to z), in addition to the unnamed general buffer. Precede a command
by a buffer name, and the command will dump into or from that buffer.

Continuing from Wednesday's example:
"ay) copies to the end of the sentence and places the text into named buffer "a
"ap pastes the text that is in buffer "a

Note that the unnamed buffer is always written to in addition to any named buffer.

You can append text to a buffer by naming it with an uppercase letter.
"Ad$ will cut to the end of the line, and append the cut text to any that was already in buffer
"a

Bookmarks in Vim

Vim has 26 named bookmarks (a to z). Place a bookmark with ma (to mz) and recall it (move the
cursor back to it) with 'a (to 'z).

Copy or paste a portion of text with:
<move to one end of desired selection>
ma
<move to other end of selection>
"by'a

This will additionally copy the section into named buffer "b.

Additional:
'' (two single quotes) toggles between current and last cursor position
'a moves to the start of the marked line
à moves to the marked character position within the marked line

Set bookmarks across different files with the 36 interfile books marks mA (to mZ, and m0 to m9).
Recall as usual with 'A or À (to 'Z, '9).

...Sum File Sizes

The tips for this week are culled from my replies to the many emails I receive asking: "How do I
do such and such?"

How do I find the total size of all Photoshop files (*.psd) in my home directory?
% find ~ -iname "*.psd" -print0 | xargs -0 du -ch
15M ./Pictures/complete/ferdi-coll3.psd
30M ./Pictures/complete/ferdi-cool.psd
1.7M ./Pictures/complete/ferdi-gala.psd
...
600k ./Pictures/web-site/misc/web5.psd
72k ./Pictures/web-site/osxfaq/banner1.psd
48k ./Pictures/web-site/osxfaq/banner2.psd
96k ./Pictures/web-site/osxfaq/btn-.psd
28k ./Pictures/web-site/osxfaq/end.psd
316M total

There! - 316M.

The Mac OS X standard 'du' command does not understand option '-h' (human readable output). Use
-k instead to see file sizes in K-bytes.

GNU package 'fileutils' contains superior versions of many basic commands, including 'du' and a
colour version of 'ls'. It is most easily got using Fink.

...Seek Out Large Directories

How do I report on all Directories who's contents exceed 100 Meg?

Previous week's tips have covered a similar question to report on all individual files that
exceed a given size. For example, find all files that exceed 20 Meg:
% find . -size +40000

To apply this to directories use:
% du -k ~ | egrep "^[0-9]{6,}"
139252 /Users/saruman/Library
267020 /Users/saruman/Pictures/complete
357432 /Users/saruman/Pictures/iPhoto Library/2001
195788 /Users/saruman/Pictures/iPhoto Library/2003/01
1149224 /Users/saruman/Pictures/iPhoto Library
1548924 /Users/saruman/Pictures
1902804 /Users/saruman

This reports on all directories who size in K-bytes is six digits or more (controlled by {6,}).

...Delete Blank Lines

How do I delete empty lines from a file?

Using 'sed':
% sed '/^$/d' in-file > out-file

Using 'tr':
% tr -s \\n < in-file > out-file

If the file has blank lines containing spaces:
% sed '/^[ ]*$/d' in-file > out-file

If the blank lines have tabs too:
% sed '/^[ \t]*$/d' in-file > out-file

BUT, OS X 'sed' command does not seem to understand '\t' as meaning tab.

...Treat a Command as a Function

This addresses a generic question, best illustrated by an example.

The command 'tr' translates or strips characters from the contents of a file. Like most commands,
it reads and writes files, or standard IO. For example, to strip spaces use:
% tr -d " " < in-file > out-file

How can we use a command like 'tr' in a shell script, to strip spaces from the contents of a
variable? The effect we want is as if we could say:
var = tr ($var);

The secret is to stream the value of the input parameter, and capture the output stream, thus:

In tcsh use:
set var=ècho $var | tr -d " "`

In sh use:
var=$( echo $var | tr -d " " )

...Remove Spaces from Filenames

How can I eliminate spaces in all my filenames?

This 'sh' shell script will do the trick, providing no two filenames in a directory differ by
only spaces, and no directory names have spaces.

% cat sh2
#!/bin/sh
target_fn=$( echo $1 | tr -d " " )
if [ ! "$1" = "$target_fn" ]; then
echo mv "$1" "$target_fn"
mv "$1" "$target_fn"
fi

% ls -R1
.:
dirname
file name
file2 name
sh
sh2
tsh

dirname:
file2 name
file3name

% find . -print0 | xargs -0 -n1 ./sh2
mv ./dirname/file2 name ./dirname/file2name

mv ./file name ./filename
mv ./file2 name ./file2name

% ls -R1
.:
dirname
file2name
filename
sh
sh2
tsh

dirname:
file2name
file3name

Grep for Several Strings (OR)

How do I use grep to search for files containing string1 OR string2?
% grep -e "apple" -e "microsoft" *.txt

'-e' introduces a regular expression to 'grep', which searches each file for an occurrence of any
of the regular expressions listed.

Using extended 'grep' ('egrep' or 'grep -E') one can specify:
% egrep "(microsoft|apple)" *.txt

Extended 'grep' understands extended regular expressions. Standard 'grep' only understands
'regular' regular expressions.

Grep for Several Strings (AND)

How do I use 'grep' to search for files containing string1 AND string2?
% grep -l "apple" *.txt | xargs grep -H "microsoft"

'-l' (letter ell) in the first 'grep' causes it to list just filenames, which are then passed to
the second 'grep'. '-H' tells the second 'grep' to print the filename for each match.

Perl and 'awk' will do this too, and I plan to cover them in a Learning Centre tutorial.

Is there a more elegant way to do this with 'grep'?

Trim Log Files

How can I trim my log files to contain just the last 1000 lines?

The OS X 'periodic' scripts in /etc/periodic do a good job of taming growing log files. They are
run via /etc/crontab.

This command trims a log file to just the last 1000 lines:
% tail -n1000 x.log > x.log.tmp ; mv x.log.tmp x.log


Find Files of a Given Type

How do I locate all Bourne shell scripts, or all text files, in my home directory?

We all know how to use 'find' to do this based on file extension. But what about file contents,
which was what was required for this question.

Use the 'file' command, combined with 'find' and 'grep':

% find . -print0 | xargs -0 file | egrep "(Bourne|Tenex)"
./sh: Bourne shell script text
./sh2: Bourne shell script text
./tsh: Tenex C shell script text

will find all 'sh' and 'tcsh' scripts. The magic is specified by /etc/magic.

Restart a Daemon

Week 45, Friday gave a script to do just this. This doesn't always work however, because some
cheeky little daemons write additional information to their .pid file:
% sudo cat /var/run/sendmail.pid
11024
/usr/sbin/sendmail -L mta -bd -q30m

Change the script to use 'head' instead of 'cat', so as to cut the additional lines:
...
sudo kill -HUP `head -n1 /var/run/$1.pid`
...

Edit Many Files with vim

Use the 'new' command to split the screen to edit many files in the same Terminal window.
:new

splits the screen in half and creates a new edit buffer, and:
:new file-1

splits and reads 'file-1' into the new edit buffer.

Use ctrl-WW (or ctrl-W ctrl-W) to toggle between buffers/files.

Vertically Split Screen in Vim

Vim can split a window vertically too, which is useful for comparing two files side by side.
:vert new [file-1]

will split the screen vertically and optionally read file-1 into the second buffer.

Diffing Two Files with Vim

The command:
% vimdiff file-1 file-2

will start vim with two files, create a vertical split, and mark all differences between the
files.

Alternatively, if you are editing file-1 already, use:
:diffsplit file-2

or
:vert diffsplit file-2

to load file-2 and view the differences.

Batch Edit Many Files with Vim

The command:
% vim f1 f2 f3 f4 f5

will cause vim to read the five named files, and start editing f1.

To move between files use:
:n - next file in list
:prev - previous file in list

:wn or :wprev is necessary to write out changes and then edit the next file, if the current file
has changed.

List the buffers with:
:buffers

and jump to a specific buffer (eg buffer 2) with:
:b2

Vim Shortcuts

For those familiar with vim shorcuts, here are some that can be added to your .vimrc file.

Submitted by Mike Schienle

" typos
ab teh the
ab pritn print

" key maps
" these rely on blocks marked as a (begin) and b (end)

" \D delete everything between marks a and b
map \D :'a,'bd<CR>

" backslash P - put buffer a at current cursor location
map \P :pu a<CR>

" backslash Y - yank a-b into buffer a
map \Y :'a,'by a<CR>

" star c - convert to lowercase
map *c :'a,'bs/\<[A-Z]/\l&/g<CR>

" star C - convert to uppercase
map *C :'a,'bs/\<[a-z]/\u&/g<CR>

Setting Variables and Environment Variables

This week's tips compare how some common tasks look in tcsh versus sh/bash. This is a handy
reference if, like me, you are continually flipping between the two and get the syntaxes mixed
up.

How to set variables and environment variables:

tcsh sh/bash
set var = value % var=value
setenv ENV value % ENV=value; export ENV

In bash (additional to sh) variables can be declared and marked as exported and/or read only:
declare [-options] var=value
-r is read only
-x is exported (environment variable)

% declare -rx ENV=cant_change

Redirection and Pipes

For tcsh:

stdout cmd >fn
stderr (>/dev/tty) >&fn2
both cmd >& fn
both diff files (cmd >fn) >&fn2
stdin cmd <fn

to append >>
no clobber >!

pipe stdout cmd | cmd2
pipe both cmd |& cmd2

For sh/bash:

stdout cmd >fn
stderr cmd 2>fn
both same file cmd >fn 2>&1 (cmd &>fn - bash)
both diff files cmd >fn 2>fn2
stdin

to append >>fn
no clobber >|fn

pipe stdout cmd | cmd2
pipe both cmd 2>&1 | cmd2

close stdout cmd >&-
close stderr cmd 2>&-
close stdin cmd <&-
open file cmd <>fn

Startup Files

The following files are executed when the shell starts up in interactive mode, and in the order
given. Both login and non-login shell startups are given.

tcsh login shell:
% tcsh -l
/etc/csh.cshrc
/etc/csh.login
~/.tcshrc
~/.login

bash login shell:
% bash --login
/etc/profile
~/.bash_profile

tcsh non-login shell:
% tcsh
/etc/csh.cshrc
~/.tcshrc

bash non-login shell:
% bash
~/.bashrc

Control Constructs 1 (Choice)

The scripting language of tcsh looks very much like the 'c' programming language (hence the 'c'
in tcsh) while sh/bash does not.

'if' construct:

#!/bin/tcsh
if ("$1" == "positive") then
echo "Yes"
else if ("$1" == "negative") then
echo "No"
else
echo "Not sure"
endif

#!/bin/sh
if [ "$1" = "positive" ]; then
echo "Yes"
elif [ "$1" = "negative" ]; then
echo "No"
else
echo "Not sure"
fi

'switch/case' construct:

#!/bin/tcsh
switch ("$1")
case "positive":
echo "Yes"
breaksw

case "negative":
echo "No"
breaksw

default:
echo "Not sure"
breaksw
endsw

#!/bin/sh
case "$1" in
"positive")
echo "Yes"
;;

"negative")
echo "No"
;;

*)
echo "Not sure"
;;
esac

Control Constructs 2 (Looping)

The scripting language of tcsh looks very much like the 'c' programming language (hence the 'c'
in tcsh) while sh/bash does not.

'for' loops:

#!/bin/tcsh
foreach word (hello goodbye au-revoir)
echo $word
end

#!/bin/sh
for word in hello goodbye au-revoir
do
echo $word
done

'while' loops:

#!/bin/sh
n=0
while [ ! $n = 10 ]
do
echo $n
n=$(expr $n + 1)
done

#!/bin/tcsh
set n = 0
while ($n != 10)
echo $n
set n = èxpr $n + 1`
end

Oddly-Named Files

If you have a filename that shows up like:
% ls Z* Z???

It may contain non-standard characters. Use tabbed completion to reveal the filename:
% ls Z<hit tab key>
% ls Zb^D"

(Note that the filenames don't appear so odd by the time they reach your browser.)

The file can be renamed with:
% mv Z<hit tab key>
% mv Zb^D" z-file

If the type-able portion of the file in not unique:
% ls d<hit tab key>
DB)b^D"o#?b^B, DB4#^ Danger

If the type-able portion of the file in not unique:
% ls d<hit tab key>
DB)b^D"o#?b^B, DB4#^ Danger

use the inode number to identify the file.

% ls -i
590795 Danger 579654 bvt 563546 page2.htm 578980 tvb
590442 D??????????? 560810 Cantata.txt 563547 page3.HTML 529632 users
590441 D??#^
...

then use 'find' to rename it:
% find . -inum 590442 -exec mv {} D-file \;

'fsck' it

Sometimes, repairing the disk can help make odd ghostly files a little more solid.

To do this boot the machine into single user mode by holding down command (apple)-S.

To repair the boot volume type:

# fsck -y

To repair other partitions type:
# fsck_hfs -y /dev/disk0sX

where X is the slice (partition) number identified by:
% pdisk
type L
read the partition number from the left column against the volume name
type q to quit.

% mount -uw /
% reboot

Deeply Nested Directories

Commands (such as fsck) can sometimes fail due to overly nested directories. Such nesting can be
created by buggy scripts.

Look for this on the file-system with:
% sudo find / -mindepth 30

(to find nesting greater than 30).

1 - Nested Directories You Don't Own

This thing about being able to delete any file - it only applies if you have write permission to
the directory containing the file. So, if a file resides within a non-writeable directory within
the trash, you can't delete it. And you can't delete the directory either because it's not empty.

2 - Immutable Files

Files can be made immutable. An immutable file has the Immutable flag set - one of the file flags
that exist over and above the attributes (permissions and owners) you see when you list files. A
special command called 'chflags' is used to set and unset flags.

3 - Open Files

If a file is currently in use by some application (open) then it cannot be deleted regardless of
your permissions for that file.

4 - The nightmare Scenario

A file may have the System Immutable flag set. This means that you cannot change it in any way;
no user can change it. Root cannot change it. And get this - root can't even unset the flag. So
even root can't delete it. (But it can be deleted.....)

Practical Advice - Trash Destruction

Here are four solutions to the four scenarios listed above.

1 - Nested Directories You Don't Own

One solution is to apply write-permission to all files in the trash. Bit if you do not own some
of the files you have to run as root. And if you run as root, you may as well simply delete the
lot. So here goes.

% sudo rm -ri ~/.Trash

This will zap all files except those that are immutable or in use.

'sudo' executes the command as root and therefore overrides all permissions
'rm' removes files
'-r' does it recursively - ie to all nested files and directories
'-i' is confirm mode - you will be prompted for each file to delete, and I suggest to use this as
a safety precaution. If you have many, many files tedium may set in, so you can replace the '-i'
with '-f' (force).

2 - Immutable files

You can check which, if any, files have the immutable flag set by specifying the '-o' option to
'ls'.

ls -alo

drwx------ 15 saruman staff - 510 Jan 6 23:35 osxfaq

-rw-r--r-- 1 saruman staff uchg 0 Feb 20 14:04 user-imutable-set

'uchg' indicates that the User Immutable Flag is set. 'schg' indicates that the System Immutable
Flag is set (see point 4 below).

Clear the immutable flag of all files, then delete:

% sudo chflags -R nouchg ~/.Trash
% sudo rm -ri ~/.Trash

'chflags' changes a file's flags settings.
'nouchg' clears the user immutable flag ('uchg' would set it)
'-R' does it recursively.

3 - Open Files

You can determine which files in the trash are open, and which application has them open, with
the 'fstat' command:

% ls -al ~/.Trash
total 0
drwx------ 5 melkor staff 126 Jun 25 16:43 .
drwxr-xr-x 24 melkor staff 772 Jun 25 16:32 ..
-rw-r--r-- 1 melkor staff 0 Jun 25 16:32 xxx
-rw-r--r-- 1 melkor staff 0 Jun 25 16:43 yyy
-rw-r--r-- 1 melkor staff 0 Jun 25 16:43 zzz

% fstat ~/.Trash/*
USER .CMD PID FD INUM MODE SZ|DV R/W MOUNT NAME
melkor vi 731 3 368860 -rw-r--r-- 0 r / /Users/melkor/.Trash/xxx

...that's the one - 'vi' is editing 'xxx'...

Close the offending files, then attempt to empty the trash again.

If you are unable to close all open files, resort to the method described in the GUI approach.

Tell Me More...
warning

sudo

sudo (super user do) is a Unix command that allows you to run another command as root. You must
enter your admin password (not the root password) when prompted.

A user without administrative privileges cannot use 'sudo'

If you were to accidentally put a space between '.' and 'Trash' :

%rm -rf ~/. Trash

'rm' would apply to both '~/.' (your home directory) and 'Trash' ( a non-existent directory). You
would wipe every single file from your home directory. Oops!

That's one reason why I suggest the '-i' option

% rm -ri ~/.Trash

so you get a prompt for each file. Hopefully you will spot your mistake on the first confirmation
and avoid loosing anything

4 - The Nightmare Scenario

If all of the above methods fail, the offending files my have the System Immutable flag set. Such
files are locked Fort Knox style. The System Immutable flag cannot be changed by root. Only by
super-root - the super-duper-user. What!?!

When your Mac is up and running in multi-user mode (the normal operating mode) it is running at
level 1. Some operations even root can't do at level 1, such as turn off the System Immutable
flag. You must run at level 0. Switching into single user mode will allow one to run at level 0
and thus change the System Immutable flag.

Do this.

Close all applications and issue the command:

% sudo shutdown +0 "Bye bye"

to shutdown multi-user mode and enter single user mode. You will lose all services such as
network connectivity while in single user mode. Alternatively, you can reboot your Mac, then
startup with both the Command (Apple) key and the 's' key held down.

% cd /Users/your-name-here/.Trash
% chflags -R noschg *

Then hit control-d to return to multi-user mode. You should now be able to delete the trash with
the Finder, or by reapplying the steps above.

Note: when you enter single user mode, type:

% whoami

if the answer is not root type:

% su

and type control-d twice when you exit.

Other Trashes

On some occasions, trash is placed at the root level of your system disc or another partition or
drive. The directory is:

/.Trashes/501

and

/Volumes/disc-name-here/.Trashes/501

See if they are empty by listing them.

% sudo ls -al /.Trashes/501
...

'501' is the User ID of the user who did the trashing.

% sudo ls -al /.Trashes

may reveal several such 50x directories.

You can check your own User ID (uid) by issuing the command:

% id

and substitute this if it is not 501.

You can then apply all the techniques given for ~/.Trash to /.Trashes/501.

And Finally

If you still have problems, and you have re-applied the above steps as necessary, you probably
have a corrupt file system.

Firstly, enter single user mode and issue:

% fsck

and answer the prompts, or else have 'fsck' run automatically with:

% fsck -y

You can also try commercial disc repair programs.

The Executable Name

This week's tips enhance the Learning Centre tutorial on scripting, part 8 and part 9.

Note that these tips are for 'sh' or 'bash' scripts.

The shell special parameter $0 contains the script name, but includes the full path name. Display
just the script name using ${0##*/}
$ cat usage
#!/bin/sh
echo "Usage 1: $0"
echo "Usage 2: ${0##*/}"

$ ~/Development/test/usage
Usage 1: /Users/saruman/Development/test/usage
Usage 2: usage

Complex Conditions

Conditions in 'if', 'elif', and 'while' statements can contain AND, OR, and NOT elements. For
example, to test if parameter 1 is "-p" AND parameter 2 is empty use:
if [ "$1" = "-p" -a "$2" = "" ]; then

The '-a' represents AND.

Use '-o' for OR.

To reverse a condition use pling (!). For example to test if a file is NOT a directory use:
if [ ! -d "$file" ]; then

Check if a Volume is Mounted

To check if a volume is mounted, for example /Volumes/your-user, it is necessary to check that
/Volumes/your-user exists AND is a mount, not a regular file or directory. One way to do this is
to examine the output from df:
mounted=$(df | grep "/Volumes/$USER")

If the volume is not mounted then the variable 'mounted' will be a null string. Note that the
environment variable $USER expands to the short name of the current user.

One can use the condition:

if [ ! "$mounted" ]; then
statement to mount /Volumes/$USER
fi

because an empty string has the test value FALSE.

Mounting and AFP Volume

An AppleShare volume can be mounted using the lines:
mkdir /Volumes/$USER
mount_afp afp://$USER@carcharoth.mayo-family.com/$USER /Volumes/$USER > /dev/null

This mounts the home directory of the current user that is located on the server 'SERVER' at
mount point /Volumes/users-short-name.

Eg, the script:

$ cat mount-user
#!/bin/sh

mounted=$(df | grep "/Volumes/$USER")

echo "Mounting /Volumes/$USER"
mkdir /Volumes/$USER
mount_afp afp://$USER@your-server-address-here/$USER /Volumes/$USER > /dev/null
else
echo "/Volumes/$USER already mounted"
fi

Will mount my home directory (saruman) located on my server under /Volumes/saruman:
$ ls /Volumes
<empty>

$ ./mount-user
Mounting /Volumes/saruman

$ ls /Volumes
saruman

$ ./mount
/Volumes/saruman already mounted

A Mount-User Script

Here is a script using the tips given this week, and the shell programming tutorials in the
Learning Centre part8 and part 9.

It mounts the current user's home directory located on the server $SERVER, on the local machine
at mount point $AFP_MOUNT/$USER.

It uses two environment variables which must be set before the script runs:

AFP_MOUNT to say where to mount the volumes (this is usually '/Volumes')
SERVER for the network address of the server

A password can optionally be given with '-p password'. Otherwise mount_afp will ask for your
password in a pop-up, or extract it from the keychain.

$ cat mount-user
#!/bin/sh

# mounts the current user's home directory on the server into the afp mount dir
#

if [ ! "$1" = "" -a ! "$1" = "-p" ]; then
echo "Usage: ${0##*/} [-p password]"
echo " mounts ${USER}'s home directory on the server at $AFP_MOUNT/$USER"
exit
fi

if [ "$1" = "-p" -a "$2" = "" ]; then
echo "Usage: ${0##*/} [-p password]"
echo " give a password"
exit
fi

# if password is given add ":" to the start to comply with mount syntax
if [ "$1" = "" ]; then
pass=""
else
pass=":"$2
fi

# check if already mounted
#
mounted=$(df | grep "$AFP_MOUNT/$USER")

# if not mounted, ensure the mount point (directory) is available
#
if [ ! -d $AFP_MOUNT/$USER ]; then
if [ -e $AFP_MOUNT/$USER ]; then
rm -f $AFP_MOUNT/$USER
fi
mkdir $AFP_MOUNT/$USER
fi
mount_afp afp://$USER$pass@$SERVER/$USER $ALM_AFP_MOUNT/$USER > /dev/null
disktool -r
fi

# return 0 if no mount was performed, 1 otherwise
#
exit 1
else
exit 0
fi

Symbolic Links

A link is a file that 'points' to another file. They come in two varieties: symbolic link and
hard link (covered on Tuesday).

A symbolic link is created using the 'ln' command, specifying option '-s' for symbolic.

% echo "I am the original file" > orig-file
% /bin/ls -l *-file
lrwxr-xr-x 1 saruman staff 9 ... link-file -> orig-file
-rw-r--r-- 2 saruman staff 23 ... orig-file

The symbolic link is shown with the first character as 'l' in a long listing, and as pointing to
the original file.
% cat orig-file
I am the original file
% cat link-file

'link-file' is simply a file that contains some text describing the absolute path to 'orig-file',
and has a flag set to say 'I am a symbolic link'. The two files can be anywhere on the file
system - different directories, partitions, disks, or machines.

When 'link-file' is accessed, the link is automatically followed resulting in 'orig-file' being
accessed.

If 'orig-file' is moved or renamed the link will be broken.


Hard Links

A hard link is created using the 'ln' command, but WITHOUT specifying option '-s'.

% ln orig-file hard-file
% /bin/ls -l *-file
-rw-r--r-- 2 saruman staff 23 ... hard-file
lrwxr-xr-x 1 saruman staff 9 ... link-file -> orig-file

% cat orig-file
% cat hard-file

'hard-file' is not a new file, but a new directory entry that points to the original file's data
(the same part of the disc). In technical terms, the two directory entries reference the same
inode.

The two files are one in the same, and it is not possible to determine which is the original and
which is the hard link.

Either file can be renamed or moved without breaking the link.

Hard links cannot be created across partitions or disks. This is because inodes belong to a
particular partition and are not unique across partitions or drives.

Aliases

An alias is managed by the Mac Finder and is not recognized by Unix. An alias has an empty data
fork, with the aliasing information stored in the resource fork.

(Create and alias (alias-file) to orig-file using the Finder.)
% /bin/ls -l *-file
-rw-r--r-- 1 saruman staff 0 ... alias-file
-rw-r--r-- 2 saruman staff 23 ... hard-file
lrwxr-xr-x 1 saruman staff 9 ... link-file > orig-file

% cat orig-file
% cat alias-file
%

Unix sees only the data fork and hence 'alias-file' as being empty. The Finder recognizes
symbolic and hard links. Copying an alias using the Unix 'cp' command will create only an empty
file.

Note that 'ls' with option -s does recognize the resource fork when calculating the number of
blocks the files uses (36k or 72 half-K blocks).

% /bin/ls -sl *-file
72 -rw-r--r-- 1 saruman staff 0 ... alias-file
8 -rw-r--r-- 2 saruman staff 23 ... hard-file
8 lrwxr-xr-x 1 saruman staff 9 ... link-file > orig-file
8 -rw-r--r-- 2 saruman staff 23 ... orig-file

Working with Links

Many commands take options to affect the way they treat a link - usually to control if they deal
with the link itself, or follow the link and deal with the target file. This applies only to
symbolic links.

'ls' can be told to follow the link with option '-L':

% ln -s ~/Documents/Letters/Channel\ Dynamics a-letter
% /bin/ls -l a-letter
lrwxr-xr-x 1 saruman staff 49 ... a-letter >
/Users/saruman/Documents/Letters/Channel Dynamics
% /bin/ls -lL a-letter
-rw-r--r-- 1 saruman staff 20480 ... a-letter

and can show inode numbers with option '-i':

% ls -l i *-file
678102 -rw-r--r-- 1 saruman staff 0 ... alias-file
678094 -rw-r--r-- 2 saruman staff 23 ... hard-file
678095 lrwxr-xr-x 1 saruman staff 9 ... link-file > orig-file
678094 -rw-r--r-- 2 saruman staff 23 ... orig-file

Note that the hard link and original file have the same i-node number.

Other commands take options '-H', '-L', and '-R' which generally mean:

-H If the -R option is specified, symbolic links on the
command line are followed. (Symbolic links encountered in
the tree traversal are not followed.)

-L If the -R option is specified, all symbolic links are
followed.

-P If the -R option is specified, no symbolic links are
followed.

The commands include:

* 'chflags'
* 'chgrp'
* 'chmod'
* 'chown'
* 'cp' HLP
* 'du' HLP
* 'find'

'test' (aka '[') can test for a symbolic link with '-L':
-L file - True if file exists and is a symbolic link.

Making a Shadow Directory

The command 'lndir' creates a shadow directory of symbolic links to another directory.

For example, make the following directory structure:

% mkdir ln-test
% cd ln-test
% touch a b
% mkdir c
% touch c/a c/b
% ls -R
.:
a b c

c:
a b

Make a shadow directory:
% cd ../
% mkdir ln-test-copy
% cd ln-test-copy/
% lndir ../ln-test
../ln-test/c:

Examine the shadow directory created by 'lndir':

% /bin/ls -lR
total 16
lrwxr-xr-x 1 saruman staff 12 Oct 5 11:44 a -> ../ln-test/a
lrwxr-xr-x 1 saruman staff 12 Oct 5 11:44 b -> ../ln-test/b
drwxr-xr-x 4 saruman staff 136 Oct 5 11:44 c

./c:
total 16
lrwxr-xr-x 1 saruman staff 17 Oct 5 11:44 a -> ../../ln-test/c/a
lrwxr-xr-x 1 saruman staff 17 Oct 5 11:44 b -> ../../ln-test/c/b

Edit Multiple Files

If you wish to edit a group of files, combine 'find' with your editor of choice ('vim' in my
case). To edit all text files (*.txt) in the current directory hierarchy use:
% vim `find . -iname "*.txt"`

In 'vim' or 'vi', type
:n

to move to the next file, or
:wn

to save changes and move to the next file.

To abort changes and move to the next file use:
:n!

And finally, to quit before the entire list of files have been edited:
:q!

Edit Conditionally on Contents

To search for all files that match a filename pattern AND that contain particular text use the
following.

For example to search for all text files (*.txt) that contain the word "memo", and then edit them
using 'vim':
vim `find . -type f -iname "*.txt" -print0 | xargs -0 grep -ilw "memo"`

* The -print0 and -0 options are to cope with filenames containing spaces.
* 'Find' option -type f is to ensure only regular files are considered.
* 'Grep' options -iw ensure the text search is case insensitive and matches only whole words.

Less (and More) Edits

If you view a file with 'less', then decide you wish to edit that file, just hit 'v'. This will
take you into the 'vi' editor, pre-loaded with the file you were viewing.

If 'vi' is not to your taste, set the environment variable VISUAL to define which editor will be
used.
% setenv VISUAL vim

You can do the same trick with 'more', except that you will need to set environment variable
EDITOR instead. Note that 'less' will use EDITOR if VISUAL is not defined.

Corruption...

If your terminal session gets corrupted, for example after accidentally editing or displaying a
binary file, use 'tput'.
% tput reset

to reset the Terminal, or
% tput init

to initialise it.

The difference between these two depends on the difference between the 'reset' and 'init'
terminal strings in the 'terminfo' database (/usr/share/terminfo/....). Mac OS X uses the
terminfo rather than termcap. Terminfo is a compiled database, and in order to view the
information you must use 'infocmp terminal-name'.

For example:
% infocmp vt100
% infocmp xterm

Cleaning Up

Tidy up a file with 'tr' and 'sed'.

The following command removes duplicate spaces, blank lines, and changes Mac style end of lines
into Unix style.
% tr -s " " < test.txt | tr \\r \\n | sed '/^$/d' > out.txt

The input file is test.txt and the output is out.txt

About POSIX Regular Expressions

REs come in three flavours: older style 'basic' or 'obsolete' as recognized by 'grep', 'sed',
'vi', ...; 'extended' or POSIX-compliant as recognized by egrep, gawk, PHP, ...; and Perl regular
expressions as recognised by Perl and PHP.

Perl are more powerful than POSIX, which in turn are more powerful than basic.

Incompatibilities exist between the different varieties, so one should check which form to use
with any given utility.

By default, searching is done a line at a time, and is case sensitive.

This week's tips will concentrate on POSIX-style REs.

Matching and Escaping with POSIX Regular Expressions
% cat test
This is a test file
to illustrate the use of
regular expressions.
Here is a dollar $ symbol.
this is the end of the test file.

To search for a simple string:
% egrep "This" test
This is a test file

You must explicitly make the search case-insensitive: the method will depend on the utility you
are using. For 'egrep' use option -i.

Characters that have a special meaning within regular expressions must be escaped to say "take
the character literally and ignore any special meaning it may otherwise have".

Special characters that must be escaped are:
. ^ $ * + ? [ ( ) | { \

For example:
% egrep 'dollar $' test
(no matches output)

% egrep 'dollar \$' test

Use single quotes to prevent the shell from interpreting the $ sign.

Note that it is not always necessary to escape special characters because they may be special
only in a particular context, but to be safe it won't harm to always escape.
% egrep 'dollar $ symbol' test

Anchors in POSIX Regular Expressions

An anchor is used to force the pattern to match either the start or end of a line.
% cat test
this is line one
and this is the second line

Searching for 'this' or 'line' will match both lines of the file:

Use '^' to anchor the pattern to the start of a line:
% egrep '^this' test
this is line one

Use '$' to anchor the pattern to the end of a line:
% egrep 'line$' test
and this is the second line

Wildcards and Repeaters

The special character '.' will match any character.

To match b<any one character>g use:
% cat test
This is a bag.
This is a bug.
This buug won't match.

% egrep "b.g" test
This is a bag.
This is a bug.

% egrep "b..g" test
This buug won't match.

These special characters say to match if the previous character is repeated a certain number of
times:
<pattern>? matches <pattern> repeated 0 or 1 times
<pattern>+ matches <pattern> repeated 1 or mores
<pattern>* matches <pattern> repeated 0 or more times

'zu?z' matches 'zz' or 'zuz'
'zu+z' matches 'zuz' or 'zuuz', 'zuuuzzzz'
'zu*z' matches 'zz, 'zuz', 'zuuz', 'zuuu...z'

Finally:
'Hello.*goodbye' will match Hello<0 or more of any character>goodbye

Ranges with POSIX Regular Expressions

Use ranges to match a specific number of characters.
<pattern>'{m,n}' matches <pattern> repeated m to n times inclusive
<pattern>'{m,}' matches <pattern> repeated m or more times
<pattern>'{m}' matches <pattern> repeated m times

'zu{3.5}z' matces 'zuuuz', 'zuuuuz', 'zuuuuuz'
'zu{3,}z' mathes 'zuuuz', 'zuuuuz', 'zuuuuu...z'
'zu{4}z' matches just 'zuuuuz'

Match a Range of Characters with POSIX Regular Expressions

Match any one of a range of characters using a bracketed expression (a range of characters
enclosed in []).
% egrep 'b[aeiou]g'

matches bag, beg, big, bog, and bug.

Invert the match by starting the sequence with the '^' character.
% egrep 'b[âeiou]g' file

matches bbg, bcg, ... bzg, but not bag, beg, etc.

Match a range of consecutive characters using a hyphen, like [a-z] for all lower-case letters, or
[0-9] for all decimal digits. For example:
% egrep '^[a-zA-Z]{3}[0-9]{2}$' file

matches 'letter letter letter digit digit' anchored to the start and end of the line (see last
week's tips for anchors and {} repeaters).

Character Classes in Regular Expressions

A character class is a name for a particular set of characters, is enclosed in :class-name: and
used within a bracketed expression.
[:alpha:] matches any alphabetic character
[:alnum:] matches any alphanumeric character
[:alpha:?] matches any alphabetic character or a '?'.

The classes are:
alnum digit punct
alpha graph space
blank lower upper
cntrl print xdigit

Character classes work in PHP and Perl, but don't seem to work in egrep in OS X 10.2.

Escaping, or Lack Thereof

It is not necessary (in fact it is incorrect) to escape the special characters ^ $ * + ? [ ( ) |
{ \ within a bracketed expression (see Monday's tip).

To match a '^' ensure it is *not* the first character in the list (otherwise it will invert the
match), and to match ']' ensure it *is* the first character in the list.
% egrep '[]%$^]' file

matches any of ] % $ ^

Match non of the above with:
% egrep '[^]%$^]' file

Note: It is permissible to escape special characters in bracketed expressions when using Perl, or
Perl compatible expressions in PHP.

Testing Regular Expressions

The easiest way to test out a regular expression is using 'egrep' and the standard input.
% egrep '<regular-expression-to-test>'

egrep will read from standard input (no filename supplied). Type a line of text. If the regular
expression matches the line of text will be echoed back, if it doesn't it won't.

Repeating using Regular Expressions

If you wish to apply a wildcard or repeater (see last week's tips) to more that a single
character, enclose the section to match in parenthesis to create a sub-expression and apply the
repeater to that.
% egrep 'hello(,hello){1,}'

matches:
hello,hello
hello,hello,hello
hello,hello,hello,hello etc

% egrep '^(Say hello(,hello){1,}, wave goodbye.){1,2}$'

matches:
Say hello,hello, wave goodbye.

or
Say hello,hello,hello, wave goodbye.Say hello,hello,hello, wave goodbye.

Viewing Log Files

This and next week I will cover all things log-file, from viewing them to configuring the logging
daemon.

Most log files live in '/var/log', including the main system log, and log files for netinfo and
the lookup daemon. If you have enabled any of mail, ftp, and printer services, you will see logs
for these too. Most general information, and critical mail information, is written to the system
log.

View log files using 'tail' (system.log in the example), statically with:
% tail -n35 /var/log/system.log

using '-n35' to show the last 35 lines instead of the default 10.

View log files dynamically with:
% tail -n35 -f /var/log/system.log

Press control-c to stop 'tail' and return to the command line.

The command:
% tail -n35 -f /var/log/system.log &
[1] 683 <--------------- note the number you see here.

will constantly interrupt your Terminal screen with the latest log file entry, and ultimately
irritate you :-)

To stop it issue:
% kill nnn

where nnn is the number you noted earlier.

zcat and tlog

Have a look in /var/log. You will notice that log files are 'rolled' periodically: daily for the
system log, and weekly for most others. Rolling involves zipping the most recent log file to
<log-file-name>.0.gz and creating a shiny new (and empty) log file. Previously zipped log files
are rolled - that is renumbered 0->1, 1->2, etc, with the oldest log file being deleted.

View older log files without the hassle of unzipping and re-zipping with the aid of 'zcat'.
% zcat /var/log/system.log.0.gz | tail -n35

Create an alias as a useful short-cut to viewing log files:
alias tlog 'tail -f -n35 /var/log/\!:1.log'

allowing one to type:
% tlog system

rather than:
% tail -n35 -f /var/log/system.log

Rolling Your Own

If you run 'apache', or 'named', or any daemon that creates many log files, you can 'roll' (see
Tuesday's tip) them yourself with a simple script placed in /etc/weekly.local. 'apache' and
'named' write their log files to a subdirectory of /var/log, and so the log files are not rolled
by the standard weekly tidy-up scripts.

If /etc/weekly.local does not exist create it as:
4.0k -r-xr-xr-x 1 root wheel 973 Nov 1 14:01 weekly.local

The script will look like:

#!/bin/sh -

##################################
# Local weekly tidy-up script #
##################################

host=$(hostname -s)
echo "Subject: $host Local weekly run output"

for logtype in httpd named
do

echo ""
echo -n "Rotating type $logtype log files:"

if [ -d /var/log/$logtype ]; then
cd /var/log/$logtype
for log in *.log
do
echo -n " $log"
if [ -f "${log}.3.gz" ]; then mv -f "${log}.3.gz" "${log}.4.gz"; fi

if [ -f "${log}" ]; then
mv -f "${log}" "${log}.0"
/usr/bin/gzip -9 "${log}.0"
fi
touch "$log"
done

case $logtype in
httpd) apachectl graceful;;

named) ndc restart;;
*);;
esac

fi

done

echo ""
echo "Complete"

Notes:

This line specifies the subdirectories of /var/log that will be rolled:

These lines restart each daemon to ensure that it recognises the new log file:

'ndc restart' may not work in Panther because it uses a different version of 'bind' to Jaguar.

The Console

Error-level, kernel, and mail critical messages are written to the console (some also go to the
system log). The console is a device under /dev/console, and all output to it is written to the
file /var/tmp/console.log.

View the console with:
% tail -n35 /var/tmp/console.log

or
% tail -n35 -f /var/tmp/console.log

The console log can also be viewed by Console.app in /Application/Utilities.

dmesg

Log messages from the Kernel, particularly those from device drivers, are written to the system
message buffer instead of log files.

View the system message buffer with:
dmesg

If you experience start-up or device driver problems, 'dmesg' may be a valuable source of
debugging information.

The Daemonic Syslogd

Unix applications make log entries via the system logger - 'syslogd'. Each log entry includes a
facility like 'mail', 'syslog', or cron', and a level ranging from 'emergency' to 'debug'.

Syslogd filters message and directs them to particular log files dependant on their facility and
level, and its own configuration. (See Tuesday.)

Facilities are:
auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, syslog, user, uucp and local0 through
local7

Levels are:
emerg, alert, crit, err, warning, notice, info and debug

Change the Syslogd Configuration

The syslogd daemon logs messages according to their facility and level (see Monday) and its own
configuration in /etc/syslog.conf.

Change what is logged, and where, by altering /etc/syslog.conf. Restart syslogd and the
corresponding deamons(s) to ensure the changes are seen. Reboot if wider-ranging changes are
made.

Each line consists of a list of 'facility.level' pairs, and a destination. All log entries that
include a listed facility, and a corresponding error level at or above that specified, will be
sent to the destination. '*' is a wildcard in the usual sense.

Log to Another Host

To send to another host, use @hostname as the destination given in /etc/syslogd. The receiving
host must have syslogd started with option -u. Log messages are sent to UDP port 514.

Note that this is vulnerable to attacks from outside, as a cracker is able to continually write
to your log files.

Test Log with Logger

The 'logger' command can be used to manually log messages, complete with facility and priority.
This is useful for testing a new 'syslogd' configuration.

Write to the mail log file at level info:
logger -p mail.info Test log message here...

Broadcast an emergency message to all logged in users:
logger -p user.emerg Emergency...

Change ftpd's Logging

Change the ftp daemon's (ftpd) logging scheme.

Type:

cat /etc/syslog.conf
~ % cat /etc/syslog.conf

*.err;kern.*;auth.notice;authpriv,remoteauth.none;mail.crit /dev/console
*.notice;*.info;authpriv,remoteauth,ftp.none;kern.debug;mail.crit /var/log/system.log

# The authpriv log file should be restricted access; these
# messages shouldn't go to terminals or publically-readable

# files.
authpriv.*;remoteauth.crit /var/log/secure.log

lpr.info /var/log/lpr.log
mail.* /var/log/mail.log
ftp.* /var/log/ftp.log
netinfo.err /var/log/netinfo.log

*.emerg *

Notice the line:

ftp.* /var/log/ftp.log

which says that facility ftp, all levels (denoted by '*') will be sent to ftp.log.

Change /var/log/ftp.log to log to a different file.

This line:

*.notice;*.info;authpriv,remoteauth,ftp.none;kern.debug;mail.crit /var/log/system.log

specifically excludes ftp messages from the system log by specifying 'ftp.none'

Change to 'ftp.*' to get the ftp daemon to log to the system log.

This line:

*.err;kern.*;auth.notice;authpriv,remoteauth.none;mail.crit /dev/console

specifies that levels 'err' and above for all ('*') facilities (thus including ftp) are sent to
the console.

The console log can be viewed by the Console.app, or by viewing /var/tmp/console.log.

Active or Passive

Ftp connections come in two flavours:

Active - in this mode the server contacts the client to transfer data to it. The client must have
the appropriate ports open in its firewall.

Passive - in this mode the client always contacts the server and therefore does not need to open
ports. Passive mode is the default.

If you wish to use active mode, or find that passive mode hangs, open a hole in the firewall
with:
% sudo ipfw add 2300 allow tcp from any 20,21 to any 1024-65535 in

(The rule number, 2300 in this example, must not conflict with any existing rule numbers.)

This will enable your client to use active ftp, but will not open your Mac up as an ftp server.

The umask

The default umask (permissions for created files) set by the ftp daemon is 027, meaning that
there is no world read access to files you upload. If you wish to change this, for example to
allow Apache access to files you have uploaded, then set the umask:
ftp> site umask 022

This affects only the current ftp session.

A Permanent umask

The umask solution from Tuesday affects only the ftp session from which it was issued. To effect
this permanently, for all logins, you need to configure the ftp daemon.

Edit the file /etc/ftpd.conf (or create it) - you must do so as the root (super) user.

Add a line:
umask all 022

FTP Debugging

The ftp daemon logs to /var/log/ftp.log

If you wish to include extra information in the log file then start the daemon with option -d.

Edit the file:
/etc/xinetd.d/ftp

and add the debug, or any other options you require, to the line
server_args = -l

Security

All Unix account holders may ftp into a server. To prevent a particular user from using ftp add
that user's account name to:
/etc/ftpusers

In order to create a user who cannot log in, but can ftp, set that user's shell to:
/sbin/nologin

and ensure that this shell is listed in:
% cat /etc/shells
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/bash
/bin/csh
/bin/sh
/bin/tcsh
/bin/zsh
/sbin/nologin

Finally, to stop an ftp user from exploring outside of their home directory, 'chroot' them. Add
their account name to:
/etc/ftpchroot

Note that this feature is broken in OS X 10.2.

Apple's decision to switch ftp daemons from ftpd in 10.1 to lukemftpd in 10.2 is of questionable
merit. Their lack of updated documentation to reflect that decision is truly lamentable. Most
users upgrading from 10.1.x are left hobbled or with malfunctioning ftp servers (especially in
regards to ftpchroot functionality), with no changes in the man pages to help them configure
their new ftp daemon.

One can either fix the broken built-in server by following the instructions here:
http://www.chezludo.com/ or replace it altogether. I elected to replace the built-in ftp server
with pure-ftpd, a robust, efficient and feature-rich ftp server that also boasts no root
exploits.

The first step is to download and unpack the source. First, change directories to wherever you
keep you downloads or source code:
cd /downloads
curl -O ftp://ftp.pureftpd.org/pub/pure-ftpd/releases/pure-ftpd-1.0.16a.tar.gz
tar xzf pure-ftpd-1.0.16a.tar.gz
cd pure-ftpd-1.0.16a/
./configure --with-everything --with-virtualchroot --without-banner --without-humor --enable-osx
--with-tls
(This will configure a 'big server' with a plethora of options, including throttling, ratios,
ftpwho, quotas, but will leave off the guady initial banner and the sprinkling of colorful banter
in the error messages, etc.)
sudo make install-strip

At this point you will need to choose which server type you desire, as pure-ftpd can run in
either standalone or xinetd mode:

Standalone Mode
You can run the server in standalone mode with this command:
sudo /usr/local/sbin/pure-ftpd &
or if you desire, use command line switches to configure the server at runtime:
sudo /usr/local/sbin/pure-ftpd -A -E -p 40000:50000 -c 5 -C 1 -I 5 -T 25 -u 1 &

The command line switches I have chosen tell the server the following:
-A chroots everyone
-E only allows authenticated users; anonymous users disallowed
-p 40000:50000 specifies the port range for passive connections
-c 5 specifies the number of clients
-C 1 specifies the number of connections per IP address
-I 5 changes the idle timeout; default 15 minutes seems excessive
-T 25 throttles the bandwidth to 25KB/sec per user
Many other switches are available. See the documentation for a complete list.

To get the standalone server to launch automagically at startup, you would have to write a
Startup Item:
http://www.macfora.com/forums/showthread.php?s=&threadid=6314

xinetd Mode
First, get a root shell so all the following commands don't have to be prefaced with sudo:
sudo -s
(As always, before editing a system level file, it is wise to create a backup first.)
cd /etc/xinetd.d/
cp -p ftp ftp.default
pico ftp

Modify the server and server_args lines as folows:
Code:

service ftp
{
disable = no
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/pure-ftpd
server_args = -A -E -p 40000:50000 -c 5 -C 1 -I 5 -T 25 -u 1
groups = yes
flags = REUSE
}

Stop xinetd (if running): killall xinetd
Start xinetd to get it to read the new ftp configuration file:
sudo /usr/sbin/xinetd -pidfile /var/run/xinetd.pid
Exit the root shell:
exit

Test to confirm that it is working:
ftp localhost

If you get something like this:
Code:
[gatorparrots:] gator% ftp localhost
Connected to localhost.
220-FTP server ready.
220 This is a private system - No anonymous login
Name (0:gator):

Congratulations! Your new FTP server is working as advertised. To enable the chroot to a single
directory, simply assign your ftp users' home directories to your ftp root directory via NetInfo
(and possibly put them in a dedicated ftp user group for added flexibility). Otherwise, the
individual users will be chrooted to their /Users/username home directory.

Periodic

Periodic system maintenance is scheduled by 'cron' (see Week 33). Cron launches a program called
'periodic' which is responsible for launching the maintenance scripts.

% cat /etc/crontab
...
#minute hour mday month wday who command
# Run daily/weekly/monthly jobs.
15 3 * * * root periodic daily
30 4 * * 6 root periodic weekly
30 5 1 * * root periodic monthly
...

The daily script is run at 3:15 am.
The weekly script is run at 4:30 am Saturday.
The monthly script is run at 5:30 am on the first of each month.

Your Mac must be running (not sleeping) for these scripts to be run.

The output from these scripts can be found in:
% ls /var/log/*.out
/var/log/daily.out /var/log/monthly.out /var/log/weekly.out

Manual Maintenance

If your Mac is not switched on in the early hours (see Monday's tip), you will have to run the
maintenance scripts manually. The is done by:
% sudo periodic daily
% sudo periodic weekly
% sudo periodic monthly

Alternatively, edit /etc/crontab and choose times at which your Mac will be running.

Or, use the freeware utility MacJanitor
http://macupdate.com/info.php/id/5856

Roll Your Own

If you wish to add your own maintenance tasks - daily, weekly, or monthly - then create the
following files as necessary.
/etc/daily.local
/etc/weekly.local
/etc/monthly.local

These are regular shell scripts, and once created are run automatically by the default daily,
weekly, and monthly scripts.

Periodic Configuration

The scripts run by periodic for daily, weekly, and monthly maintenance are to be found in:
# ls /etc/periodic/*
/etc/periodic/daily:
100.clean-logs 500.daily

/etc/periodic/monthly:
500.monthly

/etc/periodic/weekly:
500.weekly

Periodic can be called with a full path name to a directory (instead of the words daily, weekly,
or monthly), and it will run all scripts in that directory.

Periodic is configured by the file:
/etc/defaults/periodic.conf

You can add your own setting and place them in the file:
/etc/periodic.conf

See 'man periodic.conf' for more information.

An Example Log Rotator

This script can be placed in /etc/weekly.local. It will rotate log files, much as the standard
weekly script does. The standard script only rotates *.log files is /var/log. This script does
the same for subdirectories such as /var/log/httpd/*.log and /var/log/bind/*.log.

#!/bin/sh -

#########################
# Weekly tidy-up script #
#########################

host=$(hostname -s)
echo "Subject: $host - local weekly run output"

# list the subdirectories to be considered, in this case
# /var/log/httpd and /var/log/named
do

echo ""
echo -n "Rotating type $logtype log files:"

if [ -d /var/log/$logtype ]; then
cd /var/log/$logtype
for log in *.log
do
echo -n " $log"

if [ -f "${log}" ]; then
mv -f "${log}" "${log}.0"
/usr/bin/gzip -9 "${log}.0"
fi
touch "$log"
done

case $logtype in
*);;
esac

fi

done

echo ""
echo "Complete"

Filenames With Spaces

Filenames with spaces can cause problems in scripts. (See week 49 for one such example.)

When processing all files in a directory with a 'for' loop, use '*' instead of '`ls`'. For
example:
bash$ ls -1
file
file 2

bash$ for file in `ls`; do echo "File: $file"; done
File: file
File: file
File: 2

bash$ for file in *; do echo "File: $file"; done
File: file
File: file 2

Arithmetic

Bash will evaluate an arithmetic expression enclosed in $((...))
bash$ n1=3; n2=4; n3=5
bash$ echo $n1 $n2 $n3
3 4 5

This will not work:
bash$ n="(($n1+$n2)*$n3)"
bash$ echo $n
((3+4)*5)

But this will:
bash-2.05a$ n=$(( ($n1 + $n2) * $n3 ))
bash-2.05a$ echo $n
35

Or use nested $((...)) expansion:
bash$ n=$(( $(( $n1 + $n2 )) * $n3 ))
bash$ echo $n
35

I Do Declare

Bash allows one to 'declare' shell variables. One advantage of declaring variables is that they
can be given specific attributes.

Declare a read-only variable:
bash$ declare -r pi=3.14
bash$ echo $pi
3.14
bash$ pi=20
bash: pi: readonly variable

Variables can also be marked as integers, in which base bash will apply arithmetic evaluation
without the need for $((...)) (see Tuesday):

This will not work:
bash$ declare area1
bash$ area1=5*5
bash$ echo $area1
5*5

But using 'declare -i' this will:
bash$ declare -i area2
bash$ area2=5*5
bash$ echo $area2
25

Default Parameter Values

Suppose you wish to set a variable to the value of $1 if $1 is given, otherwise to some default
value.

This can be achieved with:

#!/bin/bash
#set level to $1, or the default 5 if $1 not given
if [ "$1" == "" ]; then
level=5
else
level=$1
fi

Or simpler:
level=${1:-5}

Use Printf

The printf command is very much more powerful than echo. It takes a format string and a set of
arguments to print within the context of that format string.

This statement will print a line, indented by a calculated amount, the label right-aligned, and
the first value in red.

bash$ cat tst
#!/bin/bash
declare -r norm="\033[00m" red="\033[31m"
declare -i indent=${3:-4}
declare -i spacer=20-$indent

printf "%${indent}s%-${spacer}s${red}%7d,${norm}%7d\n" " " "Values:" $1 $2

bash$ tst 1234 5678
Values: 1234, 5678

bash$ tst 1234 5678 8
Values: 1234, 5678

Setting the Prompt

The environment variable PS1 contains the bash shell primary prompt.

Change this with, for example:
$ declare -x PS1="host=\h\$ "
host=saruman$

\h is replaced by the hostname up to the first '.'. \$ is replaced by '$' for normal users, and
by '#' for root.

This example will display the current username and time in red:
$ declare -x PS1="\e[31m\u-\A\e[0m\$ "
saruman-10:56$

Other possibilities (taken from the bash manual)

\a an ASCII bell character (07)
\d date as "Weekday Month Date"
\e an ASCII escape character (033)
\h the hostname up to the first `.'
\H the hostname
\j # jobs currently managed by the shell
\l basename of shell's terminal device
\n newline
\r carriage return
\s name of shell, basename of $0
\t time in 24-hour HH:MM:SS format
\T time in 12-hour HH:MM:SS format
\@ time in 12-hour am/pm format
\A time in 24-hour HH:MM format
\u the username of the current user
\v the version of bash (e.g., 2.00)
\V release of bash, version+patchlevel
\w the current working directory
\W basename of current working directory
\! the history number of this command
\# command number of this command
\$ a # for the root user , otherwise a $
\nnn character with octal number nnn
\\ a backslash
\[ start non-printing characters
\] end non-printing characters

Bash Startup Files

Create a file called '.bash_profile' in the root of your home directory. This file is executed
when bash is invoked as a login shell (when you start a new Terminal session with Terminal.app).
In here place all settings that are passed to sub-shells - i.e. set environment variables like
PS1 (see Monday's tip) and the search PATH.

Also add the line:
. .bashrc

This will execute the second startup file.

Create a second file called '.bashrc' in the root of your home directory. This file is executed
when bash is invoked as a non-login shell (when you type 'bash' to start a sub-shell). If you
have added the line '. .bashrc' to '.bash_profile', '.bashrc' will then be executed for all
(login and non-login) interactive shells. Place all other setting in '.bashrc'.

Pushing and Popping

If you wish to change directories temporarily, wanting to return to the original directory, use
'pushd' instead of 'cd'. 'pushd' changes to the new directory, but saves the original directory.
'popd' returns to the original directory.

For example:
$ pwd
/Users/saruman
$ pushd /etc
/etc ~
$ pushd /var/log
/var/log /etc ~
$ pwd
/var/log
$ popd
/etc ~
$ pwd
/etc
$ popd
~
$ pwd
/Users/saruman
$

More History

Be default, a Terminal session saves the last 500 commands in its history. If you are in the
habit of leaving your Terminal open for long periods and would like more history (say 2000
commands) add the following to '.bash_profile' (see Tuesday's tip).
declare -x HISTSIZE=2000;

It makes sense to increase the size of the history file too.
declare -x HISTFILESIZE=2000;

If you have more than one terminal session open, then to prevent the last exited session from
overwriting the history file, set the history merge option in '.bashrc'
shopt -s histappend

This causes histories from different Terminal session to be merged.

Ignore Case

If you wish for bash's tabbed auto filename completion to ignore case, ie:
$ ls ~/doc <hit tab>

will be completed to:
$ ls ~/Documents/

create a file called '.inputrc' in the root of your home directory and add the line:
set completion-ignore-case on

Additionally, to cause auto-completion to list alternatives immediately (bash normally rings the
bell first) add the line:
set show-all-if-ambiguous on

Open Files and Folders

Use the 'open' command to simulate a double click in the Finder.

Here are some examples of its usage.

Open a folder into a Finder window:

% open Sites

Open a document with the default application:
% open Sites/index.html

Open a document with another (non-default) application:
% open -a /Applications/Internet\ Explorer.app Sites/index.html

Open multiple folders/documents/applications in one go:
% open *.html pic1.jpeg pic2.jpeg ...

Launch Applications

Launch GUI applications, both Cocoa- and Carbon-based, using 'open'.

Open an application:
% open /Applications/Safari.app

Launch a Cocoa application manually, i.e. without using 'open'.
% /Applications/Safari.app/Contents/MacOS/Safari &

The '&' at the end of the command will launch Safari as a separate process allowing you to
continue using the current shell.

Manual launching may be quicker in scripts. It can also be used to launch multiple copies of an
Application, where the Finder (and 'open') would not do so.

Launch Applications as Root

Use the 'sudo' command to launch an application running as root.

Using 'sudo open' will not work, as 'open' runs as root but the application it launches does not.
One must launch manually (see Tuesday's tip).

% sudo -b open /Applications/Clock.app/
% ps -auxc | grep "[C]lock"
saruman 2405 ... 0:00.57 Clock

(Clock is running as my user 'saruman', NOT as root.)

% sudo -b /Applications/Clock.app/Contents/MacOS/Clock
% ps -auxc | grep "[C]lock"
saruman 2405 ... 0:00.72 Clock
root 2410 ... 0:00.36 Clock

(The second Clock is running as root.)

Carbon Applications

Launch a Carbon application (in this example Internet Explorer) manually.
% /System/Library/Frameworks/Carbon.framework/Versions/
Current/Support/LaunchCFMApp /Applications/
Internet\ Explorer.app/Contents/MacOS/Internet\ Explorer &

(Note there are no line-breaks in this command.)

Carbon applications are not in the correct binary format for Mac OS X. They must be launched by
'LaunchCFMApp', which does the necessary magic.

On systems prior to Panther, you may notice that using the '-c' option with 'ps' displays
LaunchCFMApp as the application name instead of the application itself. This will cause scripts,
and commands such as 'killall', to fail.

Launch a Carbon Application as root using:
% sudo echo
% sudo -b /System/Library/Frameworks/Carbon.framework/Versions/
Current/Support/LaunchCFMApp /Applications/
Internet\ Explorer.app/Contents/MacOS/Internet\ Explorer &

The 'sudo echo' line gets around password entry problems.

NEVER use 'sudo', or do anything as root, when you've had too many beers.

Default Values

This week's tips apply to both bash script parameters like $1, and shell variables like $my_var.
In the code examples I have used a shell variable.

Set a shell variable to either the value of another variable/parameter, or to a default value if
the other variable/parameter is not set.
$ echo $test
/hello/world

$ echo ${test:-not-set}
/hello/world

I have used 'echo' to illustrate the expansion, but normally one would assign to a variable:
$ var=${test:-not-set}

In the next example the variable 'test2' is not set and so the default value is substituted.
$ echo ${test2:-not-set}
not-set
$ echo $test2

Note that test2 remains unset unless one uses ':=' instead of ':-'
$ echo ${test2:=not-set}
not-set
$ echo $test2
not-set

Error on Unset

Get bash to report an error if a variable or parameter is not set.
$ echo ${test3:?}
bash: test3: parameter null or not set

Write a custom error message with:
$ echo ${test3:?it is not set}
bash: test3: it is not set

An error is reported if a variable does not exist, as above, or is set to null (nothing):
$ test4=
$ echo ${test4:?}
bash: test4: parameter null or not set

Slicing

Take a slice of a variable.
$ echo $test
/hello/world

$ echo ${test:4}
lo/world

$ echo ${test:4:2}
lo

The first example extracts a slice from the fourth character on (the first being counted as 0),
and the second extracts two characters from the fourth character on.

Determine the number of characters in a variable's value.
$ echo ${#test}
12

Chopping

Chop the head or tail from a variable.
$ echo $test
/hello/world

This example searches for a pattern at the start of the variable ("/hell") and chops it from the
expansion:
$ echo ${test#/hell}
o/world

It is permissible to use the wildcard '*', to match 0 or more occurrences of any characters.
$ echo ${test#*l}
lo/world

In the above example, the shortest string matching '*l' was chopped ('/hel'). In the next, use of
a double # matches the longest string ('/hello/worl').
$ echo ${test##*l}
d

Extract just the script name from the full pathname held in parameter 0. This should be used
within a bash script.

Assuming $0 is /usr/local/bin/script
scriptname=${0##*/}
echo $scriptname

gives:
script

Use % instead of # (and %% instead of ##) to chop from the end of the string.
$ echo ${test%/*}
/hello

$ echo ${test%%/*}
(empty string)

Search and Replace

Search and replace within a variable.

To search for the first occurrence of 'hello' and replace it with 'goodbye':
$ echo $test
/hello/world

$ echo ${test/hello/goodbye}
/goodbye/world

To replace all occurences of a pattern use '//'.
$ echo ${test/l/L}
/heLlo/world
$ echo ${test//l/L}
/heLLo/worLd

Use # to anchor the pattern to the start of the string, and % to anchor to the end.
$ echo ${test2/#d/D}
ldld
$ echo ${test2/%d/D}
ldlD

Basic Search in Vim

Search a file for particular text, such as 'transform'. In normal mode (hit escape to exit from
insert mode) type:
/transform

If you require a complete word match, such that 'transformations' is not matched, type:
/\<transform\>

To search backwards use:
?transform

To search for the next occurrence hit 'n', and for the previous hit 'N'.

To recall a previously entered search type '/' then hit the up-arrow key until the desired search
is found.

Search and Replace in Vim

Replace typo's such as 'teh' with 'the' using search and replace:
:%s/teh/the/g

'%' means all lines in the file and 'g' means every occurrence on each line.

Replace '%' with an address range such as '.,$' which performs the replacement from the current
line (line .) to the end of file (line $).

To recall a previously entered search-replace type '/' then hit the up-arrow key until the
desired search-replace is found.

Use regular expressions in vim searches. See weeks 59 and 60 for information on regular
expressions.

Search for a line starting with 'The':
/^The

Replace 'chapter <any text> includes' with 'Section <same text> Contains:' :
:%s/[C|c]hapter $.*$ [I|i]ncludes/Section \1 Contains:/g

The construct $...$ captures text in the search pattern, while \1 replays it in the replace
string.

Customising the Search in Vim

Make searches case insensitive with:
:set ignorecase

Now pattern /the matches The, the, THE

Set smartcase so patterns in all lowercase match in a case insensitive manner, but patterns
including uppercase are case sensitive:
:set smartcase

The pattern /the matches The, the, THE
but pattern /The matches only The

Highlight all matches by setting:
:set hlsearch

and switch off with:
:set nohlsearch

and use:
:noh[ighlight]

to temporary turn of the current highlights.

Quick Search in Vim

Find the next occurrence of the word currently under the cursor. In normal mode hit '*', or '#'
to search backwards. Only whole words will be matched, unless you use 'g*' or 'g#'

To search for the next occurrence hit 'n', and for the previous hit 'N'.

Search and Edit in Vim

Use global commands to search for lines to edit. To perform a search and replace, but only on
lines that begin with 'edit', use:
:g/êdit/s/hello/Goodbye/g

The general form of this command is:
g/search-pattern/command

'search-pattern' is a regular expression and 'command' is an 'ex' command. This is the same form
as used by 'sed'.

For more information on the available commands use:
man ex

To delete all lines that start with a '.' use:
:g/^\./d

The command 'd' deletes the line.

To select all lines that do not match the pattern use 'g!' instead of 'g'

To use normal mode vim commands instead of ex commands use:
g/search-pattern/normal vim-command-list

This example will add '...' to all lines that contain 'edit'
:g/edit/normal A...

The File Command

Determine the type of a file, based on an analysis of its contents, using command 'file':

$ file ~/bin/*
add-user: Bourne-Again shell script text
dhup: Bourne shell script text
crontab: ASCII text
nodif: Tenex C shell script text
test: Mach-O executable ppc

$ file ~/sites/.../*
data: directory
images: directory
include: directory
index.php: C++ program text NOT QUITE!
java-script: directory
jf.html: HTML document text
tc.html: HTML document text

Using the File Command

Use command 'file' in conjunction with other commands to filter file types.
$ find . -print0 | xargs -0 -n1 file
./important-info: directory
./important-info/mac-todo.txt: English text
./important-info/Packing List.rtf: Rich Text Format data, ...

See week 49, Friday in particular.

Filter for all rich text files using grep:
$ find . -print0 | xargs -0 -n1 file | grep -i "rich text"
./important-info/contact-info.rtf: Rich Text Format data, version 1, Apple Macintosh
./important-info/PW.rtf: Rich Text Format data, version 1, Apple Macintosh
./Job/CV.rtf: Rich Text Format data, version 1, Apple Macintosh

Extracting Filenames

Tune Tuesday's example to print only filenames:
$ find . | xargs -n1 file | grep -i "rich text" | awk '{print $1}'
./important-info/contact-info.rtf:
./important-info/PW.rtf:
./Job/CV.rtf:

To remove the trailing ':' tell awk that the field separator is ':':
$ find . | xargs -n1 file | grep -i "rich text" | awk -F: '{print $1}'
./important-info/contact-info.rtf
./important-info/PW.rtf
./Job/Adrian-Mayo-CV.rtf
./Job/job-search.rtf

Finally, edit all the rich text files:
$ open -a textedit $(find . | xargs -n1 file | grep -i "rich text" | awk -F: '{print $1}')

In the tcsh shell use backquotes - `find ...` instead of $(find ...) .

The Hexdump Command

Use command 'hexdump' to display the contents of a binary file, or a file that contains
unprintable characters:
$ hexdump -n32 test
0000000 6109 6201 630a 6e6f 7720 736f 6d65 2074
0000010 6578 7420 616e 6420 7461 6273 2020 0a0d

The file is displayed in hexadecimal, limited to the first 32 bytes (-n32).

One can print in character format, where non-printable characters are either represented in
octal, or escaped (\t = tab)...:
$ hexdump -n32 -c test
0000000 a \t b 001 c \n n o w s o m e t
0000010 e x t a n d t a b s \n \r

...or in hexadecimal plus character, where non-printable characters are represented by a dot:
$ hexdump -n32 -C test
00000000 61 09 62 01 63 0a 6e 6f 77 20 73 6f 6d 65 20 74 |a.b.c.now some t|
00000010 65 78 74 20 61 6e 64 20 74 61 62 73 20 20 0a 0d |ext and tabs ..|

Try this:
$ hexdump -n4 /bin/*

The String Command

Search for ASCII text within binary files using command 'strings'.
a$ strings -a -16 /bin/ls
__dyld_mod_term_funcs
__dyld_make_delayed_module_initializer_calls
The kernel support for the dynamic linker is not present to run this program.
$NetBSD: cmp.c,v 1.14 1998/10/09 02:00:39 enami Exp $
@(#) Copyright (c) 1989, 1993, 1994
The Regents of the University of California. All rights reserved.
$NetBSD: ls.c,v 1.31 1998/08/19 01:44:19 thorpej Exp $
1ACFLRSTWacdfgiklnoqrstuxv
%s: directory causes a cycle
$NetBSD: print.c,v 1.22 1998/07/28 05:15:47 mycroft Exp $
%s %*u %-*s %-*s
$NetBSD: stat_flags.c,v 1.6 1997/07/20 18:53:12 christos Exp $
$NetBSD: util.c,v 1.15 1998/07/28 05:31:25 mycroft Exp $
usage: ls [-1ACFLRSTWacdfgiklnoqrstux] [file ...]
@(#)PROGRAM:ls PROJECT:file_cmds-60 DEVELOPER:root BUILT:Sun Jul 14 03:44:11 PDT 2002

'strings' searches a binary file for a sequence of printable characters, and displays each
sequence. The default minimum sequence length is 4, but can be overridden as in this example
(option -16). Option -a searches all sections of the file (for example, executable files have
code and data sections).

Shell Options

Use command 'shopt' to set shell options.

To set:
$ shopt -s <option>

To unset:
$ shopt -u <option>

To list:
$ shopt

Some useful options are illustrated below.

(1) Use 'cdspell' to correct minor typo's in filenames given to 'cd':
$ cd Dekstop
bash: cd: Dekstop: No such file or directory

Now set shell option 'cdspell':
$ shopt -s cdspell
$ cd Dekstop
Desktop

(2) Create short aliases for long directory names:
$ shopt -s cdable_vars
$ cal="/Users/saruman/Sites/calendar"
$ cd cal
/Users/saruman/Sites/calendar

Note: don't use the '~' alias in 'cdable' variables.

More Shopt Options

Continuing from Monday's tip:

(3) Enable case-insensitive globbing:
$ ls -d d*
ls: d*: No such file or directory

Now enable case-insensitive:
$ shopt -s nocaseglob
$ ls -d d*
Desktop Development Documents

Note: for case-insensitive completion see: Week 66 - Friday.

(4) Include dot files in globbing:

$ ls -d *
Desktop Documents Movies Pictures Sites osxfaq
Development Library Music Public bin

Now enable dot globbing:

$ shopt -s dotglob
$ ls -d *
.DS_Store .dir_colors .mplayer Desktop
.FBCIndex .emacs.d .my.cnf Development
.gimp-1.2 .mysql_history Documents .gvimrc
...

(5) Enable enhanced pathname expansion. Normally just '*', '?', and '[...]' are recognized (as in
'ls *' or 'ls file.???').
$ shopt -s extglob

The following extended globbing is enabled:
?(pattern-list) Match 0 or 1 occurrence of the given patterns
*(pattern-list) Match 0 or more occurrences of the given patterns
+(pattern-list) Match 1 or more occurrences of the given patterns
@(pattern-list) Match exactly one of the given patterns
!(pattern-list) Match anything but one of the given patterns

Prompt Commands

Have bash execute a command before it issues a prompt.

To print the date before each prompt:
$ declare -x PROMPT_COMMAND='date'
Mon Feb 2 12:02:41 GMT 2004
$

Write a message to the title bar of the Terminal window (in this example the output from command
date):
$ declare -x PROMPT_COMMAND='echo -n "^[]0; $(date)^G"'

Notes:
^[ = escape, got by typing contol-V then escape.
^G = control G, got by typing control-V control-G.

This works for xterm and Apple's Terminal.app

This can be useful:
$ declare -x PROMPT_COMMAND='echo -n "^[]0; $TITLE^G"'
$ declare -x TITLE="Hello"

Now any executing command or script can report to the title bar by setting environment variable
TITLE.

Aliases With Parameters

Unlike tcsh, bash aliases do not accept parameters. This is because bash provides a more powerful
mechanism by way of functions.

Write a function called 'word' which searches the Webster's dictionary at /usr/share/dict.web2:
$ function word () { grep $* /usr/share/dict/web2; }

$ word apple
apple
appleberry
appleblossom
applecart
appledrane
applegrower
applejack
...

$* expands to all parameters passed to function 'word'. $1 expands to parameter 1, etc.

To list all functions use:
$ declare -f

Extending the 'alias' Command

If you wish for the 'alias' command to list functions (see Thursday's tip) declare this function:

function alias ()
{
if [ -z $* ] ; then
builtin alias;
declare -f;
else
builtin alias $* ;
fi
}

Basic HereDoc

Redirect text to standard input using Here Documents.
$ cat hd
#!/bin/bash

echo "Test"

cat <<EOD
Hello there.
This text is the result
of a 'Here Document'
EOD

echo "End"

The section between '<<EOD' and the line containing just 'EOD' is read by the shell and becomes
standard input for the preceding command. In this case 'cat', given no filename parameters, will
read standard input (the supplied text) and write it to standard output.
$ ./hd
Test
Hello there.
This text is the result
of a 'Here Document'
End
$

Variable Expansion

Control variable expansion within a HereDoc.

Shell variables within a HereDoc are expanded as usual:
$ cat hd
#!/bin/bash

today=$(date)

cat <<EOD
Hello there. Today is $today.
EOD

$ ./hd
Hello there. Today is Mon Feb 16 11:53:46 GMT 2004.
$

But, if the delimiter is quoted no variable expansion is performed:
$ cat hd
#!/bin/bash

today=$(date)

cat <<'EOD'
EOD

$ ./hd
$

Remove Indentation

Remove shell script indentation from HereDoc text.

Non-trivial shell scripts usually nest statements, and it is natural to use indentation. To allow
the HereDoc to follow the natural flow of indentation, without including the tab characters in
the HereDoc text, use '<<-'.

$ cat hd
#!/bin/bash

if [ "$1" = "" ]; then
cat <<-EOD
Hello there.
Today is $(date).
EOD
fi

$ ./hd
Hello there.
Today is Mon Feb 16 12:08:16 GMT 2004.
$

Read Data

Use a HereDoc and a function to set variables.
$ cat hd
#!/bin/bash

read_data ()
{
read make
read model
read colour
}

read_data <<HEREDOC
BMW
3 series
Blue
HEREDOC

echo "Make: $make, model: $model, colour: $colour"

$ ./hd
Make: BMW, model: 3 series, colour: Blue
$

Comment Out / Test Variables

Thanks to Ian Ferguson for Friday's Tips.

Use HereDocs to comment out code, avoiding having to add '#' to the start of every line:

$ cat hd
#!/bin/bash

echo "before"
:<<'EOD'
if [ "$1" = "" ]; then
echo "No parameter given"
fi
EOD
echo "after"

$ ./hd
before
after
$

Here the 'if' statement has been commented out.

Use HereDocs to display an error message if a variable is not set:
$ cat hd
#!/bin/bash

var1=hello

: <<ERRORIFNOTSET
${var1?} ${var2?}
ERRORIFNOTSET

$ ./hd
./hd: var2: parameter null or not set
$

Process on Extension

Rename a group of files in a single command.

This will *not* work:

$ mv *.html *.php
usage: mv [-fi] source target
mv [-fi] source ... directory

The following script, called 'ext' for extension-based, will let you form such a command:
$ cat ext
#!/bin/bash
shopt -s nullglob
for file in *.$2; do
$1 "$file" "${file%.$2}.$3"
done

Use 'ext' like this to rename (mv) *.html to have an extension of .php:
$ ext mv html php

Use any similar command, such as 'cp', with the 'ext' script.

Add parameter checking to the script:

$ cat ext
#!/bin/bash

if [ "$3" = "" ] || [ "$4" != "" ]; then
echo 'For each file *.ext1, executes "command file.ext1 file.ext2"'
echo ' eg "cp html php" to copy all html files to php equivalents'
echo ' "mv html php" to rename all html files to have a php extension'
echo "Usage: ${0##*/} command ext1 ext2"
exit
fi

# this option ensures that no match for *.$2 returns null not *.$2
shopt -s nullglob

for file in *.$2; do
$1 "$file" "${file%.$2}.$3"
done

Recursive Process on Extension

Rename a group of files recursively in a single command. The script 'rext' does the same as
Monday's 'ext' script, but applies the command to all matching files in the directory hierarchy.
It uses the 'ext' script.

$ cat rext
#!/bin/bash
for dir in $(find . -type d); do
pushd $dir > /dev/null
ext $1 $2 $3
popd > /dev/null
done

Use 'rext' like this to copy (cp) *.php to equivalent files with an extension of .html:
$ rext cp php html

Use any similar command, such as 'mv', with the 'rext' script.


$ cat rext
#!/bin/bash

if [ "$3" = "" ] || [ "$4" != "" ]; then
echo 'Executes "command file.ext1 file.ext2" on each matching file file in'
echo ' the directory *hierarchy*'
echo ' eg "cp html php" to copy all html files to php equivalents'
echo ' "mv html php" to rename all html files to have a php extension'
echo "Usage: ${0##*/} command ext1 ext2"
exit
fi

for dir in $(find . -type d); do
pushd $dir > /dev/null
ext $1 $2 $3
popd > /dev/null
done

Process Multiple Files

If you have a script or command that is not able to take a list of files, use the 'each' script
to simulate this. 'each' executes the given command once for each file.

$ cat each
#!/bin/bash
filetype=$1
shift

for file in $filetype; do
$* "$file"
done

Use the script like this to apply 'my-script' to all .html files in the current directory:
$ each \*.html my-script -options ...


$ cat each
#!/bin/bash

if [ "$2" = "" ]; then
echo "Executes a command on each matching file in the current directory"
echo "Usage: ${0##*/} filetype command-to-execcute"
exit
fi


filetype=$1
shift

for file in $filetype; do
$* "$file"
done

Recursive Process Multiple Files

The script 'reach' does the same as Wednesday's 'each' script, but executes the given command on
all matching files in the directory hierarchy.

$cat reach
#!/bin/bash
filetype=$1
shift
find . -name "$filetype" -print0 | xargs -0 -n1 $*

Use the script like this to apply 'my-script' to all .php files in the directory hierarchy:
$ reach \*.php my-script -options ...


% cat reach
#!/bin/bash

if [ "$2" = "" ]; then
echo "Executes a command on each matching file in the directory *hierarchy*"
echo "Usage: ${0##*/} filetype command-to-execute"
exit
fi

filetype=$1
shift
find . -name "$filetype" -print0 | xargs -0 -n1 $*

Sed Extension

Create a simple wrapper for the 'sed' command (or any similar command) to enable it to process a
file directly. Normally sed uses standard in and out, and cannot write back to its input file.

$ cat sedx
#!/bin/sh
tmp=tmp-file-for-$PPID
{ sed "$1" "$2" > $tmp \
&& mv $tmp "$2"
} || rm $tmp

Note that the more obvious line:
sed "$1" "$2" > $tmp ; mv $tmp "$2"

should not be used as an error in the sed script will cause all files to be wiped.

Use sedx in this way:
$ sedx 's/XXX/ZZZ/' index.html

to apply the sed command to index.html.

The script as given takes only one input file. Use the scripts from Wednesday and Thursday to
process multiple files.

Apply to all files in the current directory:
$ each \*.html sedx 's/XXX/YYY/'

Or to all files in the directory hierarchy (recursively):
$ reach \*.html sedx 's/XXX/YYY/'


$ cat sedx
#!/bin/sh

if [ "$2" = "" -o "$1" = "-usage" ]; then
echo "Executes a sed command writing back to the original file."
echo "Usage: ${0##*/} sed-command file"
echo "Or use with (r)each"
echo "Usage: (r)each filetype ${0##*/} sed-command"
exit
fi

tmp=tmp-file-for-$PPID
{ sed "$1" "$2" > $tmp \
&& mv $tmp "$2"
} || rm $tmp

Split Screen

This week's tip refer to Apple's Terminal application (/Applications/Utilities/Terminal.app).

Select split screen mode in the Terminal by clicking the icon in the top right corner...

One can scroll the top frame to view previous commands while working as normal in the bottom
frame. This allows copy and paste between the two frames.

Copy and Paste

Copy text in terminal window by highlighting it as you would in any other application (left click
and drag). The selected text is automatically put into a buffer which can be recalled by clicking
the middle mouse button*

To paste between different windows or applications, use the usual command-C (after highlighting
text) to copy, and command-V to paste. The Terminal also supports drag and drop on the selected
text.

Two command-line utilities are provided to copy and paste:
$ pbpaste
Two command-line utilities are provided to copy and paste:

(text previously copied from a GUI application)

$ echo "hello there" | pbcopy
$ pbpaste
hello there

* A three mouse button is handy when working in Apple's Terminal or xterm.

.command Files

Create a double-clickable shell script by saving it as a '.command' file. Any standard shell
script can be saved in a file with an extension '.command', and double clicking the file will
open a new terminal and run the shell script in it.

One can also use the command line utility 'open' to run the script from the command line (again,
in a new terminal window).

.term Files

Create different sets of Terminal window settings (colours, window size, etc). Save a terminal
session using File -> Save As to create a '.term' file. Double-clicking this opens a Terminal
window with exactly the same window settings as the Terminal window from which it was saved.

Place the '.command' file in folder ~ /Library/Application\ Support/Terminal/ and the '.command'
file will appear in the Terminal's menu 'File -> Library -> ...'

Examine a '.term' file and you will see:
<key>ExecutionString</key>

Add shell commands to this:
<key>ExecutionString</key>
<string>ls -al /; ls -al ~;ps -caux;df -h;exit</string>

and you get a similar effect to a '.command' file (see Wednesday), but with control over the
Terminal window settings too. Remove ';exit' if you do not wish the Terminal window to close upon
completion of the Execution String.

Other Tricks

Drag a folder or file onto the Terminal window and the full pathname will be written to the
command line.

The Terminal title can be set on the command line with:
(bssh) declare -x PROMPT_COMMAND='echo -n -e "\033]0;$USER@$HOSTNAME\007"'
(tcsh) alias precmd 'printf "\033]0;$user@$host\007"'

The above examples display the current user and hostname, using either echo or printf.

System Information

This week I will list useful commands that you might not know about, or might have forgotten
about. Use the Unix manual to look up the details.

To gather system information:

$ arch
ppc

$ machine
ppc7450

$ uname -v
Darwin Kernel Version 7.3.0: Fri Mar 5 14:22:55 PST 2004;
root:xnu/xnu-517.3.15.obj~4/RELEASE_PPC

$ hostinfo
Mach kernel version:
Darwin Kernel Version 7.3.0:
Fri Mar 5 14:22:55 PST 2004; root:xnu/xnu-517.3.15.obj~4/RELEASE_PPC

Kernel configured for a single processor only.
1 processor is physically available.
Processor type: ppc7450 (PowerPC 7450)
Processor active: 0
Primary memory available: 768.00 megabytes.
Default processor set: 94 tasks, 249 threads, 1 processors
Load average: 2.37, Mach factor: 0.22

$ system_profiler
(much output)

Working with (B)Zipped Files

If you wish to work with zipped files, use the z* set of commands. For example, use 'zmore' to
view a file instead of unzipping, viewing, the possibly rezipping.

$ ls /usr/bin/z*

/usr/bin/zcat /usr/bin/zdiff /usr/bin/zgrep /usr/bin/zmore /usr/bin/zprint
/usr/bin/zcmp /usr/bin/zforce /usr/bin/zip /usr/bin/znew

If the files are compressed with bzip2, use the bz* set of commands:

$ ls /usr/bin/bz*
/usr/bin/bzcat /usr/bin/bzegrep /usr/bin/bzip2 /usr/bin/bzmore
/usr/bin/bzcmp /usr/bin/bzfgrep /usr/bin/bzip2recover
/usr/bin/bzdiff /usr/bin/bzgrep /usr/bin/bzless

Talking Scripts

Use the 'say' command directly (i.e. without going through osascript) to add speech to scripts:
$ say "Hello world."

Lookup some words to say with look:
$ look zy
zyga
zygadenine
Zygadenus
Zygaena
zygaenid
Zygaenidae
...

Or search the dictionary with grep:
$ grep "^d...p$" /usr/share/dict/web2
dadap
decap
dreep
droop

Or the online dictionary for the meaning of 'fuci':

$ curl dict://dict.org/d:fuci
220 pan.alephnull.com dictd 1.8.0/rf on Linux 2.4.18-14 <auth.mime>
<1904283.29426.1084790981@pan.alephnull.com>
250 ok
150 1 definitions retrieved
151 "Fuci" web1913 "Webster's Revised Unabridged Dictionary (1913)"
Fucus \Fu"cus\, n.; pl. {Fuci}. [L. rock lichen, orchil, used as
a red dye, red or purple color, disguise, deceit.]
1. A paint; a dye; also, false show. [Obs.]

2. (Bot.) A genus of tough, leathery seaweeds, usually of a
dull brownish green color; rockweed.

Note: Formerly most marine alg? were called fuci.
.
250 ok [d/m/c = 1/0/22; 0.000r 0.000u 0.000s]
221 bye [d/m/c = 0/0/0; 0.000r 0.000u 0.000s]

Software Update

Perform a software update, perhaps remotely, with the Software Update command line tool:
$ softwareupdate -l
Software Update Tool
Copyright 2002-2003 Apple Computer, Inc.
Your software is up to date.

Don't forget 'nvram' to manipluate open firmware parameters:

To list nvram variables:
$ nvram -p

To set verbose login (equivalent to Command-V on startup)
$ sudo nvram boot args="-v"

Use 'bless' to change the boot volume:
sudo bless -folder '/Volumes/vol-name/System/Library/CoreServices' -setOF

Image Manipulation

Use the Scriptable Image Processing System (sips command) to manipulate images via the command
line or AppleScript.

There is no manual page, so use:

$ sips --help

Command Line Editing

This week's tips help to make entering commands in Bash quicker and easier.

First off is command line editing:
Edit the current line (or one from the command line history recalled by pressing the up arrow
cursor key) with more than the usual cursor key and back-deleting (del).
control-A moves to the start of the line
control-E moves to the end of the line

escape-del deletes the word behind the cursor
del deletes the character behind the cursor
control-d deletes the character forward of the cursor
esc-d deletes the word forward of the cursor

control-k deletes to the end of line
control-u deletes the whole line

These commands will be familiar to Emacs users.

To use vi-style command line editing, issue the command:
$ set -o vi

and to get back to Emacs:
$ set -o emacs

Search the Command Line History

Quickly find a command line you have issued previously, and repeat or edit it. Hit control-r then
start typing the command:
$ control-r
(reverse-i-search)`':
type si
(reverse-i-search)`si': vim signal.c
type t
(reverse-i-search)`sit': vim ~/sites/index.html

Start typing any part of the command line, for example the command, or the filename, and bash
will back-search its history for the first command that matches what you have typed so far,
anywhere on the line. Keep typing until the correct command is showing, or press control-r again
to search further back. When the desired command is showing, press return to execute it, or the
forward arrow cursor key to edit it.

Repeating

Repeat a sequence of commands:
Hit the up arrow cursor key until the first command in the sequence to repeat is displayed. Then,
instead of hitting return, hit control-o. The command will be executed then the following command
will be displayed ready to issue/edit.

The sequence escape-. recalls the filename from the previous command.

Issue:
$ ls -al osxfaq

then:
$ cd escape-.

will be changed into
$ cd osxfaq

- particularly useful for long or awkward filenames.

Additionally, the shell variable $_ can be used:
$ echo $_
osxfaq

Command Line Expansion

You probably know that hitting tab will complete a filename. Use Tab to also complete commands,
aliases, and functions.

If the word begins with $, then tab completion will complete based on the currently defined shell
variables. Similarly, ~ (tilde) completes based on user names, and @ completes based on
hostnames.

For @ to work, the variable HOSTFILE must contain the name of a file, which contains entries in
the same format at /etc/hosts. (See 'man hosts'.)

Avoiding History

Prevent particular lines from entering the command history; useful if the command contains
sensitive information such as a password.

There are two techniques:

Set the environment variable HISTCONTROL:
$ declare -x HISTCONTROL="ignoreboth"

Now command lines starting with space are not entered into the history. Additionally, lines
matching the last line in the history are not added.

To control these aspects individually, use 'ignorespace' or 'ignoredups' instead of 'ignoreboth'.

The second technique is to set the environment variable HISTIGNORE to a colon-separated list of
patterns. For full details see 'man bash' and search for HISTIGNORE.

Getting the Right Help

Use Bash's help system for information on its built-in commands. (These will not be found in the
Unix man pages.)
$ help

to list them all.
$ help <command>

to display the help page for the given command.

Supply a pattern and Bash will display a help page for each command that matches the pattern, or
a short synopsis if option -s is given.
$ help -s "re*"

Getting the Right Command

When you issue a command such as 'ls', Bash may execute an alias, a function, a built-in, or one
from the search path ($PATH). Determine the type of command Bash will execute using the command
'type'. This in analogous to the 'which' command in the tcsh shell.
$ type ls
ls is aliased to `/usr/local/bin/ls --color=tty'

View all possibilities with:
$ type -a ls
ls is aliased to `/usr/local/bin/ls --color=tty'
ls is /usr/local/bin/ls
ls is /bin/ls

Forcing the Right Command

Tuesday's tip shows you how to determine which variant of a command Bash will execute. What if
you wish to override this?

Bash's search order is:
aliases, keywords, functions, built-ins, commands from $PATH

Use the following Bash built-ins to change the search order for a one-off command:

* 'command' forces only built-ins or commands from $PATH to be considered, ignoring aliases,
keywords, and functions
* 'builtin' forces only built-ins to be considered
* 'enable' forces only commands from $PATH to be considered

For example:
$ command ls

forces ls from $PATH to be executed, overriding any alias or function you may have defined for
ls.

Note: the 'tcsh' shell uses a backslash
% \ls

to override aliases and built-ins.

Aliases

List all aliases with:
$ alias

Set an alias with:
$ alias la="/bin/ls -Al"

Remove an alias with:
$ unalias

Here is a trick to alias directory names. Normally Bash does alias substitution on the first word
of a command line (the command) only...
$ pwd
/Users/saruman

$ alias pref="~/Library/Preferences/"
$ cd pref
-bash: cd: pref: No such file or directory

...hence this does not work.

But alias cd as below (note the trailing space)...
$ alias cd="cd "
$ cd pref
$ pwd
/Users/saruman/Library/Preferences

... and now it works.

Functions

In Bash, unlike the tcsh shell, aliases cannot take parameters. Bash has the more powerful
'function' construct.

Declare a function 'psx':
$ psx () { ps axww | grep -i $* | grep -v grep;}

The special variable $* is expanded to all parameters passed to the function. Alternatively, use
$1, $2, ... for the first and second (and so on) parameters. This is exactly the same scheme as
that used to expand parameters passed to shell scripts.

$ psx safari

503 ?? S 12:36.51 /Applications/Safari.app/Contents/MacOS/Safari -psn_0_2359297

Here is a function to view log files:
$ tlog () { tail -f -n40 /var/log/$1.log;}

And:
$ tlog system

will view the system log (control-C to exit).

To list all functions:
$ declare -f

or
$ declare -F

to list just the function names.

To Sub-Shell or Not to Sub-Shell

Know the difference between running a shell and sourcing it. This example illustrates.
$ declare -x NEW_ENV_VAR="XXXX"
$ new_var="xxxx"
$ echo $NEW_ENV_VAR
XXXX
$ echo $new_var
xxxx

$ cat tst
#!/bin/bash
declare -x NEW_ENV_VAR="TEST"
new_var="test"

Run the script:
$ ./tst
$ echo $NEW_ENV_VAR
XXXX
$ echo $new_var
xxxx

Notice the variables haven't changed...

Source the script:
$ source tst
$ echo $NEW_ENV_VAR
TEST
$ echo $new_var
test

...and this time they have.

Running a script as in the first example launches a new shell. When the new shell exits its
environment is not passed back to the parent shell, and hence its environment/shell variables are
lost, and those of the original shell remain unchanged.

'Sourcing' a script executes it with the current shell. The environment/shell variables set by
the script are those of the current shell and so remain set after the script has completed.

Note that:
$ source ./tst

and
$ . ./tst

are equivalent.

Parenthesis

Use parenthesis to execute commands in a sub-shell.
$ cat tst2
new_var="xxxx"
echo $new_var

(new_var="test")
echo $new_var

$ ./tst2
xxxx
xxxx

You might have expected this script to echo
xxxx
test

Because the second assignment is in parenthesis it is executed as a sub-shell, and (remembering
Monday's tip) has no effect on the parents shell's environment.

You may wish to execute many commands in a sub-shell when the output from all of them is
redirected or piped.

As a illustrative example:
$ echo hello; echo world > outfile
hello
$ cat outfile
world

$ (echo hello; echo world) > outfile
$ cat outfile
hello
world

Braces

Use braces to execute a sub-shell.

Tuesday's tip executed a list of commands in parenthesis:
$ (echo hello; echo world) > outfile

You can achieve the same effect with braces:
$ { echo hello; echo world;} > outfile

except that the command list is executed as part of the current shell, not as a sub shell. (Like
the difference between running and sourcing as explained in Monday's tip.)

Note the space after the opening brace and the semicolon before the closing brace. The syntax
requirements are stricter for braces than for parenthesis.

eval

Use 'eval' to override the shell's command line parsing order.

Eval is a subtle but useful command. Consider this script to run commands in the background.
$ cat tst
#!/bin/bash
"$*" > ~/outfile &

$ tst 'du -sk *'
./tst: line 2: du -sk *: command not found
$ tst 'ls | grep "^D"'
./tst: line 2: ls | grep "^D": command not found

The script fails because the shell takes the first token ("$*") as the command and *then* expands
$*. We need to change the shell's order of evaluation. Removing the quotes around "$*" won't
quite work:
$ cat tst
#!/bin/bash
$* > ~/outfile &

This is fine:
$ tst 'du -sk *'
$ cat ~/outfile
5524 Desktop
1992 Development
...

But the second command still fails. Again, the shell tokenises before it expands, and hence fails
to recognise the '|' as a token and therefore as the pipe symbol.

$ ls: |: No such file or directory
ls: grep: No such file or directory
ls: "^D": No such file or directory

Now try with 'eval':
$ cat tst
#!/bin/bash
eval "$*" > ~/outfile &

$ tst 'du -sk *'
$ cat ~/outfile
5524 Desktop
1992 Development
...

$ cat ~/outfile
Desktop
Development
Documents

This works. Why? The shell expands "$*" as usual. What eval does is to re-evaluate the command
line, thus giving the shell a second stab at tokenizing the '|' symbol as a pipe.

nohup

Start a sub-shell command list, or a script, using 'nohup' if you want it to remain running after
the shell has terminated.

Normally the shell sends a HUP signal to all its sub-processes when it itself receives a HUP or
and 'exit' command. 'Nohup' prevents this.

Alternatively, build a signal handler into the script to catch the HUP signal and ignore it.

Note:
Depending on your bash settings, bash may or may not send a HUP signal to all processes.

Check it with:
$ shopt huponexit
huponexit off

(HUP on exit is disabled.)

Enable it with:
$ shopt -s huponexit
$ shopt huponexit
huponexit on

To disable again:
$ shopt -u huponexit
$ shopt huponexit
huponexit off

'Huponexit' applies to all sub-processes, whereas the 'nohup' command is applied only to the
given process.

Change Your Shell

Use NetInfo to change your default shell. If your user was created pre-Panther, you may still be
using the tcsh shell instead of bash.

The GUI way:

Fire up NetInfo Manager in /Applications/Utilities and navigate to:

| / | users | your-user-name |

in the top panes. Click on your-user-name. Next, in the lower pane scroll down until you see
'shell' in the 'Property' pane. Change the value in the 'Value(s)' pane to be '/bin/bash'.

You will need to authenticate first by clicking the lock icon in the lower left corner and
entering an admin password.

The command line way:

Use the directory services command line (replace 'saruman' with your own short user name).
$ sudo dscl . -create /users/saruman shell /bin/bash

Re-instate Admin Status

Restore admin user status.

If your admin user loses admin status for whatever reason (and it has happened to a few people),
and you do not have a second admin account, you must boot into single user mode and use 'nicl' to
add yourself back into the admin group.

Hold down command-s on startup and wait for the text to finish scrolling. Issue the command:
$ mount -uw /

to mount the system disc with write permission (it will currently be mounted read-only).

Restore admin status using 'nicl'. You cannot use any of the other utilities like 'niload'
because the necessary system services daemon is not running.
$ nicl -raw /var/db/netinfo/local.nidb -merge /groups/admin users your-username-here

(All on one line.)

The 'ni' Utilities

Manage the NetInfo database.

Query the NetInfo database to see which users are using the 'sh' or 'bash' shell.

With nigrep:
$ nigrep "/bin/b?a?sh" .
12 /users/root: shell /bin/sh
67 /users/saruman: shell /bin/bash

nigrep takes a regular expression just like grep, and searches the database.

With nireport:
$ nireport . /users name shell
nobody /usr/bin/false
root /bin/sh
daemon /usr/bin/false
...
saruman /bin/bash

nireport takes a directory (/users in this example) and a list of properties to display.

List NetInfo directories with nidump. nidump displays NetInfo directories in the standard Unix
flat file format:
$ nidump group .
nobody:*:-2:
nogroup:*:-1:
wheel:*:0:root
...
saruman:*:501:

Load information into NetInfo with niload. niload will load the database from standard Unix flat
files:
$ sudo niload -dm group . < flat-file

Check 'man niload' to see if you require the 'd'elete or 'm'erge options.

Manage the database with niutil or nicl. I prefer to use 'dscl' for this.

Backup and Restore
The NetInfo database is backed up daily into:

/private/var/backups/local.nidump

You can do this yourself with:
$ nidump -r / -t localhost/local > local.nidump

Restore a broken database from backup:

Boot into single user mode - hold down command-s on startup and wait for the text to finish
scrolling. Issue the commands:
$ mount -uw /
$ mv /var/db/netinfo/local.nidb /var/db/netinfo/corrupt.nidb

The next steps were taken from an article by O'Reilly. They start up NetInfo and create an empty
database.
$ /usr/libexec/kextd
$ /usr/sbin/configd
$ /sbin/SystemStarter
$ /usr/libexec/create_nidb
$ /usr/sbin/netinfod -s local

Finally use niload to load the backed up database:
$ niload -d -r / . < /path/to/backed-up/database

Override Lookup Order

In Panther, most of the Unix flat files are searched, as well as NetInfo and Directory Services.

Check this search order for services with:
$ lookupd -configuration

Change this for any service by creating a file in directory '/etc/lookupd/'. For example, to
control how NFS mounts are searched for (by default NetInfo but not the /etc/fstab 'flat file')
create a file called mounts:

$ cat /etc/lookupd/mounts
LookupOrder Cache FF NI DS

This says to search the Flat Files, then NetInfo, then Directory Services.

Unix signals and how to handle them in Bash shell scripts

Send a signal to a running process using 'ps' and 'kill' with a simple bash function.

Function:
$ killsig () { kill -$1 $(ps x | grep $2 | grep -v grep | awk '{print $1}'); }

Usage:
$ killsig TERM processname

List some of the more common signals:
$ man kill

Remember that some signals are mapped to keys:
INT - control-c
SUSP - control-z
QUIT - control-\

List mappings with stty:
$ stty -e

Change them (map interrupt to control-x instead of control-c):
$ stty intr ^x

Catch Signals

A signal is an interrupt sent to a running Unix process. Normally the signal causes the process
(executable or script) to terminate. However, a process can catch the signal and take alternative
action.

Catch signals in bash scripts:

$ cat trap1
#!/bin/bash

trap 'echo "On no you dont"' INT
trap 'echo "not that way either"' TERM

while true; do
echo "Hello........"
# use sleep to represent a lengthy block of code
sleep 1
done

If the script receives an INT or TERM signal via 'kill', or an INT signal via control-c, it will
execute the command given by the appropriate trap and then continue from where it left off. In
this example it will display a message on the screen.

Run trap1:
$ ./trap1
Hello........
^XOn no you dont
Hello........
not that way either
Hello........
Hello........

In another window (using the killsig function from Monday):
$ killsig INT trap1
$ killsig TERM trap1
$ killsig KILL trap1

Syntax:
trap "command to execute" SIGNAL-LIST

Trap Functions

Use a function if you need a more complex trap handler:

$ cat trap2
#!/bin/bash

handlehup ()
{
echo "Got a HUP"
echo "Will now reload my config"
#......
}

trap "handlehup" HUP

while true; do
echo "Hello........"
sleep 1
done

If this script is sent the HUP signal, it will be caught by 'trap' and function 'handlehup' will
be executed. When the function completes the scrip will continue to execute from where it left
off.
$ ./trap2
Hello........
Got a HUP
Will now reload my config
Hello........
^C
$

$ killsig HUP trap2

The HUP signal (hangup) is traditionally used to tell a process to reload its configuration and
restart. Apache will respond to a HUP signal in this way (although one would normally use
'apachectl').

Trap Over a Section of Code

Limit your trap functions to just the critical part of a script, often around a number of
commands that should not be interrupted mid-way.

This script traps HUP, INT and TERM signals around the critical code, then drops the handler
around the normal code.

$ cat trap3
#!/bin/bash

handlesig ()
{
echo "Got an INT"
#......
}

# critical code - stop interrupts
trap "handlesig" HUP INT TERM

echo "Critical code"

sleep 1; sleep 1; sleep 1; sleep 1; sleep 1; sleep 1

# normal code - allow interrupts
trap - HUP INT TERM

echo "Normal code"
sleep 1000

The statement:
trap - SIGNAL-LIST

removes a handler from the listed signals and replaces it with the default (if any).
$ ./trap3
Critical code
Got an INT
^CGot an INT
Normal code
Hangup

$ killsig HUP trap3
(wait a bit until normal code is entered)
$ killsig HUP trap3

Trap Over a Section with Sub-Shells

Limit your trap functions to just the critical part of a script, as Thursday, but use sub-shells.
A handler assigned to a sub-shell is not inherited by the parent shell.

This script traps INT (control-c) signals around the critical code.

$ cat trap4
#!/bin/bash

# critical code - stop interrupts
(
trap 'echo "Caught by subshell"' INT

echo "Critical code"
)

# normal code - allow interrupts
echo "Normal code"
sleep 1000

$ ./trap4
Critical code
^CCaught by subshell
Normal code
^C
$

List all trap signals with:
$ trap -l

And find out more aboutth ebash built-in 'trap' with:
$ help trap

Background Jobs

Use background jobs for processes that either take a long time to complete and do not require
input/output from the Terminal, or start their own window (for example xterm).

These tips apply to the bash shell, but the tcsh shell also supports background jobs.

Start a new xterm window with:
$ xterm &
[1] 707

and once xterm is launched you will regain control of the the original terminal. The '&' symbol
tells bash to run the command in the background.

[1] is the job number
707 is the PID, as viewed by 'ps'

$ ps x | grep "[x]term"
707 std S 0:00.16 xterm

Use:
$ jobs -l
[1]+ 707 Running xterm &

to display a list of jobs and their process numbers and PIDs.

Note that each instance of bash (each terminal window) launches and controls its own list of
jobs. Typing 'jobs -l' on another terminal will not show the job with PID 707.

Jobs That Require IO

Generally, scripts that require input/output are not run in the background. Output will be
interspersed with output from the current foreground job, and input won't work.

Use file redirection if a background job produces output, or requires predictable input:
$ cat script
#!/bin/bash
read -p "Name: " name
echo You claim to be $name.
...

$ cat in
Adrian

$ ./script <in &>out &
[7] 809
$

When the job completes the terminal will display:

[7] Done ./script <in >&out

And to prove it worked:
$ cat out
You claim to be Adrian.

Background a Running Job

Re-gain terminal control from a job you should have backgrounded on launch. Hit control-z to
suspend the job, then type 'bg' to place it in the background. The job will then continue running
as if you had launched it in the background using '&'.
$ xterm (oops! I forgot the &)
^Z
[7]+ Stopped xterm

$ bg
[7]+ xterm &

$ (control returned to the current terminal)

Swapping Jobs

Use 'fg' and 'bg' to manage background jobs.
$ fg %n

brings job number n into the foreground. 'fg' with no parameters operates on the most recently
backgrounded job. You can also specify the PID number to 'fg' by omitting the %.

Supply terminal input to a background job (this example uses the script from Tuesday's tip):

$ ./script &
[2] 905
Name: $

[2]+ Stopped ./script

Here, the Terminal indicates the job 2 has stopped. Use 'jobs -l' to find out why...

$ jobs -l
[1]- 893 Running xterm &
[2]+ 905 Stopped (tty input) ./script

...it requires input the the terminal '(tty input').

Bring the job into the foreground:
$ fg %2
./script

Now supply the input:
Adrian
You claim to be Adrian.

$ bg

will send the job back into the background if it needs to run more. 'bg' can take %job-number or
PID as for 'fg'. Without any parameters it assumes the most recently foregrounded job.

Waiting...

If a shell (or script) exists while background jobs that it controls are still running, those
jobs become orphaned. They can only be stopped with:
$ kill -KILL <PID>

Prevent a script from creating orphaned jobs using 'wait'.
$ wait

will prevent the shell (and therefore the script) from exiting until all background jobs under
its control have completed.
$ wait %n

waits only for job number n. You may specify a PID too, without the preceding '%'.

[ vs Test

Use:
$ man test

or:
$ man [

to learn about the conditions one can use in Bash and Bourne shell scripts for 'if', 'while', and
'until' statements.

Conditions in Bash are evaluated by the command 'test', which can test files, strings, and
numbers. The command is also called '[' which explains the form of a Bash condition.

$ if [ 3 -lt 7 ]; then
echo "3 is less than 7"
else
echo 'very odd!'
fi
3 is less than 7

An equivalent form is:
$ if test 3 -lt 7; then ...

Why is ';' required before 'then'?

Conditions and Statements

Make more use of Bash's conditions - understand that a condition is actually a command. Bash will
execute the 'condition' and test the return value.
0 (success) = true
not 0 (failure) = false

This ties in with Monday's explanation of the 'test' and '[' commands.

For example:

$ if diff f1.txt f2.txt; then
echo "Files are the same"
else
echo "Files are different"
fi
Files are the same

The return value of a command is held in shell variable $?
$ diff f1.txt f2.txt
$ echo $?
0

AND and AND and OR or OR

Use AND and OR constructs for more complex conditions.

$ a=1;b=2;c=3
$ if [ $a -lt $b -a $b -lt $c ]; then
echo "a < b < c"
else
echo "not so"
fi
a < b < c

The test statement uses operators -a for AND and -o for OR.

Alternatively, use Bash's AND and OR operators:

$ if [ $a -lt $b ] && [ $b -lt $c ]; then
echo "a < b < c"
else
echo "not so"
fi
a < b < c

The difference between the two:

In the first example the 'test' command '[ ... ]' evaluated the complete expression.

In the second example 'test' was called twice, and bash evaluated 'first test result' AND 'second
test result'.

Bash's Built-in Conditions

Use Bash's built-in tests for integer arithmetic (and only integer arithmetic). The use of '<'
instead of '-lt' makes tests more readable. The expression is enclosed in (( ... )) rather than [
... ].

$ if (( ($a < $b) && ($b < $c) )); then
echo "a < b < c"
fi
a < b < c

Alternatively:

$ if (($a < $b)) && (($b < $c)); then
echo "a < b < c"
fi
a < b < c

Understand the difference between the two forms.

More Complex Tests

Understand how to form complex expressions. Sometimes expressions need to be bracketed to force
the correct order of evaluation. (Though in the simple example below bracketing is not strictly
necessary.)

$ a=3;b=2;c=1

$ if [ ( $a -lt $b ) -a ( $b -lt $c ) ]; then
echo "a < b < c "
else
echo "not so "
fi
-bash: syntax error near unexpected token `$a'

Syntax error, we must escape the bracketed expression.

Try:

$ if [ '( $a -lt $b ) -a ( $b -lt $c )' ]; then
echo "a < b < c "
else
echo "not so "
fi
a < b < c

3 < 2 < 1 ! You jest.

Now try:

$ if [ $ $a -lt $b $ -a $ $b -lt $c $ ]; then
echo "a < b < c "
else
echo "not so "
fi
not so

Much better.

Figure out why we get the erroneous result in the second form.

Bash Special Characters

Know your special characters.

Bash has many characters which it treats specially. If any of these characters appear naked
(unquoted or unescaped) on the command line, they are replaced or interpreted.

Here's a handy reference.

# introduce a comment
; command separator
{...} signifies a command block
(...) forces execution in a sub-shell
&& logical and (placed between commands)
|| logical or (placed between commands)

~ expanded to current user's home directory
~user expanded to user's home directory
/ directory/filename separator

$var expands variable 'var'
`...` executes a command and substitutes the output
$(...) Bash's preferred syntax for command substitution
$((...)) evaluates an integer arithmetic expression
((...)) evaluates an integer arithmetic in a condition

' strong quote
" weak quote
\ escape next character (cancel special meaning)

* wildcard
[...] character set wildcard
? single character wildcard

& forces command to be executed in the background
< redirect stdin
> redirect stdout
| pipe
! pipeline logical NOT

Any more...?

Quoting and Escaping

Prevent bash from interpreting special characters (see Monday's tip).

There are three methods:

1) Backslash escaping
$ a=5;b=4;c=3
$ echo \$1 \* \$b > \$c
$1 * $b > $c

None of $, *, or > were interpreted specially.

2) Single quotes (strong quoting)
$ echo '$a * $b > $c'
$a * $b > $c

Everything with in '...' is treated literally: the shell does not interpret any special
characters.

3) Double Quotes (weak quoting)
$ echo "$a * $b > $c"
5 * 4 > 3

Everything but $ and \ within "..." is treated literally: this allows you to escape special
characters while still expanding variables.

If you wish to mix literal $ and variable expansion $ use either backslash to escape the $:
$ echo "\$1 * $b > \$$c"
$1 * 4 > $3

Quoting Tricks

1) Quoting Parameters

Sometimes it is necessary to use single quotes around parameters, as in the awk example below:
$ echo "Adrian's" | awk '{print "You cant have " $1 " $1"}'
You cant have Adrian's $1

If I wanted to output "can't" instead of "cant", a first attempt might be:
$ echo "Adrian's" | awk '{print "You can\'t have " $1 " $1"}'
> (hit control-c)

It is not possible to escape the ' in can't because it lies within a '...'ed string.

Use:
$ echo "Adrian's" | awk '{print "You can'\''t have " $1 " $1"}'
You can't have Adrian's $1

This splits the awk parameter into three adjacent strings:
'{print "You can'
\'
't have " $1 " $1"}'

2) Remember that single quotes are not special within double quotes, so you can also use:
$ echo "You can't have my \$"
You can't have my $

or even:
$ echo "You can't have my $"
You can't have my $

because $ with nothing following is take as a literal $

Shell Evaluation

Understand the shell's evaluation order.

This does not work:
$ file="~/Desktop/"

$ ls $file
/usr/local/bin/ls: ~/Desktop/: No such file or directory

$ ls ~/Desktop/
A File

Tilde (~) is a shell special character, and has no meaning to the file system. Placing it in
quotes prevents the shell from expanding it. You might think it is expanded in the expression:
ls $file

but this is not so. The shell expands '~' before it expands $variables, and therefore never sees
the '~' until too late.

Ways around this:

1) Use eval
$ eval ls $file
A File

eval gives the shell a second go at expansion. The first pass $file is expanded, and then tilde
is expanded by 'eval'.

(See Week 78, Thursday for examples of 'eval'.)

2) Don't quote
$ file=~/Desktop
$ echo $file
/Users/saruman/Desktop

$ ls $file
A File

The first line is not quoted, so file contains the full pathname instead of '~/Desktop'

If you need to quote the assignment to variable file, use $HOME.
$ file="$HOME/Desktop"

$ ls $file
A File

$HOME is expanded on the assignment whereas ~ is not.


Multi-level Quoting

Count the number of times the shell will evaluate an expression.

This is a simple example which illustrates the principle.
$ ls A\ Dir
A File

$ echo ls A\ Dir > script

$ ./script
ls: A: No such file or directory
ls: Dir: No such file or directory

The script does not work because the shell processed the backslash escaped space when executing
echo. The file does not therefore contain the escaping:
$ cat script
ls A Dir

One must escape the escaping by writing escape-space to the file. Both must be escaped themselves
- we must write escape-escape escape-space:
$ echo ls A\\\ Dir > script
$ ./script
A File

Mount AFP Shares

Mount an AFP share using the Terminal.

An AFP share can be mounted anywhere on the filesystem - for example in /Volumes or at the root
of your home directory. The 'mount point' must be an existing directory.

For example, to mount my home directory from another machine (melkor) onto the mount point
'melkor' in my home directory on this machine, I use:
$ cd ~
$ mkdir melkor
$ mount -t afp afp://myuser:mypassword@melkor.mayo-family.com/myuser melkor
mount_afp: the mount flags are 0000 the altflags are 0020

'@melkor.mayo-family.com/myuser' is the hostname or IP address of the server followed by the AFP
share point as seen in the Finder's 'Connect...' dialogue.

$ ls
Desktop Documents Movies Pictures Sites melkor
Development Library Music Public bin osxfaq

To unmount use:
$ umount melkor

and the directory will be deleted.

AFP Mounting Script

This script is useful if you have accounts on several machines (desktop and laptop). It mounts
your home directory from the other machine.

The simplest usage is:
$ mount-user

$ mount-user -usage
mount youruser's home directory from the server yourserver

Usage: mount-user [-p password] [-h host] [-u user]
[-p password] - use an AFP password if required
[-h host] - override yourserver as the server to mount
[-u user] - override youruser as the user to mount

It assumes the following.

Your user name is the same on both machines, if not override with option '-u otherusername'.

The server name is defined in the environment variable $ALM_SERVER, if not override with option
'-h hostname/ip-address'.

The mountpoint will be $ALM_AFP_MOUNT/hostname/username, where $ALM_AFP_MOUNT is an environment
variable. The pathname in $ALM_AFP_MOUNT must exist. /Volumes is the usual pathname, but you may
want to change it to avoid confusion with Finder-mounted shares (eg to /nfs).

The script will create all the directories as necessary.

#!/bin/bash

# mounts the current user's home directory on the server
# $ALM_SERVER into the mount $ALM_AFP_MOUNT/...
#

# function: usage ([error-message])
# print out the usage lines and exit
# takes an options error meeeage to print at the
# end of the usage line
usage ()
{
echo "mount ${USER}'s home directory from server $ALM_SERVER"
echo ""
echo "Usage: ${0##*/} [-p password] [-h host] [-u user]"
echo " [-p password] - use an AFP password if required"
echo " [-h host] - override $ALM_SERVER as server to mount"
echo " [-u user] - override $USER as username to mount"
echo ""
if [ "$*" != "" ]; then echo "Error: $*."; fi
exit
}

# function: exit_if_set (current-value, parameter-description)
# called to prevent a parameter being given a
# value more than once
exit_if_set ()
{
if [ "$1" != "" ]; then
usage "$2 already has the value $1"
else
return
fi
}

# process each parameter by looping and shifting
# $2 -> $1 each iteration
#
opt=""
while [ "$1" != "" ]
do
if [ "${1:0:1}" = "-" ]; then
# found a -<char> option introducer
case "$1" in
# these options require a value, remember option for next iteration
# error if previous iteration was also an option that required a value
"-usage") usage;;
"-p" | "-h" | "-u")
if [ "$opt" = "" ]; then
opt="$1"
else
usage "no value given for $opt"
fi
;;

# invalid option
*) usage "invalid option $1";;
esac
else
case "$opt" in
# previous iteration was option that required a value, set the value
"-p") exit_if_set "$password" "password"; password="$1";;
"-h") exit_if_set "$host" "host"; host="$1";;
"-u") exit_if_set "$user" "user"; user="$1";;
"") usage "invalid value $1";;
esac
# a value has been found for -<char>, so reset this flag
opt=""
fi
shift
done

# this case is where the last option was -<char>,
# thus no value has been given
if [ "$opt" != "" ]; then
usage "no value given for $opt"
fi

# set default values for optional parameters
if [ "$host" = "" ]; then host="$ALM_SERVER"; fi
if [ "$user" = "" ]; then user="$USER"; fi

# main processing
#

# if password is given add ":" to the start to comply
# with mount syntax
if [ "$password" != "" ]; then

password=":"$password
fi

# check if already mounted
mounted=$(df | grep "$ALM_AFP_MOUNT/$host/$user")

# if not mounted, ensure the mount point (directory) is available
# delete just in case either exist as files, and hide
# any errors
if [ ! -d $ALM_AFP_MOUNT/$host/$user ]; then
rm -f $ALM_AFP_MOUNT/$host/$user 2> /dev/null
rm -f $ALM_AFP_MOUNT/$host 2> /dev/null
mkdir -p $ALM_AFP_MOUNT/$host/$user
fi
echo "Mounting afp://$user$password@$host/$user on $ALM_AFP_MOUNT/$host/$user."
mount_afp afp://$user$password@$host/$user $ALM_AFP_MOUNT/$host/$user > /dev/null
if [ "$?" != "0" ]; then
echo "Error: failed to mount."
exit 255
fi
disktool -r
fi

# return 0 if no mount was performed, 1 otherwise
exit 1
else
exit 0
fi

Enable NFS

The easiest way to enable NFS on OS X is to use NFS Manager. This is briefly described in the OSX
Tips.

NFS Manager writes Shares and Connections (Mounts) to NetInfo. This tip shows how to enable NFS
to work from the traditional Unix flat files

/etc/fstab defines which shares to automount (it is necessary on the client machine)

/etc/exports defines which directories to share (it is necessary on the server)

First, force the automount daemon to read the flat files. By default it will use NetInfo (Panther
and Jaguar).

Create the directory /etc/lookupd and the file mounts as below.

$ ls -al /etc/lookupd/
total 4
drwxr-xr-x 3 root wheel 102 Apr 18 19:23 .
drwxr-xr-x 114 root wheel 3876 Aug 15 22:39 ..
-rw-r--r-- 1 root wheel 28 Aug 19 13:19 mounts

$ cat /etc/lookupd/mounts
LookupOrder Cache FF NI

On Jaguar (but not Panther) it is necessary the force the RPC daemon to start by ensuring the
RPCSERVER is set to -YES- in /etc/hostconfig:
$ grep RPC /etc/hostconfig
RPCSERVER=-YES-

Next, reboot the machine and set up the shares (server) and mounts (client)

Set Up NFS Shares and Mounts

To define share points on the server edit /etc/exports.
$ cat /etc/exports
/Users -alldirs -maproot=nobody -network=192.168.0.0 -mask=255.255.255.0
/Users/Shared -ro -mapall=nobody

In this example directory /Users is exported.

'-alldirs' allows any subdirectory of /Users to be mounted by a client

'-maproot=nobody' ensures that root user on the client does not have root access on the server;
ESSENTIAL, unless you know exactly what you are doing

'-network...' and '-mask...' define the IP range that is allowed to mount the share. It is
ESSENTIAL that this range includes only trusted machines (see the note below).

I have exported /Users/Shared as read only (-ro). This is accessible to ALL machines (no IP
restrictions). It is exported Read Only, and all users are mapped to the unprivileged user
'nobody'.

See 'man exports' for more details.

To define mount points on the client:
$ cat /etc/fstab
# local mounts
#
...

# NFS mounts
#
carcharoth:/Users /nfs/Carcharoth nfs -b,-i,-P 0 0

This line looks for the share /Users on the host carcharoth (you may use either a hostname or IP
address) and mounts on the directory /nfs/Carcharoth. '/nfs' must exist, but the actual
mountpoint 'Carcharoth' will be created automatically.

'nfs' says that the share is nfs

The options -b,-i -P are described in 'man mount_nfs'.

The last two ( 0 0 ) describe 'dump frequency' and the order for file system checking. They are
not used.

Note:

NFS does not use passwords for authentication, but Unix User and Group IDs. It is assumed that
these are consistent across the server and all clients. That is, a user's account on the client
machine will have the same user id and group id as their account on the server (and all other
clients). This is generally true on a proper server-client network where user account information
is held centrally on the server.

If a client machine connects to your server, it need only have user with the same user id and
group id as a user on the server to gain full access to that user's files on the server.

Hide Commands that Include Passwords

Typing a password on a command line can pose a security risk as the password will be saved in
Bash's command line history, and worse, written to the history file.

Add this line in to your bash startup file (/etc/profile, /etc/bashrc, ~/.bash_profile, or
~/.bashrc):
declare -x HISTCONTROL="ignorespace"

A command line starting with space will not be added to the history. Thus start any command that
contains a password with space as a security precaution.

Delete Lines

This week's tips explore sed, a handy utility to execute a script of editing commands against its
input.

Delete blank lines from a file.
$ cat double-space
This is a file with

the lines double-spaced

and some extra blank lines

too.

Use:
$ sed '/^$/d' double-space
This is a file with
the lines double-spaced
and some extra blank lines
too.

sed works by matching each line of input against a delimited regular expression (^$ in this
example). If the line matches then sed applies the editing commands that follow. 'd' deletes the
line.

See weeks 59 and 60 for information on regular expressions, and part 6 of the Learning Centre.

To write the results back to the original file use this trick:
$ sed '/^$/d' double-space > tmp; mv tmp double-space

Delete lines containing specific text. Use the same technique with an appropriate regular
expression. This example deletes all lines starting with '$'.
$ sed '/^\$/d' text

Search for Lines

Use sed to search and print matching lines, grep-style.

sed normally echoes each input line to its output. Option '-n' suppresses this. Editing command
'p' prints each matched line. Combining the two tells sed to print just matched lines.
$ cat text
This file has a number
of lines, some of witch contain
the word witch, and some
that do not.

$ sed -n '/witch/p' text
the word witch, and some

Beyond grep, you may wish to edit the file contents and display just the edited lines:
$ sed -n 's/witch/which/p' text
of lines, some of which contain
the word which, and some

Line Ranges

Edit a range of lines using sed.

Previous examples have used a regular expression to match lines to be edited. sed can be made to
operate on a range of lines, or a single line.

Delete all lines between begin-marker and end-marker inclusive.
$ sed '/begin-marker/,/end-marker/d' filename

Or print them:
$ sed -n '/begin-marker/,/end-marker/p' filename

Or edit them:
$ sed '/begin-marker/,/end-marker/s/witch/which/' filename

If you expect the pattern 'witch' to occur more than once on a line use option 'g' (global) to
replace all occurrences on the line.
$ sed '/begin-marker/,/end-marker/s/witch/which/g' filename

Multiple Edits

Apply more than one editing command to a matched line.

Specify multiple commands to sed using one of two techniques.

Option -e:
$ cat this
this is this
and
that is that

$ sed -e 's/this/that/g' -e 's/that/twit/g' this
twit is twit
and
twit is twit

Separate multiple commands with ';':
$ sed 's/this/that/g;s/that/twit/g' this
twit is twit
and
twit is twit


Script Files

Use a script file if the sed script becomes long or complex.

Write the script in a normal text file putting each command on a new line. Use option '-f' to
specify a scrip file.
$ cat this
this is this
and
that is that

$ cat script
s/this/that/g
s/that/twit/g

$ sed -f script this
twit is twit
and
twit is twit

Print/Delete Lines

This week's tips explore Awk, a handy utility that executes a script of editing commands against
its input. Also take a look at Week 85 - The Sed Stream Editor: sed has similar capabilities.

Print lines containing specific text. This example prints all lines starting with '$'.
$ awk '/^\$/{print($0)}' text

Delete blank lines from a file.
$ awk '/.+/{print($0)}' double-space

To write the results back to the original file use this trick:
$ awk '/.+/{print($0)}' double-space > tmp; mv tmp double-space

Awk works by matching each line of input against a pattern (in these examples, regular expression
'^\$' matching a line starting with '$', and '.+' matching a line containing at least one
character). If the input line matches then Awk applies the actions that follow in {...}.
'print($0)' prints the entire line, as does a simple 'print'.

If no pattern is given then every line is matched. If no action is given then the input line is
printed. The above example can be simplified to:
$ awk '/^\$/' text

Note the difference between printing and deleting is a reversed regular expression.

Print Fields

Print specific fields of each matched input line.

Monday's tip used the action {print($0)}. $0 represents the entire input line. $n represents
field n. Fields are separated by white space.

A common application for this is to print specific information from the output of Unix commands
such as ls and ps.

Print just the pid number (field 1):
$ ps ax | grep "[p]ostfix" | awk '{print($1)}'
437

Print the month, date, and filename from a long listing:
$ ls -l | awk '{print($6,$7,$9)}'

Sep 13 csv
Sep 13 double-space
Sep 13 many
Aug 30 script
Aug 30 text
Aug 30 this
Sep 13 xxx

Use printf to format the output:
$ ls -l | awk '{printf("Date: %s %s, File %s\n",$7,$6,$9)}'
Date: , File
Date: 13 Sep, File csv
Date: 13 Sep, File double-space
Date: 30 Aug, File script
Date: 30 Aug, File text
Date: 30 Aug, File this
Date: 13 Sep, File xxx

The first line 'Date: , File' results from the first line written by ls -l. This can easily be
removed by sed, grep, or awk.

To learn about printf, see:
$ man 3 printf

Line Ranges

Process a range of lines using Awk.

Previous examples have used a regular expression to match lines. Awk can be made to operate on a
range of lines, or a single line.

Print all lines between begin-marker and end-marker inclusive.
$ cat text
begin-marker
the word witch, and some which
end-marker
do not.

$ awk '/begin-marker/,/end-marker/{print $0}' text
begin-marker
the word witch, and some which
end-marker

Or simply:
$ awk '/begin-marker/,/end-marker/' text

Multiple Commands

Execute many commands against each input line.

Separate commands with ';' and all commands will be executed against each matched line.

To print the length of each line, and the line itself, use:
$ awk '{print(length($0)); print}' text
22
12
begin-marker
...

This could be done with a single print statement, but illustrates the use of multiple commands.

Extending Wednesday's tip, we can print lines that fall outside a specific range:
$ awk '/begin-marker/,/end-marker/ {next}; {print}' text
that do not.

Matched lines execute 'next', which skips the line. Unmatched lines do not, so 'print' is
executed. Note it is different to the first example where two commands were applied to the same
pattern match:
/pattern/{command1; command2}

Here we are applying the first command to the first pattern, and the second command to the second
pattern (which is empty to match all lines).
/pattern1/{command1}; /pattern2/{command2}

This does not work:
$ awk '/begin-marker/,/end-marker/ {next; print}' text

Multiple Patterns

Specify many patterns to match against each line:
$ cat many
line one
the second line
and the third
the fourth
line five
the six
and the last line

$ awk '/^the//^line/' many
line one
the second line
the fourth
line five
the six

Apply a different command to each pattern:
$ awk '/^the/{printf("THE: %s\n",$0)}; /^line/{printf("LINE: %s\n",$0)}' many
LINE: line one
THE: the second line
THE: the fourth
LINE: line five
THE: the six

Apply a command to lines that match one pattern or the other using the 'or' (||) operator:
$ awk '/^the/||/^line/{printf("MATCHED: %s\n",$0)}' many
MATCHED: line one
MATCHED: the second line

MATCHED: the fourth
MATCHED: line five
MATCHED: the six

Apply a command to lines that match both patterns using the 'and' (&&) operator:
$ awk '/the/&&/line/{printf("MATCHED: %s\n",$0)}' many
MATCHED: the second line
MATCHED: and the last line

Reading CSV Files

Change the default field separator used by Awk.

Comma Separated Value (CSV) files are a popular method of exporting tables of data (spread sheets
can export in CSV format). For example:
$ cat csv
scott,sheppard,mr,editor in chief,01
adrian,mayo,mr,editor,02
jan,forbes,miss,goddess,69

Awk will normally use white space as a field separator. To change this use the -F option:
awk -F "," '{print $2,$4}' csv
sheppard editor in chief
mayo editor
forbes goddess

Alternatively, we can set the Awk variable 'FS' to be ',' using BEGIN.
$ awk 'BEGIN {FS=","} {print $2,$4}' csv
sheppard editor in chief
mayo editor
forbes goddess

Commands in {...} following BEGIN are executed before the Awk script starts.

The field separator is actually a regular expression and can take a list of separators, or
regular expression wild cards.
$ cat csv2
scott:sheppard,mr,editor in chief-01
adrian:mayo,mr,editor-02
jan:forbes,miss,goddess-69

$ awk 'BEGIN {FS=",|:|-"} {printf "1=%s, 2=%s, 3=%s, 4=%s, 5=%s\n",$1, $2, $3, $4, $5}' csv2
1=scott, 2=sheppard, 3=mr, 4=editor in chief, 5=01
1=adrian, 2=mayo, 3=mr, 4=editor, 5=02
1=jan, 2=forbes, 3=miss, 4=goddess, 5=69

Editing With Awk

Examples so far (including week 86) have used Awk to filter, but not to edit. Week 85 showed how
to use 'sed' to edit files. Awk can also edit files.

Search and replace with Awk.
$ cat text
This file is one witch has a number
begin-marker
the word witch, and some of witch

end-marker
do not.

$ awk -v t=which '/begin-marker/,/end-marker/ {sub('/witch/', t, $0); print $0}' text
begin-marker
the word which, and some of witch
end-marker

This example applies the script {sub('/witch/', t, $0); print $0} to each line within the
markers. 'sub' is a function that substitutes its pattern (witch) for the value of a variable
(t), on its third parameter. In this case $0 represents the whole input line, but field n could
also be specified using $n. Variable t is set to 'which' using the command line option -v.

Notice that only the first occurrence on each line of pattern 'witch' is substituted. To
substitute all patterns on a line use 'gsub'. (Global SUBstitute)

Line Counting

Count lines, and more.

Awk uses variables like any other language. This example uses simple arithmetic to count the
number of lines in a file:
$ cat text
This file is one witch has a number
begin-marker
the word witch, and some of witch
end-marker
do not.

$ awk 'BEGIN {n=0} {n=n+1} END {printf ("Total lines in file %d\n", n)}' text
Total lines in file 6

{n=0} is executed once at the start of the script by the BEGIN statement. {n=n+1} is executed
unconditionally for each line. END {...} executes its statements just once at the end of the
script.

Here is another example which also prints the number of each line beginning 'the', and the value
of field two:
$ awk 'BEGIN {n=0} /^the/{printf("Line %d: %s\n", n, $2)} {n=n+1} END {printf ("Total lines in
file %d\n", n)}' text
Line 3: word

Use a Script File

Complex Awk scripts can be placed in a file. Repeating Wednesday's tip using a script file:
$ cat awk-script
BEGIN {n=0}
/^the/{printf("Line %d: %s\n", n, $2)}
{n=n+1}
END {printf ("Total lines in file %d\n", n)}

Each statement is placed on a new line in the script file. Use option '-f' to tell Awk to read
its script from a file:
$ awk -f awk-script text
Line 3: word

More Power

Awk is a fully-fledged programming language, with c-style statements such as 'for', 'while', and
'if'.

In triplicate:
$ cat file
line 1
line 2

$ awk '{for (i = 0; i < 3; i++) print $0}' file
line 1
line 1
line 1
line 2
line 2
line 2

This is an example using 'if' statements:
$ cat posts
Mayo 34 posts
Forbes 35 posts
Sheppard 12 posts
Trevor 345678 posts
Hollis 17 posts

$ cat awk-script
BEGIN { print "More than 34 posts"; max = 0; name = ""}
{if ($2 > 34) print $0}
{if ($2 > max) {max = $2; name = $1}}
END { printf ("Max posts %d by %s\n", max, name); print "---\n"}

$ awk -f awk-script posts
More than 34 posts
Forbes 35 posts
Trevor 345678 posts
Max posts 345678 by Trevor
---

Repair Discs

Use diskutil to verify and repair disks. To repair, the disk must not be in use, and diskutil may
have to run as root.

For example, use df to find the device node:

$ df
Filesystem Size Used Avail Use% Mounted on
/dev/disk0s3 12G 6.5G 5.9G 53% /
/dev/disk0s5 12G 3.4G 9.1G 27% /Users
/dev/disk0s7 5.9G 578M 5.3G 10% /Games
/dev/disk0s9 5.9G 21M 5.8G 1% /Music

(or 'diskutil list')

and then:
$ diskutil verifyDisk /dev/disk0s7
Started verify/repair on disk disk0s7 Games-saruman
Checking HFS Plus volume.
Checking Extents Overflow file.
Checking Catalog file.
Checking Catalog hierarchy.
Checking volume bitmap.
Checking volume information.
The volume Games-saruman appears to be OK.
Verify/repair finished on disk disk0s7 Games-saruman

Or use the mount point (usually /Volumes/name):
$ diskutil verifyDisk /Games

Use 'diskutil repairDisk' to repair.

Repair Permissions

Use diskutil to verify and repair file permissions on the system disc. You can only specify an OS
X boot volume, and to repair, the diskutil command must be run as root.

If you encounter odd problems with system functions or applications not behaving correctly,
'Verify and Repair Permissions'.

Use df to find the device node, as for Monday's tip, or specify the mount point, and then:
$ diskutil verifyPermissions /dev/disk0s3
Started verify/repair permissions on disk disk0s3 OSX-saruman
Determining correct file permissions.
...

To repair the current classic boot volume use:
$ diskutil repairOS9Permissions
You cannot repair OS 9 permissions on this machine (No Classic folders found)

OK, so I don't have OS 9 installed :-)

Verify Preferences

Verify all those .plist preference files in ~/Library/preferences.

If an application is playing up, it may be due to a corrupted preference file. Use command
'plutil' to verify the preference file.
$ plutil -lint ~/Library/Preferences/com.apple.iChat.plist
/Users/saruman/Library/Preferences/com.apple.iChat.plist: OK

This will only check the syntax of the file, not the deeper meaning as understood by the owning
application.

To check all preferences:
$ find ~/Library/Preferences -name "*.plist" -print0 | xargs -n1 -0 plutil -lint

(This uses -print0 and -0 to cope with spaces in filenames.)

Periodic Housekeeping

OS X tidies the file system by removing old temporary files, compressing and deleting log files,
removing old system messages, backing up the NetInfo database, updating the 'locate' database,
and more...

Periodic system maintenance is scheduled by 'cron' launching a program called 'periodic'.

% cat /etc/crontab
...
...

The daily script is run at 3:15 am.
The weekly script is run at 4:30 am Saturday.
The monthly script is run at 5:30 am on the first of each month.

The output from these scripts can be found in:

% ls /var/log/*.out
/var/log/daily.out /var/log/monthly.out /var/log/weekly.out

Periodic looks in /etc/periodic/parameter-given-periodic and executes all scripts in the
directory.

Run the maintenance scripts manually:
% sudo periodic daily
% sudo periodic weekly
% sudo periodic monthly

Alternatively, edit /etc/crontab and choose times at which your Mac will be running:

If you wish to add your own maintenance tasks - daily, weekly, or monthly - then create the
following files as necessary.
/etc/daily.local
/etc/weekly.local
/etc/monthly.local

These are regular shell scripts, and once created are run automatically by the default daily,
weekly, and monthly scripts.

These files will not be overwritten by system updates which might otherwise affect the files in
/etc/periodic.

File System Monitoring Commands

Use commands to monitor what is happening in the file system.

Use fs_usage to report system calls in real-time. For example, to report all fs activity to a log
file:
$ fs_usage -f filesys > log

To monitor just fs activity of the posfix master daemon:
$ fs_usage -w -f filesys master

fs_usage must be run as root.

A similar command, sc_usage, monitors all system calls.

Use fstat to report on open files. For example, to monitor files opened by user saruman:
$ fstat -u saruman

To monitor files as they are opened, use lsof.

Capture Patterns

This week continues the sed tips started in week 85.

Use capture patterns to record and replay matched patterns.
$ cat capture
abc (ref:) def

To simply delete (ref:) (see week 85) use:

$ sed 's/(ref:)//' capture
abc def

To capture (all or part of) the pattern, surrounding the segment to capture with $ ...$. To
replay it use \1:
$ sed 's/($ref:$)/\1/' capture
abc ref: def

And to emphasise what is happening:
$ sed 's/($ref:$)/\1 again \1 and again \1/' capture
abc ref: again ref: and again ref: def

The next example uses a pattern to match the entire line in three parts, and surrounds each part
with a capture pattern. It then replays the patterns in a different order.

Move '(ref:nnnn)' from the middle of the line to the end.
$ cat refs
For news (ref:1) see "All the News".
For help (ref:223) see "Technical Help".
For technical details (ref:5488) see "How it Works".

$ sed 's/$^.*$$ (ref:[0-9]\{1,4\})$$.*$$/\1\3\2/' refs
For news see "All the News". (ref:1)
For help see "Technical Help". (ref:223)
For technical details see "How it Works". (ref:5488)

The pattern to move is matched with:
' (ref:[0-9]\{1,4\})'

which is a regular expression looking for ' (ref:' and then 1 to 4 digits, and then ')'.

sed is told to substitute the entire line with the first segment, the third segment, and the
second segment (the matched ref).

HTML to PHP-BB Code

Use pattern capture to convert HTML code to PHP-BB code.

HTML looks like:
Hello in bold italic

PHP-BB looks like:
[b][i]Hello in bold italic[/i][/b]

Apply a sed command to file 'html':
$ cat html
Hello in bold italic
$ sed 's/<$.*$>/[\1]/' html
[b>Hello in bold italic</b]

This has not worked because the '.*' is 'greedy' in that it matches the longest pattern it can
find. We must modify the pattern to explicitly exclude another '<': (See weeks 59 and 60 on
regular expressions.)
$ sed 's/<$[^<]*$>/[\1]/' html
[b]Hello in bold italic

Ah yes, and one must tell sed to 'globally' apply the pattern to the whole line, not just the
first match:
$ sed 's/<$[^<]*$>/[\1]/g' html
[b][i]Hello in bold italic[/i][/b]

Comment Out

Use sed to apply commands to entire blocks.

For example, comment out a block of C++ or PHP code using '//' style comments.

First, surround the code with markers (eg xxxx and yyyy).

$ cat code.php
function Manage_User()
{
// create a new session and connection to the 'users' database
$this->session = new Sess_User();
$this->user = new DB_Forums_Users('CONF_SITE_DBNAME', 'CONF_SITE_DBPASS');

// get the user's nickname, if any
$nickname = $this->session->name();

xxxxx
// if the user has a nickname, check they still exist in the user's databse
// and if not, log them out and remove their session information
if ($nickname != '') {
if (! $this->user->isRegistered($nickname)) {
$this->session->logout(SESS_FORGET);
}
}
yyyyy

}

Apply the sed command:

$ sed '/xxxx/,/yyyy/s/^/\/\//' code.php
function Manage_User()
{
// create a new session and connection to the 'users' database
$this->session = new Sess_User();
$this->user = new DB_Forums_Users('CONF_SITE_DBNAME', 'CONF_SITE_DBPASS');

// get the user's nickname, if any
$nickname = $this->session->name();

//xxxxx
// // if the user has a nickname, check they still exist in the user's databse
// // and if not, log them out and remove their session information
// if ($nickname != '') {
// if (! $this->user->isRegistered($nickname)) {
// $this->session->logout(SESS_FORGET);
// }
// }
//yyyyy

}

To remove the comments use:
$ sed '/xxxx/,/yyyy/s/^\/\///;/xxxx/d;/yyyy/d' code.php

(It is also possible, but not as easy, to come up with a sed command to remove the comments, but
keep in the xxxx and yyyy lines in but commented out.)

Case Insensitivity

Make sed pattern matches case insensitive.

Normally, sed pattern matching is case sensitive.
$ cat case
THIS IS UPPER CASE
this is lower case

$ sed -n '/CASE/p' case
THIS IS UPPER CASE

$ sed -n '/case/p' case
this is lower case

Use a script like this to match any case:
$ cat case1.script
y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/
/case/p

$ sed -n -f case1.script case
this is upper case
this is lower case

y translates letters in the first // to those in the second //, thus making the input line all
lowercase.

This matches and prints both lines, but unfortunately mangles the original line. If the original
line needs to be kept, we can use function lists and sed's holding space - see Friday's tip.


Function Lists

Use more complex operations on a matched line with function lists.

The script from Thursday can be improved:
$ cat case2.script
h
y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/
/case/ {
x
p
}

$ sed -n -f case.script case
THIS IS UPPER CASE
this is lower case

h copies the current line to the pattern holding space.

y translates letters in the first // to those in the second //, thus making the input line all
lowercase.

/case/ {...} matches all lines containing 'case' and executes the commands in {...} on those
lines

x swaps the holding space and the current line - so having matched in a case insensitive manner
we replace the mangled line with the original

p, as usual, prints the line.

Odd Behavior in Shell Conditions

This week highlights a few common problems with shell scripts and commands, illustrating where
the problem lies and how to correct it.

A command called 'test', aka '[', allows one to make all kinds of tests and comparisons. They can
involve files, strings, and numbers.

See Week 82 and:
$ man test

or

$ man [

One can compare strings to test which 'comes first', alphabetically speaking:
$ if [ "aaa" > "zzz" ]; then echo "Yes"; else echo "No"; fi
Yes

So why 'Yes'? "aaa" is not > "zzz".

The shell interprets '>' as a redirection character, and so it must be escaped:
$ if [ "aaa" \> "zzz" ]; then echo "Yes"; else echo "No"; fi
No

The first version would have created the file "zzz". Perhaps if we had tried this next example
first, it would have given a clue as to what was happening.
$ if [ "zzz" < "aaa" ]; then echo "Yes"; else echo "No"; fi
bash: aaa: No such file or directory

No

Double quotes can also be used to escape:
$ if [ "zzz" "<" "aaa" ]; then echo "Yes"; else echo "No"; fi
No
$ if [ "aaa" "<" "zzz" ]; then echo "Yes"; else echo "No"; fi
Yes
$

So let's try some numbers:
$ if [ 7 ">" 2 ]; then echo "Yes"; else echo "No"; fi
Yes

As expected, but what about this:
$ if [ 17 ">" 2 ]; then echo "Yes"; else echo "No"; fi
No

The '>' operator works on strings (or numbers which it interprets as strings). For numeric
comparison use '-gt' and '-lt':
$ if [ 17 -gt 2 ]; then echo "Yes"; else echo "No"; fi
Yes

Odd Behavior in Bash Conditions

Bash has the very useful integer arithmetic operator $((...)). See Week 82 Thursday.
$ echo $((2+7))
9

$ $( ((2>7)) )
$ echo $?
1

$ $( ((2<7)) )
$ echo $?
0

Interpreting the result of the two conditions, in shell-speak 0 represents TRUE and 1 represents
FALSE.

The construct evaluates expressions and relational expressions. It can also take $variables as
well as literal values like 7.

Try this:
$ if $((2<7)); then echo "Yes"; else echo "No"; fi
bash: 1: command not found
No

So why '1: command not found'?

A common mistake is to use $((...)) in a condition. $ effectively executes the expression to
return the value (1), but so does an 'if' statement. An 'if' statement executes the command
immediately following it (often '[ ... ]'), and so tries to execute '1' (why does it return 1 and
not 0?) returned from the evaluation of '2<7'.

Drop the $:
$ if ((2<7)); then echo "Yes"; else echo "No"; fi
Yes

The next examples are more long-winded, but also work. You can see why they work if you
understood the explanation above.
$ if (($((2<7))==1)); then echo "Yes"; else echo "No"; fi
Yes
$ if [ $((2<7)) -eq 1 ]; then echo "Yes"; else echo "No"; fi
Yes
$ if [ $((7<2)) -eq 1 ]; then echo "Yes"; else echo "No"; fi
No

Odd Behavior in Filenames

I tried this command:
$ cd ~/Library/Preference
$ /bin/ls *.plist
ls: illegal option -- .
usage: ls [-ABCFGHLPRSTWZabcdfghiklnoqrstuvx1] [file ...]

Odd - what is happening here? The clue lies in the 'illegal option' message. I have a file there
called:
-BringToFront.plist

The shell expands *.plist. Whenever a command includes an unescaped *, it is the shell that
interprets and expands it, not the command it is being passed to. So the command seen by 'ls' is:
ls -BringToFront.plist Fire.plist Vim.plist ...

-BringToFront.plist appears to 'ls' to be a list of options.

So how to get around this? Obviously the '*' cannot be escaped: it won't be expanded by the
shell, and 'ls' will complain.
$ ls "*.plist"
ls: *.plist: No such file or directory

Most commands take the special option '--' which says 'end of options'. Anything following is
interpreted as an argument, not an option.
$ ls -- *.plist
-BringToFront.plist
Fire.plist
Vim.plist
...

When specifying a filename directly one will encounter similar difficulties:

$ ls -- -*
-i
$ rm -i
usage: rm [-f | -i] [-dPRrvW] file ...
unlink fil

Either specify the filename with a path prefix to hide the leading '-', or use the -- trick
again:
$ rm ./-i
$ rm -- -i

Some versions of Unix include the primitive 'unlink' command, which takes no options and thus
doesn't get confused by filenames starting '-'...
$ unlink -i
-bash: unlink: command not found

...but not OS X.

Odd Behavior in Complex Shell Conditions

The 'test' or '[' command is able to perform 'and' and 'or'. '-a' signifies 'and', '-o' signifies
'or'. See Week 82 Friday.

It is often necessary to use ( ... ) to force the correct order of evaluation. In this example we
want:

A and (B or C)
$ if [ 1 -eq 0 -a ( 1 -eq 1 -o 1 -eq 1 ) ]; then echo "Yes"; else echo "No"; fi
-bash: syntax error near unexpected token `('

The bash shell is interpreting the brackets and not passing them to 'test'.

We could try to escape the entire expression by surrounding it with quotes:
$ if [ "1 -eq 0 -a ( 1 -eq 1 -o 1 -eq 1 )" ]; then echo "Yes"; else echo "No"; fi
Yes

This is actually the wrong answer, it should give 'no' (FALSE and (TRUE or TRUE) = FALSE). Why
does it give the wrong answer? Because we are now passing a string into 'test', not an
expression, and a string tests as TRUE (or FALSE for an empty (null) string).

The secret is to escape each bracket from the shell, so the expression expression is passed the
'test' as en expression not a string.
$ if [ 1 -eq 0 -a $ 1 -eq 1 -o 1 -eq 1 $ ]; then echo "Yes"; else echo "No"; fi
No

Odd Behavior in Shell Escaping

Here is a file that contains two instance of the sequence '!*'.
$ cat ps
This is a file
!* with two
!* sequences
in it.

Suppose we wish to run the file through grep to search for the lines containing '!*'. The first
attempt fails:
$ grep !* ps
grep ps ps

It oddly echoes 'grep ps ps' because the shell interprets '!*' as 'the parameters passed to the
last command'.

(Very useful in itself sometimes:)
$ ls a-very-long-filename-indeed-that-we-might-refer-to-in-the-next-command
a-very-long-filename-indeed-that-we-might-refer-to-in-the-next-command
$ cat !*
cat a-very-long-filename-indeed-that-we-might-refer-to-in-the-next-command
Hello

To avoid shell interpretation:
$ grep "!*" ps
grep "ps ps" ps

Still no good, because bash interprets '!' even within double quotes. See Week 83 for an
explanation of the shell's interpretation of special characters in " " and ' '.

Trying single quotes:
$ grep '!*' ps
This is a file
!* with two
!* sequences
in it.

Still no go. Remember that grep takes regular expressions. See Weeks 9, 10, 52, and others. The
'*' needs to be escaped from grep too, to say "match '*', don't interpret it as a pattern
repeater".
$ grep '!\*' ps
!* with two
!* sequences

Or:
$ grep \!\\\* ps
!* with two
!* sequences

Here we escaped:
'!' from the shell with '\!'
'*' from the shell with '\*'

Remember that '\' is a special character to the shell, so to send '\*' to grep we need to escape
both with '\\' and '\*'. Hence '\\\*'.

NetInfo Commands

This week presents some useful scripts to manage user accounts on the command line - add a new
user, a new group, and add existing users to a group.

There are many commands to query and change the NetInfo database, two of the most useful being:

dscl - Directory Services Command Line

A replacement for the older niutil that manipulates the information in NetInfo, and in fact in
any Directory Services node.

For example, to add a new user:
$ dscl . create /users/joe

and to add property-value pairs:
$ dscl . create /users/joe name joe
$ dscl . create /users/joe shell /bin/bash
$ dscl . create /users/joe realname "joe jones"

Any NetInfo subdirectory can be created, and property-value pairs added, in a similar manner.

nireport - Print tables from NetInfo

This reports on the values of the listed properties for specified NetInfo directories.

For example, list all users:
$ nireport . /users name
nobody
root
daemon
unknown
smmsp
lp
postfix
...

List all group names and their group IDs.
$ nireport . /groups name gid
nobody -2
nogroup -1
wheel 0
daemon 1
kmem 2
sys 3
tty 4
...

Other commands are:
nifind, nigrep, niload, nidump

In single user mode, when NetInfo is not running, use nicl.

These are all documented in the Unix manual.

Tuesday gives some handy one-lines for querying the NetInfo database, while Wednesday, Thursday,
and Friday give scripts to add new users and groups to NetInfo, and add users to groups.

Query NetInfo

Here are some tips on using nireport to extract information on users and groups from NetInfo.
These examples can be used in Bash scripts.

Check if a user exists:
$ user=jan
$ if [ ! -z "$(nireport . /users name | grep -w $user)" ]; then echo "Exists"; fi
Exists

$ user=xxxx
$ if [ ! -z "$(nireport . /users name | grep -w $user)" ]; then echo "Exists"; fi
root@saruman ~/bin
$

Check if a user is in a group (users root then jan in group admin):
$ group=admin
$ user=root
$ in=$(nireport . /groups name users | grep -w "$group.*$user"); if [ ! -z "$in" ]; then echo "In
group"; fi
In group

$ user=jan
$ in=$(nireport . /groups name users | grep -w "$group.*$user"); if [ ! -z "$in" ]; then echo "In
group"; fi
$

Convert a group name to GID:
$ group=admin
$ echo "$(nireport . /groups gid name | grep $group | cut -f 1)"
80

Check if this is the user's primary group:
$ user=jan
$ group=jan
$ gid="$(nireport . /groups gid name | grep $group | cut -f 1)"

$ pri=$(nireport . /users name gid | grep -w "$user[[:space:]].*$gid"); if [ ! -z "$pri" ]; then
echo "Primary group"; fi
Primary group

$ user=jan
$ group=admin
$ gid="$(nireport . /groups gid name | grep $group | cut -f 1)"

$ pri=$(nireport . /users name gid | grep -w "$user[[:space:]].*$gid"); if [ ! -z "$pri" ]; then
echo "Primary group"; fi
$

Get the UID and GID for a given user:
$ user=jan
$ nireport . /users name uid gid | grep $user
jan 520 520

Add a New User

This script adds a new user account to OS X. Give the user's first and last names, the desired
user id, and if they are to be a normal (staff) or admin user.

The user id is supplied because normal account creation in OS X does not give this option. The
user's short name will be equal to their first name. A home directory will be created so the user
is a fully-fledged OS X account holder and can log in in the normal manner.

The script will prompt for a password for the new user.

Lots of checks are made, as you can see from the script comments.

Grab the script from here.

NOTE: THIS SCRIPT IS WRITTEN FOR PANTHER (10.3)

#!/bin/bash

# Create a user.
# Takes the user's firstname (=shortname), lastname, uid, and staff|admin
# and creates:
# a new user in NetInfo passwd
# a new /Users/firstname home directory

usage ()
{
echo "Create a new staff or admin user"
echo "Usage: ${0##*/} firstname lastname uid staff|admin"
if [ "$*" != "" ]; then echo " Error: $*"; fi
exit 1
}

# The script must be run by root

#
if [ "$USER" != "root" ]; then
echo "Must be run as root."
exit 1
fi

# Check parameters
#
if [ $# -ne 4 ]; then
usage
fi

first=$1; last=$2; uid=$3; accnt=$4

# check that the users does not already have a home directory
if [ -e /Users/$first ]; then
usage "User $first already exists at /Users/$first"
fi

# search NetInfo for the given user - it should not exist
str="$(nireport . /users name | grep -w $first)"
if [ ! -z "$str" ]; then
usage "User $first already exists (but does not have a home directory)"
fi

# search NetInfo for the given uid - it should not exist
str="$(nireport . /users uid | grep -w $uid)"
usage "User ID $uid already exists"
fi

# search NetInfo for the given group - it should not exist
str="$(nireport . /groups name | grep -w $first)"
usage "Group $first already exists"
fi

# search NetInfo for the given gid - it should not exist
str="$(nireport . /groups gid | grep -w $uid)"
usage "Group ID $uid already exists"
fi

# ensure either staff or admin is given
if [ $4 != staff ] && [ $4 != admin ]; then
usage "Give account type as 'staff' or 'admin'"
fi


# Add the new user to NetInfo
#
# add user and essential properties
dscl . create /users/$first
dscl . create /users/$first name $first
dscl . create /users/$first passwd "*"
dscl . create /users/$first hint ""
dscl . create /users/$first uid $uid
dscl . create /users/$first gid $uid
dscl . create /users/$first home /Users/$first
dscl . create /users/$first shell /bin/bash
dscl . create /users/$first realname "$first $last"
dscl . create /users/$first picture "/Library/User Pictures/Fun/Smack.tif"
dscl . create /users/$first sharedDir Public

# add some other properties that are usually in NetInfo
dscl . create /users/$first _shadow_passwd ""
dscl . create /users/$first _writers_hint $first
dscl . create /users/$first _writers_real_name $first

# add the new group
dscl . create /groups/$first
dscl . create /groups/$first name $first
dscl . create /groups/$first passwd "*"
dscl . create /groups/$first gid $uid

echo "New user and group $first created"

# Add admin users to the admin group
#
if [ $4 = admin ]; then
dscl . merge /groups/admin users $first
dscl . merge /groups/appserverusr users $first
dscl . merge /groups/appserveradm users $first
echo "$first added to groups admin, appserverusr, appserveradm"
fi

# Create the home directory, populate from the template, and set owners
#
mkdir /Users/$first
if [ ! -d /Users/$first ]; then
echo "Unable to create the user's home directory /Users/$first"
exit
fi

ditto -rsrc /System/Library/User\ Template/English.lproj/ /Users/$first
chown -R ${first}:$first /Users/$first
echo "Home directory /Users/$first created and populated"

# Now give the user a password
#
echo "A password for this account must be given, it is currently blank"
passwd $first

exit 0

Add a New Group

This script adds a new group to OS X. Give the group name and id. The group will be created
without any users belonging to it. Add users to the group with add-user2group (see Friday).




#!/bin/bash

# Create a group.
# Takes a group name and gid and creates a new group in NetInfo groups

usage ()
{
echo "Create a new group"
echo "Usage: ${0##*/} groupname gid"
exit 1
}

# The script must be run as root
#
exit 1
fi

# Check parameters
#
if [ $# -ne 2 ]; then
usage
fi

group=$1; gid=$2

# search NetInfo for the given group - it should not exist
str="$(nireport . /groups name | grep -w $group)"
usage "Group $group already exists"
fi

# search NetInfo for the given gid - it should not exist
str="$(nireport . /groups gid | grep -w $gid)"
usage "Group ID $gid already exists"
fi

# Add the new group to NetInfo
#
# add group and essential properties
dscl . create /groups/$group
dscl . create /groups/$group name $group
dscl . create /groups/$group passwd "*"
dscl . create /groups/$group gid $gid
#dscl . create /groups/$group users "" breaks add-user2group if added as a blank value

echo "New group $group created"
echo "Now add users to it with add-user2group"

exit 0

Add Users to a Group

This script adds one or many users to a group. Give the group names and a list of user short
names.




#!/bin/bash

# Add new users to a group.
# Adds a user (or many users) to an existing group in NetInfo

usage ()
{
echo "Add a user (or several users) to an existing group"
echo "Usage: ${0##*/} group user [user...]"
exit 1
}

# Ensure user is root
#
exit 1
fi

# Check parameters
#
if [ $# -lt 2 ]; then
usage
fi

group=$1

# search NetInfo for the given group - it should exist
str="$(nireport . /groups name | grep -w $group)"
if [ -z "$str" ]; then
usage "Group $group does not exist"
fi

# get the group number from the name
gid="$(nireport . /groups gid name | grep $group | cut -f 1)"

# Drop the group and loop thro' additional parameters (users) to add to group
#
shift

for user in $*; do
# check if the user exists
struser="$(nireport . /users name | grep -w $user)"
# check if the user already belongs to the group
stringroup=$(nireport . /groups name users | grep -w "$group[[:space:]].*$user")
# check if this is the user's primary group
strprimary=$(nireport . /users name gid | grep -w "$user[[:space:]].*$gid")

#echo "user $struser, ingroup $stringroup, primary $strprimary"

# ensure that the user exists...
if [ -z "$struser" ]; then
echo "User $user does not exist"
# ...and does not already belong to the group...
elif [ ! -z "$stringroup" ]; then
echo "User $user already belongs to group $group - not added again"
# ...and this is not the user's primary group
elif [ ! -z "$strprimary" ]; then
echo "This is the user's primary group - not added"

else
# add user to the group
dscl . merge /groups/$group users "$user"
echo "$user added to group $group"
fi
done

exit 0

Executables as Conditions

These examples are written for the Bash shell. They are applicable to other shells but the syntax
may need modification. The construct $(...) should be replaced by `...` in tcsh.

See also weeks 56 and 65.

Incorporate the result of executing a command into a condition.

For example, I recently had to write a script that waited (looped) until a particular volume was
mounted. 'df | grep' is an easy way to test this:
$ volname=Games
$ df | grep $volname
/dev/disk0s7 5.9G 578M 5.3G 10% /Games

The technique is to loop while the output from the above command is null. Note that '[ -z string
]' is true when the string is zero length (null).

Attempt 1:
$ while [ -z df | grep $volname ]; do echo -n "."; done
-bash: [: missing `]'
grep: ]: No such file or directory

'[' expects a string, so we must capture the output of executing 'df | grep $volname' and test
that string.

Attempt 2:
$ while [ -z $(df | grep $volname) ]; do echo -n "."; done
-bash: [: too many arguments

The output from grep is a string with spaces, so we must quote the output so it appears to '[ -z
... ]' as a single string.

Use "$(...)" to return a single string from the output of a command or pipe:
$ while [ -z "$(df | grep $volname)" ]; do echo -n "."; sleep 2; done;
.....

This loops until the volume $volname appears.

Note 'echo -n' is used to avoid a newline for every dot.

Use 'sleep n' in the body of a wait loop to avoid hogging CPU time. 'n' is in seconds.

Tricky Quoting

Continuing from Monday's tip.

Understand quotes within quotes.

Suppose the volume name included spaces (or shell special characters).
$ volname="New Games"
$ while [ -z "$(df | grep $volname)" ]; do echo -n "."; sleep 2; done; echo
grep: Games: No such file or directory
.^C

We must quote $volname. But, as the expression:
"$(df | grep $volname)"

is itself already quoted, how can we quote $volname too? Using single quotes '...' for the outer
set will not work because the shell won't interpret '$(...)'.

You might expect this to work:
$ while [ -z "$(df | grep \"$volname\")" ]; do echo -n "."; sleep 2; done; echo
grep: Trailing backslash
.grep: Trailing backslash

It doesn't because Bash is being quite clever. Within "$(...)" Bash re-sets its idea of nested
quotes, and the very simple solution is:
$ while [ -z "$(df | grep "$volname")" ]; do echo -n "."; sleep 2; done; echo
......

Quotes in Quotes

This useful trick is not always obvious until it is pointed out.

These should be obvious:
$ word=Hi

$ echo "He said $word"
He said Hi

$ echo $word
Hi

$ echo '$word'
$word

What's the output from this command, and why?
$ echo "He said '$word'"

Test for a Numeric Variable

Bash does not seem to provide an easy way of testing if a variable is numeric ('123' as opposed
to 'abc' or 'a2').

Here are two indirect ways of making this test. If you know of a better method let me know.

The first technique uses grep and a regular expression to search for a string of all digits. If
the string is not all digits grep returns a null string, which is interpreted as false.
$ var=12
$ if [ "$(echo $var | grep "^[[:digit:]]*$")" ]; then echo number; fi
number

The second technique relies on 'test' (or '[') throwing an error and returning '2' if -eq is
given a non numeric value. Otherwise '0' or '1' is returned for TRUE or FALSE.
$ var="123"
$ [ $var -eq 0 ] 2> /dev/null
$ if [ $? -eq 0 -o $? -eq 1 ]; then echo number; fi
number
$

$ var="12AB"
$ [ $var -eq 0 ] 2> /dev/null
$ if [ $? -eq 0 -o $? -eq 1 ]; then echo number; fi
$

Variable Variables

Bash can implement variables variables, in a round-about way. PHP programmers will be familiar
with variable variables.

Basically, we want to construct a variable name from another variable:

From:
$var=BSD

We want to create a variable called:
system_BSD

Let's set to 'yes' variable 'system_???', where ??? is the value of $var:
$ var=BSD
$ system_$var=yes
-bash: system_BSD=yes: command not found

This fails because Bash interprets $var as BSD, then treats the resulting 'system_BSD=yes' as a
command. We need to evaluate $BSD, then tell Bash to re-parse the line from scratch using 'eval'.

To set a variable variable use:
$ var=BSD
$ eval "system_$var=yes"
$ echo $system_BSD
yes

We can't echo the value in the same manner:
$ eval "echo $system_$var"
BSD

We want $var to be 'eval'uated, but not $system:
$ eval "echo \$system_$var"
yes
$

Underline a String

Underline a generated string, which may be of any length.

$ cat ./underline
#!/bin/bash

read -p 'Give a string to underline: ' string


len=${#string}
echo $string
while ((len != 0)); do
echo -n "-"
((len = len - 1))
done
echo

$ ./underline
Give a string to underline: This is a test.
This is a test.
---------------

The construct:
${#var}

returns the length in characters of 'var'.

This little script uses Bash built-in integer tests and arithmetic:
((len = len - 1))

Notice anything unusual/unexpected?

'echo -n' is used so as not to follow each dash with a newline.

This example would normally be coded as a Bash function and called whenever the main script needs
to underline a string.

Return Arbitrary Values

Return an arbitrary value from a Bash function.

Bash functions return an exit status between 0 and 255. This is testable via the shell special
variable $?. But how do we return an arbitrary number, or even a string?

One way to do this is to get the function to 'echo' the return value, and use the $(...)
construct to capture the output from the function. $(...) is usually used to capture the output
from commands, but it works equally well for functions.

Here I have converted the underline example from Monday's tip into a function that returns the
underline string.

$ cat ./underline
#!/bin/bash

# Function underline
#
# Generate an underline string
#
# $1: the string to underline
# $2: the optional underline character, default value is '-'
#
# return: the underline string
#
underline ()
{
local -i len # to hold the length of the string in $1
local line # to hold the generated underline string
local char=${2:-"-"} # set the underline to $2 or the default if not supplied

len=${#1}
line=${line}$char
((len = len - 1))
done

echo $line
}

# Main script
#
read -p 'Give a string to underline: ' string
echo $string
dashes=$(underline "$string")
echo $dashes

$ ./underline
Give a string to underline: This is a test...
This is a test...
-----------------

Function underline returns its value with:
echo $line

and this value is captured in 'dashes' by the main script using:
dashes=$(underline "$string")

This is a useful construct:
char=${2:-"-"}

Return many values using arrays.

Here is a simple example using an array to receive many return values from a function. 'count'
returns four strings.

$ cat numbers
#!/bin/bash

# Function count
#
count ()
{
#return one two three four
echo one two three four
}


# Main
#
declare -a numbers

numbers=($(count))

echo ${numbers[0]}
echo ${numbers[1]}
echo ${numbers[2]}
echo ${numbers[3]}

$ ./numbers
one
two
three
four

Stop Echoing for Passwords

Stop input being echoed when a script reads a password.

A script can print a prompt, read text from the user, but the text is echoed. If a password is
being read, it would be nice to prevent this from being echoed.

The secret is to use:
stty -echo

to switch off echoing.
$ cat getpass
#!/bin/bash

read -p "Give your name: " name
stty -echo
read -p "And your password: " pass; echo
stty echo
read -p "And your shoe size: " ss

$ ./getpass
Give your name: Adrian
And your password:
And your shoe size: 9

Access the Last Parameter

Get the value of the last parameter passed to a script/function when the number of parameters is
*not* known.

$2 id the second parameter, but how do we get the n'th parameter without knowing 'n'?

The number of parameters is given by the special parameter $#. We need '$$#'.

None of these attempt work:

$ cat lastp
#!/bin/bash
echo $#
echo $$#
echo ${$#}
echo \${$#}

$ ./lastp one two three
3
2428#
2428
${3}

The secret is to evaluate {S#} first, then evaluate $-the-result-of-the-first-evaluation. Bash's
'eval' function does just this, letting Bash parse the line twice.
$ cat ./lastp
#!/bin/bash

eval echo \$$#

eval last=\$$#
echo $last

$ ./lastp one two three four
four
four

The first parse effected by eval changes \$ into $ and $# into 4, leaving $4. The resulting line
is then parsed normally echoing $4 (or assigning $4 to last).

Change the Shell's Null Globbing

Change the shell's behaviour for glob expansion.

Normally, if globbing (wildcard expansion) expands to nothing, the shell returns the glob instead
of null. This is not always what is required in a shell script.

For example, *.txt expands to two files:
$ for file in *.txt; do echo -n $file ": "; head $file; done
f.txt : Hello
g.txt : there

but *.text doesn't match:
$ for file in *.text; do echo -n $file ": "; head $file; done
*.text : head: *.text: No such file or directory

We'd prefer the loop to be silent rather than give a spurious error message. This can be effected
by setting Bash's 'nullglob' option:
$ shopt -s nullglob
$

(no error message)

At the end of the script, the default behaviour should be restored.
$ shopt -u nullglob
*.text : head: *.text: No such file or directory

$

Enable FTP

This week's tips show how to set up a simple FTP service complete with 'chrooting' and access
classes. Note: read the complete week's tips before exposing your FTP set-up to the Internet.

Enable FTP the GUI way. In System Preferences select Sharing, tab Services, and check FTP Access.
The fire-wall will be automatically be opened.

Enable FTP the command line way.

1) Open the fire-wall ports 20 and 21, and all high-numbered ports connected to from FTP clients
(from source ports 20, 21).
$ sudo ipfw add 03000 allow tcp from any to any 20-21 in
$ sudo ipfw add 03010 allow tcp from any 20,21 to any 1024-65535 in
$ sudo ipfw list
...
03000 allow tcp from any to any 20-21 in
03010 allow tcp from any 20,21 to any 1024-65535 in
...

Rules 3000 onwards are not used by the OS X default firewall set-up. You might like to 'ipfw
list' to check this first if you use a non-standard set-up.

Note: System Preferences will sulk when it sees you have been playing with its fire-wall rules.
If you want SP to play again, do:
$ sudo ipfw flush
Are you sure? [yn] y

Flushed all rules.

(re-)Launch SP and enable the firewall.

To start FTP edit the 'ftp' service in xinet.d. (FTP is set up as an on-demand service launched
by xinet.d, not as a permanent daemon.)
$ sudo pico /etc/xinetd.d/ftp

Set the disable line to 'no'
disable = no

Re-set xinet.d so it re-reads the edited file (bash):
$ sudo kill -HUP $(cat /var/run/xinetd.pid)

or (tcsh, bash)
$ sudo kill -HUP `cat /var/run/xinetd.pid`

Test by connecting with 'ftp user@hostname/IP-address':
$ ftp saruman@carcharoth
Connected to carcharoth.mayo-family.com.
220 carcharoth.mayo-family.com FTP server (tnftpd 20040810) ready.
331 Password required for saruman.
Password:
230-
Welcome to Darwin!
230 User saruman logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>...
ftp> quit
221-
Data traffic for this session was 0 bytes in 0 files.
Total traffic for this session was 480 bytes in 0 transfers.
221 Thank you for using the FTP service on carcharoth.mayo-family.com.

Allow and Deny Explicitly

Disallow FTP access by default.

The standard Apple-supplied ftp configuration allows all users to use FTP, except those
explicitly listed - i.e. allow by default and deny explicitly. A more security-conscious set-up
would be to deny by default and allow explicitly.

Change the /etc/ftpusers file from:
$ cat /etc/ftpusers
# list of users disallowed any ftp access.
# read by ftpd(8).
Administrator
administrator
root
uucp
daemon
unknown
www

to:
$ cat /etc/ftpusers
# list of users allowed ftp access.
# read by ftpd(8).
saruman allow
loraine allow
... list all the accounts that may ftp...

# deny all other users
* deny

This file must be edited as root, or via 'sudo'.

Jail and Umask

Jail users to their home directory.

The standard Apple-supplied ftp configuration allows all users the same access to the file system
they would enjoy by when logging in. That is, they can 'cd /' and view many system files. A more
security-conscious set-up would 'jail' most users such that 'cd /' took them to their home
directory. For theses users, their home directory is the root of their file system and they can
never move to a level above it.

Edit/create the file /etc/ftpchroot:
$ sudo pico /etc/ftpchroot

so that it looks like this:
$ cat /etc/ftpchroot
# all users are chrooted
*

Tomorrow's tip shows how to jail some users, and not others (without explicitly listing all
chrooted users).

Change the default umask.

The default ftp umask in 027, as opposed to the more usual 022. This means that users uploading
files to be served by Apache, for example, create files that Apache cannot read.

FTP provides a 'chmod' command:

ftp> help chmod

chmod change file permissions of remote file

but a more convenient solution may be to change the default umask for all users to be 022.

Edit/create the file /etc/ftpd.conf:
$ sudo pico /etc/ftpd.conf

so that it looks like:
$ cat /etc/ftpd.conf
# normally 027
umask all 022

Test with:
$ ftp loraine@carcharoth
Connected to carcharoth.mayo-family.com.
....
230 User loraine logged in.
ftp> umask
200 Current UMASK is 022
ftp> quit
...
$

Define Classes of Users

To get finer control over FTP access, umask settings, chrooting, etc, FTP allows one to define
classes. Settings can be defined against a class, and each user assigned to the appropriate
class. A user will then inherit the attributes set for the class to which they belong.

Edit the ftp config file to define classes.
$ sudo pico /etc/ftpd.conf

In the configuration below, I have defined two classes, 'free' and 'restricted'. I've 'chrooted'
the free class to '/' so they can view the entire file system (the default setting). Restricted
users can only view their home directory (%d). Note that 'homedir' for restricted users is set to
'/', which is their home directory because of their chroot setting.
$ cat /etc/ftpd.conf

# users of class 'free' (see /etc/ftpusers) chroot to /
# with their ftp home directory set to their login home
chroot free /
homedir free %d

# users of class 'restricted' chroot to their home directory
# with their ftp home directory set to their new root (ie their login home)
chroot restricted %d
homedir restricted /

# change the umask from the default of 027
umask all 022

The last directive sets the umask for 'all', which represents all classes.

Security

FTP is not the most secure of services. Set up carefully it should be fine, but remember that
passwords and all communications are sent unencrypted.

If you run ssh (Secure SHell) then you might like to take advantage of secure copy and secure
FTP. You must already have ssh set up in order to use these services.

scp is like the regular Unix copy, but copies across networks using an ssh login. To copy the
file 'test' in my home directory to the host saruman.wless I can use:
$ scp ~/test saruman.wless:~/test

To use scp precede the path on the remote machine with 'hostname:' or 'ip-address:'.

The equivalent of 'ftp' is 'sftp'. As far as I know it is not possible to configure sftp to the
same degree as ftp. For example, it is not possible to jail users to their home directories.

Note: To use scp ensure that file /etc/sshd_config has the line:
# override default of no subsystems
Subsystem sftp /usr/libexec/sftp-server

Finally, to restrict the range of ports to which a client can call-back, use (for example):
# set port range for passive for all classes
portrange all 40000 40999

The default is 1024-65535. After having made this change you can then restrict the number of open
ports in the firewall configuration (see Monday's tip) to:
03010 allow tcp from any 20,21 to any 40000-40999 in

Strings and Vis

Use 'strings' to remove junk characters.

The strings command searches a file for printable strings and displays them. It can be used on a
text file to remove spurious non-printable characters.
$ strings text-file > text-file.clean

Peek inside commands.

Use strings on commands such as 'ls' to discover embedded text:
$ strings /bin/ls
__dyld_mod_term_funcs
__dyld_make_delayed_module_initializer_calls
The kernel support for the dynamic linker is not present to run this program.
$FreeBSD: src/bin/ls/cmp.c,v 1.12 2002/06/30 05:13:54 obrien Exp $
@(#) Copyright (c) 1989, 1993, 1994
The Regents of the University of California. All rights reserved.
$FreeBSD: src/bin/ls/ls.c,v 1.66 2002/09/21 01:28:36 wollman Exp $
COLUMNS
CLICOLOR
1ABCFGHLPRSTWZabcdfghiklmnopqrstuvwx
...

Use 'vis' to display messy files.

Vis displays control characters as escape sequences allowing you to view files that might
otherwise spew junk to the terminal.

File and Hexdump

Use 'file' to discover file types.
$ file safari-bm.html
safari-bm.html: UTF-8 Unicode English text, with very long lines

$ file safari-bm.txt
safari-bm.txt: OS/2 URL object text (WWW) <http://mayo-family.com/
http://album>

$ file /bin/ls
/bin/ls: Mach-O executable ppc

$ file /
/: sticky directory

$ file ~
/Users/saruman: directory

$ file /dev/disk0
/dev/disk0: block special (14/0)

$ file /dev/rdisk0
/dev/rdisk0: character special (14/0)

and so on...

Use 'hexdump' to display a file in hexadecimal.
$ hexdump /bin/ls | head -n 5
0000000 feed face 0000 0012 0000 0000 0000 0002
0000010 0000 000f 0000 0778 0000 0095 0000 0001
0000020 0000 0038 5f5f 5041 4745 5a45 524f 0000
0000030 0000 0000 0000 0000 0000 1000 0000 0000
0000040 0000 0000 0000 0000 0000 0000 0000 0000

hexdump is useful for revealing control characters in text files, or examiming the contents of a
binary file

Note that Mach-O executables start with the hex sequence 'feed face' :-)

When is it Time to Leave?

Use 'leave' so you never again forget lunch.

Lunch at one:
$ leave 0100
Alarm set for Mon Dec 27 13:00. (pid 10494)

Coffee in 10 minutes:
$ leave +10
Alarm set for Mon Dec 27 12:38. (pid 10497)

Leave will remind you periodically about the forthcoming event, and with increasing frequency as
the event approaches.
$ You have to leave in 5 minutes.
...
$ Just one more minute!
...
Time to leave!
...
Time to leave!
...
Time to leave!
...

When you log out the nagging will finish. If you don't log out, kill leave using its pid as
echoed back when the reminder was set.
$ kill -KILL 10494

To kill all leaves use:
$ kill -KILL $(ps | grep ' [l]eave ' | awk '{print $1}')

(in tcsh use:)
$ kill -KILL `ps | grep ' [l]eave ' | awk '{print $1}'`

Record a Terminal Session

Use 'script' to record a sequence of commands.

If you are doing something you may want to repeat later, or wish others to do, use the script
command to record you terminal session to a file (~/typescript if no filename is given). Option
-a appends to an existing script results file.
$ script
Script started, output file is typescript
$ cd ~
$ ls
Desktop Documents Movies Pictures Sites osxfaq test.clean
Development Library Music Public bin test typescript
$ rm test
$ rm test.clean
(at this point press control-D to exit the script session)
$ exit
Script done, output file is typescript

View the script file:
$ cat ~/typescript
Script started on Mon Dec 27 12:45:12 2004
$ cd ~
$ ls
Desktop Documents Movies Pictures Sites osxfaq test.clean
Development Library Music Public bin test typescript
$ rm test
$ rm test.clean
$ exit

Script done on Mon Dec 27 12:45:34 2004
$

Of course, if all you want is the commands you typed, use history.
$ history -c
$ cd ~
$ ls
Desktop Documents Movies Pictures Sites osxfaq .....

finish the session
$ history > ~/what-i-typed

Who's Logged In

There are several command for monitoring which users are currently logged into you Mac or server.
$ who
saruman console Dec 19 17:29
ferdi ttyp2 Dec 23 21:30 (10.0.2.1)

User 'ferdi' is logged in from another machine, as given by the IP address in brackets. You can
send that user a message direct to their console:
$ write ferdi [ttyp2]
get off

The tty number is optional if they are logged in just once.

Command 'w' gives a little more information.
$ w
21:30 up 6 days, 2:36, 2 users, load averages: 0.45 0.21 0.21
USER TTY FROM LOGIN@ IDLE WHAT
saruman console - Sun17 4days -
ferdi p2 middle-earth.wle 21:30 - -bash

Command 'users' lists only local users:
$ users
saruman

Command 'last' give a history of who was, and who still is, logged in:
$ last | head -n 5
ferdi ttyp2 10.0.2.1 Thu Dec 23 21:30 still logged in
saruman ttyp2 10.0.2.1 Thu Dec 23 21:25 - 21:26 (00:00)
saruman ttyp2 10.0.2.1 Thu Dec 23 21:19 - 21:25 (00:05)
saruman ttyp3 Thu Dec 23 21:01 still logged in
saruman ttyp3 Thu Dec 23 21:01 - 21:01 (00:00)

'uptime' is a good to boast about how long your server has been up and running.
$ uptime
21:16 up 6 days, 2:22, 2 users, load averages: 0.21 0.36 0.28

Directory Service Commands

More useful scripts to manage user accounts on the command line - delete a group, delete a user
from a group, and delete a user.

First, here are some tips on commands to query the user accounts in NetInfo.

Use 'dcsl list' to display groups and users.
$ dscl . list /groups
admin
appserveradm
appserverusr
bin
claire
...

$ dscl . list /users
appserver
claire
cyrus
daemon
...

Use 'dscl search' to check for property-value pairs.

$ dscl . search /groups name claire
Search results for dsAttrTypeStandard:name
Search results for dsAttrTypeNative:name
claire dsAttrTypeNative:name = (claire)

Use 'dscl read' to display properties and values.
$ dscl . read /groups
name: dsRecTypeNative:groups

$ dscl . read /groups/claire
AppleMetaNodeLocation: /NetInfo/DefaultLocalNode
GeneratedUID: 0066A0DA-ED62-11D8-B80E-000393B2D604
GroupMembership:
Password: *
PrimaryGroupID: 507
RecordName: claire

Use 'nifind' to check if an entry (group, user, etc) exists.
$ nifind /users/claire .
/users/claire found in ., id = 90

$ nifind /users/xxxxxxxxx .
$

Delete Group

Use this script to delete a group from NetInfo.


$ cat del-group
#!/bin/bash

# Delete a group.
# Takes the group name and deletes it from NetInfo

declare quiet="no" # -q option not specified
declare group # hold te given group name
declare gid # hold the group id derived from the group name
declare ans # reply from prompt

usage ()
{
echo "Delete a group"
echo "Usage: ${0##*/} [-q] groupname"
echo " -q - quiet: no warnings or prompts for confirmation"
echo " otherwise a warning is issued if the group to"

echo " be deleted is a user's primary group"
if [ "$*" != "" ]; then echo; echo "Error: $*"; fi
exit 1
}

#
echo "Must be run as root"
exit 1
fi

# Check parameters
#
if [ "$1" = "-q" ]; then
quiet="yes"
shift
fi

if [ $# -ne 1 ]; then
usage
fi

group="$1"

# search NetInfo for the given group - it should exist
str="$(nifind /groups/$group .)"
usage "Group $group does not exist"
fi

# Check if this is a primary group for some user and warn if so
# but not in quiet mode
if [ $quiet = "no" ]; then

gid="$(nireport . /groups name gid | grep -w "^$group" | cut -f 2)"

# print a warning if it is any user's primary group
str="$(nireport . /users gid | grep -w $gid)"
echo "WARNING: this is a primary group:"
nireport . /users gid name | grep -w "^$gid"

read -p "Type a to abort: " ans

if [ "$ans" = "a" ]; then
echo "Aborted"
exit
fi
fi
fi

# Delete the group from NetInfo
#
# sanity check
if [ "$group" = "" ]; then exit; fi

dscl . delete /groups/$group

echo "Group $group deleted"
exit 0

Delete User From Group

Use this script to delete a user from a group, or from all groups.


$ cat del-user4group

#!/bin/bash

# Remove a user from a group, or all groups
# Removes an existing user from an existing group in NetInfo, or from
# all groups to which that user belongs (but not their primary group)

declare groups # hold the given group name or the list of groups
declare user # hold the user account name
declare str strgroup stringroup strprimary # working

usage ()
{
echo "Remove a user from a group or all groups"
echo "Usage: ${0##*/} group|all user"
echo " for 'all' the user is removed from all but their primary group"
exit 1
}


#
exit 1
fi

# Check parameters
#
if [ $# -lt 2 ]; then
usage
fi

groups="$1"; user="$2"

# If group is all, expand into the list of groups to which the user belongs
#
if [ $groups = "all" ]; then
groups="$(id -Gnr $user)"
fi

# Loop to remove the user from each group
#
for group in $groups; do


# check if the group exists
strgroup="$(nifind /groups/$group .)"
# check if the user is listed for the group (not listed in own primary)
stringroup="$(nireport . /groups name users | grep -w "^$group[[:space:]].*$user")"
strprimary="$(nireport . /users name gid | grep -w "^$user[[:space:]]*$gid")"

# ensure that the group exists...
if [ -z "$strgroup" ]; then
echo "Group $group does not exist"
echo "Not removing from primary group $group"
# ...and that the user is listed in the group
elif [ -z "$stringroup" ]; then
echo "User $user not listed in $group"
else
# remove user from the group

dscl . delete /groups/$group users $user
echo "User $user removed from group $group"

fi
done

exit 0

Delete User From Group

Use this script to delete a user from a group, or from all groups.


$ cat del-user4group

#!/bin/bash

# Remove a user from a group, or all groups
# Removes an existing user from an existing group in NetInfo, or from
# all groups to which that user belongs (but not their primary group)

declare groups # hold the given group name or the list of groups
declare user # hold the user account name
declare str strgroup stringroup strprimary # working

usage ()
{
echo "Remove a user from a group or all groups"
echo "Usage: ${0##*/} group|all user"
echo " for 'all' the user is removed from all but their primary group"
exit 1
}

#
exit 1
fi

# Check parameters
#
if [ $# -lt 2 ]; then
usage
fi

groups="$1"; user="$2"

# If group is all, expand into the list of groups to which the user belongs
#
if [ $groups = "all" ]; then
groups="$(id -Gnr $user)"
fi

# Loop to remove the user from each group
#
for group in $groups; do


# check if the group exists
strgroup="$(nifind /groups/$group .)"
# check if the user is listed for the group (not listed in own primary)
stringroup="$(nireport . /groups name users | grep -w "^$group[[:space:]].*$user")"
strprimary="$(nireport . /users name gid | grep -w "^$user[[:space:]]*$gid")"

# ensure that the group exists...
if [ -z "$strgroup" ]; then
echo "Group $group does not exist"
echo "Not removing from primary group $group"
# ...and that the user is listed in the group
elif [ -z "$stringroup" ]; then
echo "User $user not listed in $group"
else
# remove user from the group

dscl . delete /groups/$group users $user
echo "User $user removed from group $group"
fi
done

exit 0

Delete User

Use this script to delete a user.

The user and their primary group are deleted. The user is removed from any other groups. The
users home folder is archived then deleted.

It uses the previous two scripts.


$cat del-user

#!/bin/bash

# Delete a user.
# Takes the account name (short name) and:
# removes the user from all groups
# removes the user's primary group (of the same name)
# removes the user's account in NetInfo
# archives and deletes the user's home directory in /Users/shortname

declare user # to hold user's account name
declare str # working

usage ()
{
echo "Delete a user account, group, and group membership"
echo "Usage: ${0##*/} username"
exit 1
}

#
exit 1
fi

# Check parameters
#
if [ $# -ne 1 ]; then
usage
fi

user="$1"

# search NetInfo for the given user - it should exist
str="$(nifind /users/$user .)"
usage "User $user does not exist"
fi

# Delete the user from NetInfo
#
# delete the user from all groups
del-user4group all $user

# delete the user's primary group
del-group -q $user

# delete the user from NetInfo
dscl . delete /users/$user

echo "User $user deleted"

# Archive the user's home directory
#
# check that the user has a home directory
if [ -e /Users/$user ]; then
# archive it
cd /Users
tar -czf ${user}-archive.tgz $user
cd -

# delete it CHECKING THAT AN ARCHIVE WAS CRESATED
if [ -e /Users/${user}-archive.tgz ]; then
rm -rf /Users/${user}/
fi
fi

echo "User's home directory archived and deleted"

exit 0

Change Network Locations

Switch network locations from the command line.

The command:
$ scselect

lists all network locations, as viewed in system Preferences::Network.

The command:
$ scselect location-name

changes the current network location to the selected one.

New Since

List all files created since the last restart:

$ find ~ -newer /mach.sym

This finds all files in your home that are newer than /mach.sym. /mach.sym is one of the files
created when OS X boots.

Create a file at the start of the day to keep track of all files you have created or edited that
day.
$ touch ~/marker

then later...
$ find ~ -newer ~/marker

More examples.

Find all new or changed directories, but up to a maximum nesting depth of 2.
$ find ~ -newer /mach.sym -type d -maxdepth 2

'-type' can also be 'f' for file, 'l' for symbolic link, or other special file types - see 'man
find'.

Find all new files larger than 1MB. '-size' is specified in blocks which are a 512 bytes or a
half K. '-ls' lists the details of each file.
$ find ~ -newer /mach.sym -size +2000 -ls

'-size' can match against an exact number of blocks by omitting the '+' sign, or against files
which are smaller than the specified size by specifying a '-' sign. E.g. -size -2000 for less
than 1MB.

Complex Conditions

(See also weeks 1, 13, 31, and 49)

Use complex selection criteria with 'and' and 'or' constructs.

Given the following files, and noting the modification dates and sizes:
$ ls -l
total 49M
... 15M Jan 15 12:12 ferdi-coll3.psd
... 30M Jan 23 11:59 ferdi-cool.psd
... 1.7M Jan 15 12:12 ferdi-gala.psd
... 1.6M Jan 23 15:59 ferdi-polish.psd

Find those files modified less than 1 day ago:
$ find . -mtime -1
.
./ferdi-cool.psd
./ferdi-polish.psd

Find those files modified more than 1 day ago:
$ find . -mtime +1
./ferdi-coll3.psd
./ferdi-gala.psd

Find those files less than 5M in size:
$ find . -size -10000
.
./ferdi-gala.psd
./ferdi-polish.psd

Find those files greater than 5M in size:
$ find . -size +10000
./ferdi-coll3.psd
./ferdi-cool.psd

Find those files modified less than one day ago AND bigger than 5M:
$ find . -mtime -1 -a -size +10000
./ferdi-cool.psd

Find those files modified more than one day ago AND smaller than 5M:
$ find . -mtime +1 -a -size -10000
./ferdi-gala.psd

Find those files
modified less than one day ago AND bigger than 5M
OR
modified more than one day ago AND smaller than 5M:
$ find . $ -mtime -1 -a -size +10000 $ -o $ -mtime +1 -a -size -10000 $
./ferdi-cool.psd
./ferdi-gala.psd

Note the use of brackets (escaped from the shell) to ensure that the AND and OR expressions are
evaluated in the correct order - that is the two ANDs are evaluated, then the results ORed.

Because 'find' assumes AND by default, the expression in this case can be shortened to:
$ find . $ -mtime -1 -size +10000 $ -o $ -mtime +1 -size -10000 $
./ferdi-cool.psd
./ferdi-gala.psd

Also, because 'find' evaluates AND before OR, the brackets can be omitted.
$ find . -mtime -1 -size +10000 -o -mtime +1 -size -10000
./ferdi-cool.psd
./ferdi-gala.psd

If the expression were of the form:
(a -o b) -a (c -o d)
the brackets would be necessary.

Curtailing Find

Stop 'find' from traversing other file systems, recursing, and following symbolic links.

Use option '-x' to stop 'find' from looking inside mounted file systems:
$ ls /Volumes/
Inside Mac Media OSX-saruman guest

$ find -x /Volumes
/Volumes
/Volumes/guest
/Volumes/Inside Mac Media
/Volumes/OSX-saruman

Only /Volumes is searched, the three mounted file systems within it are not. Compare running this
command without option '-x'.

Find with no recursion.
$ find * -prune -size +10 -ls

This finds all files in the current directory greater than 5K, and lists their details with
'-ls'. '-prune' stops find from recursing into each directory.

Note that:
$ find . -prune -size +10 -ls

will not work because 'find' will not recurse into the specified directory '.'.

This will work:
$ find . -maxdepth 1 -size +10 -ls

Make find follow symbolic links.

Find does not by default follow symbolic links - 'home' in this example.
$ ls -l
total 49700
... Jan 15 12:12 ferdi-coll3.psd
... Jan 23 11:59 ferdi-cool.psd
... Jan 15 12:12 ferdi-gala.psd
... Jan 23 15:59 ferdi-polish.psd
... Jan 23 16:46 home -> /Users/saruman $ find . -name "*.psd"
./ferdi-coll3.psd
./ferdi-cool.psd
./ferdi-gala.psd
./ferdi-polish.psd

Use option -L tell 'find' to follow symbolic links:
$ find -L . -name "*.psd"
./ferdi-coll3.psd
./ferdi-cool.psd
./ferdi-gala.psd
./ferdi-polish.psd
./home/Pictures/complete/ferdi-coll3.psd
./home/Pictures/complete/ferdi-cool.psd
./home/Pictures/complete/ferdi-gala.psd
./home/Pictures/complete/ferdi-polish.psd
...

Option '-H' tells 'find' to follow symbolic link given on its command line, rather than symbolic
links found during the search.

This will not follow 'home':
$ find -H . -name "*.psd"

This will:
$ find -H * -name "*.psd"

Find and Du

Find the top ten largest files/directories.
$ du -sk ~/* | sort -nr | head -n 10
186164 /Users/saruman/Sites
132596 /Users/saruman/Development
46928 /Users/saruman/Documents
33216 /Users/saruman/osxfaq
14372 /Users/saruman/Movies
12304 /Users/saruman/.Trash
10252 /Users/saruman/Desktop
580 /Users/saruman/.gimp-2.0

'du' (Disk Usage) displays the size of all files/directories (~/*). '-s' summarises presenting
just the total size for directories, 'sort -nr' sorts into numeric reverse (descending) order,
and 'head -n 10' displays just the first 10 lines.

Improve this with 'find' to include only directories.
$ find ~ -type d -maxdepth 1 -print0 | xargs -0 du -sk | sort -nr | head -n 10
14372 /Users/saruman/Movies
12304 /Users/saruman/.Trash
10252 /Users/saruman/Desktop

(options '-print0' and '-0' cater for filenames with spaces - see the man pages or previous
weeks.)

Summarise to a greater depth.
$ find ~ -type d -maxdepth 2 -print0 | xargs -0 /usr/bin/du -sk | sort -nr | head -n 20
1772768 /Users/saruman/Pictures/iPhoto Library
378168 /Users/saruman/Library/Caches
308072 /Users/saruman/Pictures/complete
142848 /Users/saruman/Sites/albums
80224 /Users/saruman/Development/developer-downloads
77160 /Users/saruman/Pictures/people
69472 /Users/saruman/Library/Mail
54920 /Users/saruman/Pictures/web-site
50224 /Users/saruman/Development/test
26304 /Users/saruman/Pictures/Photoshop-effects
20076 /Users/saruman/Library/Application Support
19100 /Users/saruman/Documents/1dot1
19012 /Users/saruman/osxfaq/sf03

Searching the Path

Use '-regex' to filer against the whole path.

Previous week's covered '-name' and '-iname' which match against names of found files. '-regex'
matches the entire path.

Find all directories called test, wherever they are in the directory hierarchy.
$ find ~ -regex ".*/test/.*"

will find:
/Users/saruman/Development/test/*

and
/Users/saruman/Pictures/Gimp/test/*

Also, option '-E' says to use extended regular expressions, and '-iregex' says to match in a
case-insensitive manner.

Easily remove the full path name.
$ find ~ -name "*.psd" -execdir echo {} \;
ferdi-coll3.psd
ferdi-cool.psd
ferdi-gala.psd
ferdi-polish.psd

vs the regular:
$ find ~ -name "*.psd"
/Users/saruman/Development/test/find/ferdi-coll3.psd
/Users/saruman/Development/test/find/ferdi-cool.psd
/Users/saruman/Development/test/find/ferdi-gala.psd
/Users/saruman/Development/test/find/ferdi-polish.psd
...

On the Command Line

Coping with spaces in filenames is problematic for both the command line beginner and the more
experienced scripter. This week presents some solutions.

Escape spaces in filenames. To enter a filename that includes spaces on the command line use:
my\ file
"my file"

or
'my file'

Alternatively, type the first part of the filename and hit tab to auto-complete it.

Escaping can also be used to address a filename that includes odd characters such as " or \.
$ ls a*
a"b a\b

$ cat a'"'b
double quote in name
$ cat a\"b
double quote in name
$ cat a'\'b
back-slash in name
$ cat a\\b
back-slash in name

Find, Xargs, and Spaces

'Find' is a command often used to root out particular files in a directory hierarchy and process
each file, often by passing the filenames to 'xargs'.

However, it will fail for filenames with spaces:
$ ls -1 file*
file one
file two

$ find . -name "file*" | xargs ls -l

ls: ./file: No such file or directory
ls: one: No such file or directory
ls: ./file: No such file or directory
ls: two: No such file or directory

Fix this using the '-print0' option of 'find', and the '-0' option of 'xargs'.

$ find . -name "file*" -print0 | xargs -0 ls -l
-rw-r--r-- 1 saruman saruman 0 Feb 7 14:34 ./file one
-rw-r--r-- 1 saruman saruman 0 Feb 7 14:34 ./file two

'-print0' tells 'find' to null-separate filenames when it produces the list of found files,
instead of space-separating them. '-0' tells 'xargs' to expect null-separated filenames.

Going Loopy

Choose the correct looping method to preserve spaces in filenames.

To process these two files:
$ ls -1 file*
file one
file two

This will not work:

$ cat loop
#!/bin/bash
ls=$(ls file*)

for i in $ls; do
echo "File: $i"
done

$ ./loop
File: file
File: one
File: file
File: two

This will:

$ cat loop
#!/bin/bash

for i in file*; do
echo "File: $i"
done

$ ./loop
File: file one
File: file two

When filename expansion is done directly in the loop each filename is fed to the loop
individually, hence there is no confusion between spaces in filenames and spaces as separators in
a list. When an intermediate variable is used the loop is fed a space-separated list.

Bash's IFS

Wednesday's example cannot be used if the list of filenames is generated by a method more complex
than wildcard expansion: for example using 'find.

This will not work:

$ cat ./find-and-loop
#!/bin/bash
list=$(find . -atime -1)

for i in $list; do
echo "File: $i"
done

$ ./find-and-loop
File: .
File: ./file
File: one
File: ./file
File: two

Neither will using:
for i in $(find . -atime -1); do

Get clever with Bash's Internal File Separator (IFS). Normally the separator is any of space,
tab, or newline. 'Find' (and 'ls' ) produce new-line separated lists, so changing the IFS to just
new-line does the trick.

$ cat find-and-loop
#!/bin/bash

list=$(find . -atime -1 )
IFS="
"

for i in $list; do
echo "File: $i"
done

$ ./find-and-loop
File: .
File: ./file one
File: ./file two

Note: There is a single newline character between the quotes in IFS="
"

Parameters With Spaces

Process parameters with spaces.

This will not work:

$ cat params
#!/bin/bash

echo 'For $*'
for i in $*; do echo "Parameter: $i"; done

echo
echo 'For "$*"'
for i in "$*"; do echo "Parameter: $i"; done

$ ./params "param one" "param two"
For $*
Parameter: param
Parameter: one
Parameter: param
Parameter: two

For "$*"
Parameter: param one param two

Neither solution gives us:

Parameter: param one
Parameter: param two

Instead of $* to represent all parameters, we use Bash's special $@ symbol.

$ cat params
#!/bin/bash

echo 'For $@'
for i in $@; do echo "Parameter: $i"; done

echo
echo 'For "$@"'
for i in "$@"; do echo "Parameter: $i"; done

$ ./params "param one" "param two"
For $@
Parameter: param
Parameter: one
Parameter: param
Parameter: two

For "$@"
Parameter: param one
Parameter: param two

The second variant "$@" give us what we want.

Re-size and Move with Escape Sequences

Re-size and move a terminal window from the command line, or from a script, using the following
escape sequences.

Re-size to a specific number of rows and columns (eg 50 rows by 100 columns):
$ echo -n "^[[8;50;100t"

(Note: the sequence displayed as ^[ represents the escape character, obtained by typing control-v
then the 'esc' key.)

Re-size to the maximum screen width by giving columns as 0:
$ echo -n "^[[8;50;0t"

Re-size to the maximum screen height by giving rows as 0:
$ echo -n "^[[8;0;100t"

Move to a specific position (eg 10 pixels from the left, and 100 pixels from the top):
$ echo -n "^[[3;10;100;t"

When re-sizing to the maximum screen width and/or height, it is advisable to position the screen
at the top left:
$ echo -n "^[[3;0;0;t^[[8;0;0t"

Define Aliases and Functions

Define aliases and functions to move and resize. For example, to make a big screen use an alias:

(Bash)
$ alias big='echo -n "^[[3;0;0;t^[[8;0;0t"'

(tcsh)
% alias big 'echo -n "^[[3;0;0;t^[[8;0;0t"'

To size a screen use a Bash function or tcsh alias:

(Bash)
$ sz ()
> {
> echo -n "^[[8;$1;$2;t"
> }

(tcsh)
alias sz 'echo -n "^[[8;\!:1;\!:2;t"'

Then use:
$ sz 50 100 $ sz 25 80

These escape sequences work in both Apple's Terminal.app and X11 xterm.

Focus and Dock with Escape Sequences

Move a window to the background or the foreground from the command line, or within a script,
using the following escape sequences. (See Monday's tip too.)
echo -n "^[[6t;"; sleep 5; echo -n "^[[5t;"

The above command hides the terminal window, waits 5 seconds, then brings it back into focus. The
'sleep 5' part could be a script that takes a while to run. When the script completes the
terminal window will re-appear to let you know.

(Note: the sequence displayed as ^[ represents the escape character, obtained by typing control-v
then the 'esc' key.)

Move a window into the Dock.
$ echo -n "^[[2t;"; sleep 5; echo -n "^[[5t;"

The above command docks the window, then after 5 seconds brings it back into focus.

These sequences can be places in bash/tcsh aliases.

Terminal.app Nifty Features

Apple's Terminal.app has a few nifty features that you may or may not have encountered.

Split Screen. Click the little 'split' icon situated top left above the vertical scroll bar, then
drag the horizontal divider to adjust the top/bottom split ratio. The top pane can be used as a
scroll-back to view and copy previous commands without disturbing the main window.

Drag and Drop. Drag any file or folder (or multiple files and folders) onto the terminal window
and the full path name will be written to the prompt line. For example, to list the contents of a
folder, type 'ls ' then drag the folder from the Finder onto the Terminal window and hit return.

Set a background. Most visual aspects of the terminal can be changed from menu item
Terminal::Window Settings, including setting a background picture. Alternatively Alt-drag a
picture onto the Terminal window to instantly set a background image.

Click to position cursor. This option has to be enabled in menu item Terminal::Windows
Settings::Emulation. Check 'Option click to position cursor. You can now option click anywhere on
the current command line to position the cursor at that point. This also works in text editors
such as Pico and Vim.

Alternative Terminals

The standard terminal for OS X is Apple's Terminal.app, but there are many other alternatives.

If you have installed X11, you will have xterm. Xterm is started automatically when X11 starts
up, and you can type:
$ xterm&

to get a new Xterm window. (Remember to add the & otherwise the first terminal will be locked
running Xterm as a foreground task.)

Xterm is hugely configurable as witnessed by 'man xterm'

Start new Xterm terminals with Command-N. Select X11 menu Applications::Customise to add new
commands to the Applications menu. For example, add:

Name: xterm
Command: /usr/X11R6/bin/xterm -sb -sl 5000 -rightbar -fg white -bg black -geometry 100x50+40+20
Shortcut: n

Now Command-N will start a new Xterm window sized 100 columns by 50 rows, positioned (40,20)
pixels from the top left of the screen (-geometry 100x50+40+20).

'-fg white -bg black' sets white text on a black backgound.

'-sb -sl 5000 -rightbar' sets a scroll bar on the right-hand side, and a scroll-back buffer of
5000 lines.

Xeyes is fun, watch the eyes watching your cursor. Set up a customised command as follows:

Name: eyes
Command: /usr/X11R6/bin/xman -bothshown -notopbox
Shortcut: whatever-you-want or blank

Xman is useful, a nice front end to the Unix manual. Set up a customised command as follows:

Name: xman
Command: /usr/X11R6/bin/xman -bothshown -notopbox
Shortcut: whatever-you-want or blank

iTerm is an alternative terminal.

iTerm

iTerm is a full featured terminal emulation program written for OS X using Cocoa. It supports
language encodings, VT100/ANSI/XTERM emulation and many convenient GUI features.

Zterm is useful if you need to talk via a serial or USB port (to configure routers for example).

Zterm

ZTerm is a terminal emulation program for the Macintosh. In its day, many people used it to
connect to Bulletin Board Systems and download files. Now we have the internet. Its still a
useful utility for those systems that only offer dialup connections and for connecting to devices
through a serial port, like many routers. For newer Macs that don't have a normal serial port,
ZTerm can talk to ports on USB to serial adapters, through the appropriate driver software
supplied with the adapter.

The .Plist Files

This week's tips are an accompaniment to the Daily Mac OS X tips comprising weeks 28 February -
Trouble-Shooting (Applications), 7 March - Trouble-Shooting II (System), 14 March -
Trouble-Shooting III (Hardware), and 21 March - Trouble Shooting IV (Advanced).

The Unix trouble-shooting tips give command-line alternatives to methods discussed in the OS X
Daily tips.

The 'defaults' command provides easy access to the OS X preference files. It interprets the XML
encoding allowing one to easily read and write property values, or create new property values.
$ man defaults

View and change the NSGlobalDomain. This contains default preferences that apply to an
application if it doesn't override them it its own domain.

$ defaults read -globalDomain
{
AppleAntiAliasingThreshold = 8;
AppleAquaColorVariant = 1;
AppleCollationOrder = "en_GB";
AppleHighlightColor = "0.776500 0.776500 0.776500";
AppleICUDateFormatStrings = {
1 = "yyyy'.'MM'.'dd";
2 = "d' 'MMM', 'yyyy";
3 = "d' 'MMMM', 'yyyy";
4 = "EEEE', 'd' 'MMMM', 'yyyy";
...

$ defaults read -globalDomain AppleLocale
en_GB

$ defaults write -globalDomain AppleLocale en_US
$ defaults read -globalDomain AppleLocale
en_US

View a .plist file with XML decoding. This translates the XML encoding into something more
readable.

$ defaults read com.apple.iChat
{
ABDirectoryResultColumnTitle = "Instant Messaging";
AutoLogin = 1;
AutosaveChats = 1;
BuddyInfoSelectedTab = 0;
"BuddyList.SecondarySortOrder" = 1;
"BuddyList.SortOrder" = 3;
"BuddyList.Visible" = 1;
CardsBlockingPresentityPictures = (
"60765A82-A567-11D7-A842-000393B2D604:ABPerson",
"56C0BB88-C7D2-11D6-950E-000393B2D604:ABPerson"
);
...

NOTE: DO NOT WRITE THE DECODED VERSION BACK TO THE ORIGINAL .PLIST FILE DIRECTLY, use:
$ defaults write domain 'the-plist-from-defaults-read'

Use vim to view:
$ defaults read com.apple.iChat | vim -
Vim: Reading from stdin...

Use less to view:
$ defaults read com.apple.iChat | less
...

View a single property:
$ defaults read com.apple.iChat AutoLogin
1

Note: 0 = false/no, 1 = true/yes.

Change a property:
$ defaults write com.apple.iChat AutoLogin 0
$ defaults read com.apple.iChat AutoLogin
0

Note: 'defaults' automatically looks in ~/Library/Preferences and adds the .plist extension.

WARNING: don't modify the .plist of a running application.

Use plutil to verify the syntax of .plist files.

$ plutil ~/Library/Preferences/com.apple.iChat.plist
/Users/saruman/Library/Preferences/com.apple.iChat.plist: OK

$ plutil -s ~/Library/Preferences/*.plist

Option '-s' reports only error conditions, and is silent on success.

Force-Quit

Use 'kill' or 'killall' to force-quit errant applications.

The 'kill' command requires a process ID, not an application name. Discover it with:

$ ps xc | grep -iw ical
6577 ?? S 0:41.67 iCal

(Put the application name in quotes if it contains spaces or characters special to the shell.)

Quit it with:
$ kill -QUIT 6577

Force-quit it with:
$ kill -KILL 6577

Do it on one line by creating a bash function:
$ function killer () { kill -KILL $(ps xc | grep -wi "$*" | awk '{print $1}'); }

$ killer ical

Kill applications belonging to other users. Add option 'a' to the 'ps' command so it lists
processes owned by other users too. You must either be root or use 'sudo' to issue the 'kill'
command.

Kill a process by name. Use the 'killall' command.
$ killall ical
No matching processes belonging to you were found

$ killall iCal

(Killall is case-sensitive.)

Killall can match process names by regular expressions too - check out the man page.

To shut down the computer from the command line:
$ sudo shutdown -r now

Launching and File Association

The Finder, and the 'open' command, both use Launch Services to associate documents with
applications. The Finder also uses this information to associate icons with documents.

The 'lsregister' command can be launched from the command-line, and gives scope for manual
maintenance of the Launch Services database.

Problems involving incorrect icons or documents failing to open (with the proper application) can
be solved using 'lsregister'
$ /System/Library/Frameworks/
ApplicationServices.framework/Versions/A/
Frameworks/LaunchServices.framework/Versions/
Current/Support/lsregister

(all on one line)

lsregister: [OPTIONS] [-domain { system | local | user | network }]... [path]...
Search the paths for application bundles and add each found item to the Launch Services
database. For domain specifications, ask CF for the list of application locations in
the given domain(s).

-kill Reset the global Launch Services database before doing anything else
-lint Print information about plist errors while registering bundles
-convert Register apps found in older LS database files
-load Load the LaunchServices service plugin if it's not already loaded.
-lazy n Sleep for n seconds before registering apps if the local cache
is aleady populated.
-r Recursively register directory contents, do not recurse into
packages or invisible directories.
-R Recursively register directory contents, including the contents
of packages and invisible directories.
-f force-update registration info even if mod date is unchanged
-v Display progress information.
-dump Display full database contents after registration.
-h Display this help.

Issuing the command:
...the/long/path/lsregister -kill -r -domain system -domain local -domain user

will repair the Launch Services database.

I've not investigated this command much, but it does have potential for querying, maintaining,
and resetting the LS database.

Keep It Tidy

Unix is pretty much self-maintaining. A little periodic maintenance is required, and is scheduled
to run in the early hours daily, weekly, and monthly.

Scheduling is controlled by the system 'cron' scheduler, which reads its scheduling information
from /etc/crontab:

$ cat /etc/crontab
# /etc/crontab
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#
#

If your Mac is switched off during these hours you may want to re-schedule the periodic
maintenance. See:
$ man 5 crontab

for an explanation of the format of crontab.

Alternatively, run the maintenance manually with:
$ sudo periodic daily
$ sudo periodic weekly
$ sudo periodic monthly

Other candidates for removal include log files in /var/log that are not periodically zipped and
rotated. For example, find all log files over 10 Meg:
$ find /var/log -size +20000
/var/log/named/default.log
/var/log/secure.log

Get more details with:
$ find /var/log -size +20000 -ls

Install 'anacron'. This can be obtained from the many Unix ports, including fink.

It is like cron but catches up on scheduled events that were missed when your Mac was asleep or
not running.
$ fink describe anacron
Information about 1951 packages read in 4 seconds.

anacron-2.3-4: Periodic command scheduler
Anacron executes commands at intervals specified in days. Unlike cron, it does not assume that
the system is running continuously. It's ideal for machines such as laptops...

Remove Browser caches. I run almost all browsers to test my web development. As a consequence
browser caches greatly increase the time required for incremental backups. I have written a
simple script that removes the caches for Camio, FireFox, iCab, Internet Explorer, Mozilla,
Omniweb 5, Opera 7, and Safari.

#!/bin/bash

if [ "$1" = "-usage" ]; then
echo "Cleans the caches for the browsers:"
echo "Camio, Firebird, iCab, IE, Mozilla, OmniWeb5, Opera, Safari"
echo "Usage: ${0##*/}"
exit
fi

# just in case!
cd ~/Library || exit 1

echo "Removing caches..."

# Camio
echo -n " Camio - "
{ cd ~/Library/Application\ Support/Chimera/Profiles/default/ \
&& rm -r $(ls -d *.slt | head -1)/Cache \
&& echo removed;
} 2> /dev/null \
|| echo "none"

# Firefox
echo -n " Firefox - "
{ cd ~/Library/Application\ Support/Firefox/profiles/*default/ \
&& rm -r Cache \
&& echo removed;
} 2> /dev/null \
|| echo "none"

# iCab
echo -n " iCab - "
{ rm -r ~/Library/Preferences/iCab\ Preferences/iCab\ Cache/ \
&& echo removed;
} 2> /dev/null \
|| echo "none"

# Interner Explorer
echo -n " Internet Explorer - "
{ rm ~/Library/Caches/MS\ Internet\ Cache/IE\ Cache.waf \
&& echo removed;
} 2> /dev/null \
|| echo "none"

# Mozilla
echo -n " Mozilla - "
{ cd ~/Library/Mozilla/Profiles/default/ \
&&| rm -r $(ls -d *.slt | head -1)/Cache \
&& echo removed;
} 2> /dev/null \
|| echo "none"

# OmniWeb
echo -n " Omniweb - "
{ rm -r ~/Library/Caches/com.omnigroup.OmniWeb5/Cache.bundle \
&& echo removed;
} 2> /dev/null \
|| echo "none"

# Opera
echo -n " Opera - "
{ rm -r ~/Library/Caches/Opera\ Cache \
&& echo removed;
} 2> /dev/null \
|| echo "none"

# Safari
echo -n " Safari - "
{ rm -r ~/Library/Caches/Safari \
&& echo removed;
} 2> /dev/null \
|| echo "none"

echo "Done."

The locations of the caches work for me, but I'm not sure with the Mozilla set of browsers how
much the random folder names vary.

Disc Drives

/Applications/Utilities/Disk Utility allows one the verify and repair permissions on the system
volume, and verify and repair the structure of other volumes. These functions can be performed
from the command line using 'diskutil'

To refer to a specific disc/volume, specify its mount point, disc identifier, or device node, as
below.

The mount point is usually /Volumes/disc-name, but in my case I have mounted directly under /.

$ diskutil verifyDisk /Music/
Started verify/repair on disk disk0s9 Music-saruman
The volume Music-saruman appears to be OK.
Verify/repair finished on disk disk0s9 Music-saruman

To discover the disc identifier and device node use disktool.

$ disktool -l
...
***Disk Appeared ('disk0s3',Mountpoint = '/', fsType = 'hfs', volName = 'OSX-saruman')
***Disk Appeared ('disk0s5',Mountpoint = '/Users', fsType = 'hfs', volName = 'Users-saruman')
***Disk Appeared ('disk0s7',Mountpoint = '/Games', fsType = 'hfs', volName = 'Games-saruman')
***Disk Appeared ('disk0s9',Mountpoint = '/Music', fsType = 'hfs', volName = 'Music-saruman')

$ diskutil verifyDisk disk0s9
...

$ diskutil verifyDisk /dev/disk0s9
...

To repair use:
$ diskutil repairDisk /Music Started verify/repair on disk disk0s9 Music-saruman
The volume Music-saruman appears to be OK.

To verify and repair permissions (of the boot volume), use:

diskutil verifyPermissions /OSX-saruman
diskutil repairPermission /OSX-saruman

NOTE: the name of your system volume will differ from mine.

To repair the boot volume you must boot into single user mode. It is not possible to repair a
volume that is mounted and cannot be unmounted (ie is always in use).

Restart holding down the Command-S key combination and wait until the text has finished
scrolling. Next mount the system volume as writable (it is currently mounted as read-only at this
stage of the boot process).

Type:
$ mount -uw /

Then check/repair the file system with:
$ fsck -y

or, for journalled filesystems (the default in Pather):
$ fsck -fy

If fsck reports problems with something like:

***** FILE SYSTEM WAS MODIFIED *****

re-run fsck until it reports that no modifications were necessary.

Hit control-d to continue the boot process, or type:
$ reboot

Repair non-boot volumes. Sometimes it is not possible to repair other volumes because they cannot
be unmounted (for example if you have your home directory on another partition). In this case the
offending volume has to be repaired in single user mode just like the system volume.

For example, to repair HFS disc 0 partition 9:
fsck_hfs -fy /dev/rdisk0s9

Refer to 'disktool' above to get the disc and partition numbers, or use command 'df' in single
user mode.

Basic File and Directory Commands

The next four weeks will form a Unix command reference by listing lots of commands under specific
areas of usage. There are so many commands available that it is impossible to know them all, or
even remember all those one knew and used just last month.

See the rest of the tips for lots of example usage of these commands.

See the Learning Centre for tutorials on these commands, and explanations of Unix permissions.

And of course, don't forget the Unix manual 'man'.

ls ... list the contents of the current directory
Option -l (ell) to list with details
Option -a to list hidden dot files, -A to miss out '.' and '..'
Option -L to follow symbolic links
Option -F to identify different types of files, or
Option -G to identify with colour
(See tips week 24)

cp ... copy files and directories
Option -R for recursive copy to copy whole directory subtrees

mv ... move or rename files and directories
Rename a file, move it to another directory, or do both at the same time

rm ... delete files and directories
Option -R or -r to recursively move whole directory subtrees
Option -f to override permissions where possible
Option -i for interactive confirmation of each deletion

cd ... change current directory
'cd..' to move up a directory
'cd' to move to your home directory
'cd -' to move back to the last directory

pwd ... print working directory
Display the current directory

pushd ... change directory and stack it Each time pushd is issued, the current directory is
remembered on the directory stack Use popd to retrace your steps

popd ... move back to the last pushed directory 'pushd' and 'popd' work on a stack system where
the last directory pushed is the first to be popped

dirs ... print the directory stack
This is the stack is written by 'pushd' and read by 'popd'

(See tips weeks 30 and 32)

mkdir ... create a new directory
Option -p to make many (nested) directories in one go

rmdir ... delete a directory
The directory must be empty
Use 'rm -r' to delete non-empty directories

chmod ... change file and directory permissions
Option -R to change permissions in a whole directory subtree
Option -L to follow symbolic links
Option -h to change the mode of a link instead of the file it points to

chown ... change owners of files and directories
Options -R, -L, and -h as for 'chmod'
Change both user and group owners by specifying user-owner:group-owner

chgrp ... change group owner of files and directories
Options -R, -L, and -h as for for 'chmod'

chflags ... change the file flags
Unix files and directories have many flags such as immutable, append-only, archived, that can be
set using this command
Options -R and -L as for for 'chmod'

Commands to View Files

cat ... display a file
Cat actually concatenates many files and writes the result to standard out (the screen)
Redirect to a file to actually join files
Option -b to number non-blank lines
Option -s to collapse multiple blank lines
Option -v to display non-printing characters so they are visible

more ... display a file in a paged manner
Scroll up and down with the cursor keys
Page up and down with the 'u' and 'd' keys
Search for text with /text-to-search-for
And lots, lots *more*, see 'man more'

less ... display a file in a paged manner
Like more, but much more than more
See 'man less'

lesskey ... specify key binding for less
Specify alternative key binding for controlling less

head ... display the first 10 lines of a file
Option -n specifies a different number of lines to display

tail ... display the last 10 lines of a file
Option -n specifies a different number of lines to display

(See tips week 4)

zcat, zmore ... cat and more zipped files

bzcat, bzmore, bzless ... cat, more, less on bzipped files

wc ... line, word, and character count files
Options -l, -w, and -c to control what is counted

file ... determine the type of a file

pr ... display files with pagination
Display files page by page with headers and footers

vis ... display files with non-printable characters
Does a better job than 'cat -v'

strings ... search a binary file for printable strings
Option -a to search the whole file, not just binary sections (applicable mainly to executables)

hexdump ... display binary files
The output shows the hexadecimal value of each byte
Option -n len to display only the first 'len' bytes

xxd ... display binary files
Like 'hexdump' but displays both the hexadecimal and character representation of each byte (or
'.' for non-printable characters)

(See tips week 95)

emacs ... powerful text editor

vi/vim ... the best text editor
(See tips weeks 50, 53, and 69)

pico ... a simple text editor

Commands to Compare Text Files

split ... split a file into many files
Split a large file into many small files each 1000 lines long
Option -l to specify a different number of lines for the small files

sort ... sort the lines of a file into alphabetic order
Each line is considered a number of columns, so you can sort on any column, or key
Option -t to specify the key separator (space, command, tab, etc)
Option -k to specify which key/column to sort on
Option -b to ignore leading blanks
Option -f to fold case (make the sort case insensitive)
...and more....

uniq ... filter out repeated lines of a file
The file must be sorted first
For repeated lines only the first copy is written out
Option '-f fields' to ignore the first 'fields' fields of each line in the comparison
Option '-s chars' to ignore the first 'chars' characters of each line in the comparison
Option -c to precede each output line with the number of repetitions in the original input

join ... perform a database join on files
Specify a field (as for 'sort') by which the files are compared
A line is displayed for each pair of lines in the two files where the compared fields match

paste ... merge corresponding lines of two file

comm ... display lines common to two files
The two files must be sorted first
Three columns are produces for lines that are in file one only, file 2 only, and both files
Options -1, -2, -3 to suppress printing of their respective columns

diff ... compare two files
Compare two files and print the differences in terms of small differences in lines, new lines,
deleted lines
Use to compare two versions of the same file
Option -b to ignore differences in white space
Option -B to ignore differences in blank lines
Option -i to ignore changes in case

diff3 ... compares three files
Comparison between three files f1, orig, and f2 compares the differences between f1 and f2
relative to orig
Option -A to incorporate all the changes between orig and f2 into f1 - this acts like a merge
operation to merge the changes made to f2 (relative to orig) into f1

sdiff ... compare and merge two files
Compare the differences between two files and merge them to a third file
Option -b to ignore differences in white space
Option -B to ignore differences in blank lines
Option -i to ignore changes in case

zdiff, bzdiff ... diff zipped, bzipped files

Commands to Search for Content

grep ... search a file for text

Search a file for specific text, or a pattern specified by a regular expression (see 'man
re_format')
Option -i to specify case-insensitivity
Option -w to match whole words
Option -r to search all files in a directory subtree
Option -v to display lines that do NOT match
Options -n, -l, -L, -h, -H to control the what is printed in terms of matched text, filenames,
and line numbers
Option -E to use extended regular expressions instead of basic
Option -F to use fixed strings (this is much quicker than pattern matching)

egrep ... search a file for text
The same as 'grep -E', i.e it used extended regular expressions

fgrep ... search a file for text
The same as 'grep -F'

zgrep, bzgrep ... grep a zipped, bzipped file

bzegrep ... egrep a bzipped file

bzfgrep ... fgrep a bzipped file

perl ... a text manipulation language
From the man pages:
Perl is a language optimized for scanning arbitrary text files, extracting information from those
text files, and printing reports based on that information. It's also a good language for many
system management tasks. The language is intended to be practical (easy to use, efficient,
complete) rather than beautiful (tiny, elegant, minimal).

Commands to Change File Content

awk ... pattern scanning and manipulation language
Awk reads a text file line by line and applies a series of search, replace, and edit operations
to lines that match given criteria.
Awk can read a script file for its processing commands
the language combines a 'C'-like programming language with regular expression search and replace.

gawk, nawk ... variations of awk
Not available in the standard Mac OS X install

sed ... stream editor
Sed reads a text file line by line and applies a series of search, replace, and edit operations
to lines that match given criteria
Sed can read a script file for its processing commands
the language uses regular expression search and replace functionality, but lacks the 'C'-like
programming language.

tr ... translate on character into another
Reads a text file and translates all occurrences of a given character into another
By default reads and writes standard in and out, so redirection is necessary to work on files
Don't write the output back to the input file, otherwise it will end up empty

col ... strip chars
Col can be used to clean-up files
Option -b to remove multiple backspaces as output by the man pages
Option -x to convert tabs characters to multiple spaces

(See tips week 10)

cut ... filters columns from files
Cut can process a file that has columns of text (for example tab-separated tables) and extract
just the specified columns
Option -f to list the columns to be output, like '-f 3,5' for 3 and 5, '-f 3-5' for 3 to 5, or
'-f 1,3-5' for 1 and 3 to 5.
Option -d to specify the field separator (by default tab), like '-d " "' for space, or '-d ","'
for comma

expand ... expand tabs to spaces

unexpand ... compress spaces to tabs

fmt ... format a text file
Read a file and write a version who's lines do not exceed a maximum specified length, or
optionally do not come in under a minimum specified length
Option -c to centre text
Option -p to allow indented paragraphs
Option -s to collapse multiple white space inside lines

fold ... fold long lines
Simple version of fmt which breaks long lines to 80 characters or less
Option -w to specify a maximum width other than 80

Commands to Search for Files

Last week and the next three weeks will form a Unix command reference by listing lots of commands
under specific areas of usage. There are so many commands available that it is impossible to know
them all, or even remember all those one knew and used just last month.




To search for files in the files system, use:

find ... find files matching a number of criteria
Option -E to specify extended regular expressions
Option -L to follow symbolic links for found files
Option -X when find is used with xargs
Option -s to search hierarchies in lexicographical order

Check out the many primaries, which specify which files are matched based on many, many criteria
such as name, time/date, owners, size, type, permissions, etc
(See tips weeks 1, 13, 31, 49, 97, 98)

locate ... search a database of files
This relies on the locate database being up to date
(See tips week 1)

locate.updatedb ... update the locate database in
(See tips week 1)

The best way to update the locate database database to avoid revealing private filenames is by
the command:
$ sudo periodic weekly
(See tips weeks 64, 88)

type ... find an executable (bash shell only)
Option -a to find all versions of the executable
(See tips week 77 - Tuesday)

which ... find an executable (csh and tcsh)
(See tips weeks 20 and 47 - Monday)

whereis ... find an executable (search system path)

Also:

ls -r * ... recursive directory listing

echo * ... list all files in the current directory

Commands About and For Users

id ... display identity of the current user
Option -p to display in a more readable format
Option -G to display group numbers, useful for shell scripts

groups ... display groups to which a user belongs

(See tips week 17)

logname ... display the current user's login name

finger ... display information on a user
Create ~/.plan and ~/.project files to advertise more information to finger

passwd ... change the password of the current user

chpass, chfn, chsh, ... add or change user information
Check the man pages for compatibility with OS X

umask ... change the umask for this and child processes
(See week 47 - Wednesday)

login ... log into the machine
Option -p to inherit the current environment

rlogin ... log into a remote machine
This is depreciated in favour of the more secure 'ssh' - see under 'Networking Commands' later in
the series.

su ... switch to another user's identity

sudo ... execute a command as root
Option -s to run a shell as the new user
Option -u to run as a specified user rather than root
Option -l to list what you are allowed to sudo
Options -k, -K to timeout the user's password to sudo

users ... display who is currently logged in
who ... as 'users' but with more detail
w ... as 'who' but with even more detail

whoami, who am i ... useful for amnesiacs

last ... list all logins with date, time, and status

(See tips week 17)

write ... send a message to another user
Use 'who' to list other users and their consoles
(See tips week 95 - Friday)

wall ... send a message to all logged in users

talk ... chat to another user
Use 'who' to list other users and their consoles

mesg ... allow/disallow messages from other users

Useful Utility Commands

man ... the Unix manual
Option -k to search the manual
Option -f to report a one-line summary of a command

apropos ... the equivalent of man -k
This and 'man -k' search for the given command as a string and will display all manual entries
containing the string

whatis ... the equivalent of man -f
This and 'man -f' search for the given command as a whole word

makewhatis ... (re)build the whatis database

(See weekly tips weeks 2 and 20)

catman ... create formatted version of man pages
This speeds up the 'man' command.

info ... the GNU on-line manual
Try 'info info' for more details
Try 'info command-name' for information on the (GNU) command

at ... execute a command at a specific time

atq ... list the pending 'at' jobs

atrm ... remove an 'at' job form the queue

atrun ... the daemon that launches 'at' jobs
This should be added to the system crontab and run periodically (say every 5 minutes) to schedule
'at' jobs.

calendar ... scheduling and reminder service

(See tips week 41)

cron ... system task scheduling daemon

crontab ... user task scheduling daemon

(See tips week 33)

leave ... remind you when it's time to leave
Do we need reminding of this? :-)
(See tips wee w95 - Wednesday)

banner ... write banner text to a printer

bc ... an arbitrary precision calculator

dc ... an arbitrary precision reverse polish calculator

cal ... display a monthly, yearly calendar
Option -j displays day numbers

units ... convert quantities from one units system to another

Commands to Control the Terminal

clear ... clear the terminal window

resize ... resize the terminal window
Option -s to simply specify rows and columns
$ resize -s 50 10

stty ... change the setting for the terminal
Option -a to display the current settings
Option -g to display in a form that can be used by -f
Option -f to set the display reading commands from a file

reset ... reset the terminal display
Useful after accidentally printing a binary file that messes up the terminal's settings

tty ... display the user's current terminal

toe ... display all terminfo entries by name
Reads the compiled terminfo database in /usr/share/terminfo/...

tic ... terminfo compiler
Reads terminfo descriptions and compiles them into the terminfo database in
/usr/share/terminfo/...

infocmp ... compare and print terminfo descriptions

captoinfo ... convert termcap description to terminfo database

infotocap ... convert terminfo description to termcap description

tput ... query the terminfo database

Commands to Control the Shell

alias ... alias a command line to a keyword
(Bash and tcsh)

function ... assign a function to a keyword
(Bash only)

(See tips weeks 12, 20, 71, 77, 99 - Tuesday)

tee ... split a pipe into to streams
Can be used to direct the output of a command to a file and to terminal at the same time

set ... set a shell variable
unset ... remove a shell variable
setenv ... set an shell environment variable
unsetenv ... remove a shell environment variable
(tcsh only)
(See tips week 54 - Monday)

expect ... script a dialogue with interactive programs

script ... make a transcript of a terminal session

bindkey ... to change shell key functionality

batch ... execute commands in non-busy periods
The command is delayed until system load levels are low.
The load level is controlled by atrun, see Wednesday's tip.

exec ... executes command in place of shell

nohup ... execute a command immune to hangups
The command will not be stopped bu 'kill -HUP' or when the current shell exits
(See tips week 78)

lockfile ... prevent file from being accessed by more than one process

rehash ... refresh shell's executable list
(tcsh)
(See week 21 - Friday)

hashstat ... give hash hit statistics
(tcsh)

exit ... quit non-login shell

logout ... quit a login shell

Commands for Info on Machine and Discs

The last two weeks and the next two weeks will form a Unix command reference by listing lots of
commands under specific areas of usage. There are so many commands available that it is
impossible to know them all, or even remember all those one knew and used just last month.




For information on the machine (host), including hardware, operating system, and kernel use:

arch ... display the machine architecture

machine ... display the processor type

uname ... display o/s name, version, and processor type
Option -a to print full (all) information

hostinfo ... display information on the host kernel

The above commands are often used by the 'make' and './configure' system to help build the
correct binaries for the host platform.

hostname ... set and display the host's network name
Option -s to display hostname without domain name

uptime ... display for how long system has been running

date ... display and set the system date

AppleSystemProfiler ... display comprehensive details
This command is no longer included in Panther. It is incorporated into the System Profiler
command in
Application:Utilities (or from the About This Mac box).

(See tips week 11)

ioreg ... show the i/o registration for all hardware

For information on disc usage use:

du ... display disc usage statistics
Option -a to report for all files in the hierarchy
Option -c to display a grand total
Option -h to display file sizes in K-, M-, and G-bytes
Option -s * to display information for each file

df ... display disc free information for all discs
Option -h to display file sizes in K-, M-, and G-bytes
Option -i to display inode statistics too

fsck ... verify and repair filesystems
Option -f to force check, useful for journaled devices
Option -y to complete check without manual intervention
(See tips weeks 55, 100)

mount ... display information on mounted devices
As well as mounting devices, the mount command on its own shows information on each mounted
device, including whether it is local and whether it is journaled.

disktool -l ... list discs
Disktool is part of Apple's command line interface to the file system.

Commands for System and Kernel Stats

These commands display information on system, memory, and file usage

sc_usage ... display live system call stats
For example, 'sudo sc_usage TextEdit'

fs_usage ... display live file system usage
For example, 'sudo fs_usage TextEdit' then use text edit and open/save files.

lsof ... list open files
For example 'lsof' or 'lsof /dev/console'

fstat ... display file status
For example 'fstat' or 'fstat ~' or 'fsstat ~/letter.doc'

vm_stat ... display virtual memory statistics
Watch out for page-outs increasing by 100's per second: you need to buy more memory or run fewer
processes.

latency ... display context switches &interrupts
This must be run as root

dmesg ... display system message buffer
(See tips week 61 - Friday)

ac ... display user connect time for accounting

These commands operate on the kernel.

ktrace ... perform kernel tracing
For example, 'ktrace ls' executes 'ls' and trace it's kernel activity. This produces the file
'ktrace.out'.

kdump ... display a 'ktrace.out' file human readable
For example 'kdump' if the trace file is in the current directory.
Option -f to specify the trace file

zprint ... display information on kernel zones

sysctl ... display and set kernel state variables
Option -a to display all state variables and their current settings

kextload ... kernel extension load

kextunload ... kernel extension unload

kextstat ... display kernel extension statistics

Commands for Running Processes

kill ... stop/restart processes by PID
Sends a signal to a process. Give the signal number or name, and the PID (Process ID) of the
process to kill.

man 2 sigaction ... information on signals
Use this command to learn about the possible signals that 'kill' can send to running processes.

killall ... stop/restart processes by name
Like kill but a case sensitive process name is required instead of a PID.

(See tips week 45)

ps ... list running processes
Option -x to also list processes not attached to the terminal
Option -a to list processes belonging to other users
Option -ww to make a wide listing, otherwise the output is truncated
Option -c to list only the process (command) name, not the full command line

top ... display information on running processes

(See tips week 19)

nice ... execute a process with a given priority

renice ... alter the pripority of an existing process

exec ... execute a command in place of shell

(See tips week 47)

nohup ... execute a command immune to hangups
The command will not be stopped by 'kill -HUP' or when the current shell exits
(See tips week 78)

leaks ... check process for memory leaks
For example 'leaks TextEdit'

heap ... list allocated memory in process's heap
For example 'heap TextEdit'

printenv ... display environment of the current shell

Commands for Network Fetching

curl ... grab a URL from a remote server
Supports the following protocols:
HTTP, HTTPS, FTP, GOPHER, DICT, TELNET, LDAP, FILE
For example: 'curl http://www.apple.com'

wget ... grab a URL from a remote server
Like curl, but not included with OS X. Available for OS X via ports such as Fink.

fetchmail ... get mail from SMTP, POP, IMAP etc

tin... read Usenet newsgroups

trn ... read Usenet newsgroups

ftp ... file transfer protocol
Command line ftp client to get and fetch files from remote servers
(See tips week 63)

ftpd ... ftp daemon
This runs an ftp service
(See tips week 94)

telnet ... connect using the telnet protocol
Connect to remote machines that run telnet, or connect to specific ports to test services such as
FTP, SMTP. and HTTP by hand.

ssh ... start a secure shell on a remote machine
The session is encrypted, including the transfer of password information used to log into the
remote machine.

sshd ... ssh daemon
This runs an ssh service

scp ... secure copy files to/from a remote machine
Files are copied over an encrypted ssh session

sftp ... secure version of FTP
The ftp session is run over an encrypted ssh session

openssl ... manage TLS/SSL certificates

certtool ... create new key pairs for certificates

rcp ... remote file copy
Copy files to from a remote host running an rlogin server. Superseded by 'scp'.

rlogin ... login to a remote machine
The remote machine must be running an rlogin server. superseded by 'ssh'

rsh ... execute a shell on a remote machine
Superseded by 'ssh'

rexec ... remote execution
Superseded by 'ssh'

pine ... command line email client

Commands for Network Information

hostname ... set or display the hostname
Option -s to display hostname without domain name part.
hostname is normally set automatically from /etc/hostconfig

ifconfig ... configure network interfaces
Without any parameters displays the configuration for all network interfaces
For example, 'ifconfig en0 down' then 'ifconfig en0 up' resets en0.

ipconfig ...set the IP mode of an interface
Usage is:
ipconfig set <interface> BOOTP|MANUAL|DCHP|NONE|INFORM

route ... maintain network routing tables
The routine information for a destination can be queried with:
route -v get apple.com
route -v get localhost

arp ... maintain address resolution protocol tables
The tables map IP addresses to Ethernet addresses
Option -a to display all table entries

service ... start and stop xinetd services
option --list to list the services
The possible commands are 'start' and 'stop'.

netstat ... display the network status
Shows the state of all network connections (sockets)
Option -I to specify a particular interface like en0

tcpdump ... dump tcp activity
Option -v, -vv, -vvv to increase verbosity of reports
For example, to listen to network traffic with apple.com use 'sudo tcpdump -vvv host apple.com'

tcpflow ... dump and analyse tcp activity
Like tcpdump but not included with OS X. Available from ports such as Fink.

nslookup ... interactive domain name query (DNS)

dig ... look up DNS information
Works both as a one-off command or interactively.

host ... look up DNS information
Option -t to specify the type of DNS record to look up.
For example 'host -t ns apple.com' to fetch name server records for apple.com. 't can also be
'mx', 'a', 'soa', etc.
Option -a to request all DNS information for the host

named ... DNS daemon
This runs a DNS service

ntpd ... network time protocol daemon
This runs an ntp server

ntpq ... query ntp time server

ntptimeset ... set network time

Miscellaneous Commands

The last three weeks and the this week will form a Unix command reference by listing lots of
commands under specific areas of usage. There are so many commands available that it is
impossible to know them all, or even remember all those one knew and used just last month.


See the Learning Centre for tutorials on these commands.


To set and display environment and shell variables. The commands are marked as (tcsh) and (bash)
if they are built-in for the respective shells, and (Unix) if they are standard Unix commands.

set ... set a shell variable (tcsh)
set ... set shell options (bash)
unset ... unset a shell variable (tcsh and bash)
setenv ... set an environment variable (tcsh)
unsetenv ... unset an environment variable (tcsh)
env ... set and display the environment (Unix)
printenv ... display environment variables (tsch and Unix)
(See tips week 54)

These commands are useful in shell scripts.

expr ... evaluate an expression
The expression can be arithmetic, relational, or logical
Used in scripting as èxpr ...` or $(expr ...)
$ a=$(expr 1 + 1 )
$ echo $a
(See tips week 54 Friday
http://www.osxfaq.com/tips/unix-tricks/week54/friday.ws
)

getopt ... get and parse command line options
This is the standard command line parsing utility used by many commands to process options and
command line arguments.

mktemp ... generate unique temporary file names
Option -d to make a directory
Option -t to include a prefix

printf ... formatted print to display
See 'man 3 printf' for more details of the formatting string.

sleep ... pause processing of a script
the script is paused for the given number of seconds to let other processes execute

[ and test ... tests an expression returning 0 (TRUE) or 1 (FALSE)
Use in conditional statements such as 'if' and 'while'

xargs ...
Option -0 the expect null-separated filenames, Used with 'find -print0' to cope with filenames
with spaces
Option -n to limit the number of arguments on each invocation. Used if the generated command line
is too long (has too many arguments).
(See tips weeks 49, 98 Thursday)

lockfile ... prevent file from being accessed by more than one process

These commands compress files and make archives.

zip unzip ... compress/uncompress files and folders
gzip gunzip ... GNU version of zip
bzip2 bunzip2 ... uses a better compression algorithm
Option -1 to -9 to adjust the level of compression,
-1 is fastest but with the least compression
(See tips weeks 5 Friday, 37, 75 Tuesday)

tar ... archive files into a single file
Option -c to create a new archive
Option -x to extract from an existing archive
Option -r/-A to append files to an existing archive
Option -z to (un)compress with gzip

Miscellaneous commands.

visudo ... safe edit of /etc/sudoers
'sudoers' should not normally be edited by a standard test editor. visudo locks the file and does
sanity checks.

vipw ... safe edit of /etc/passwd
'passwd' should not normally be edited by a standard editor. vipw locks the file and does sanity
checks.
This command is redundant as OS X uses shadow passwords and therefore does not use /etc/passwd
except in single user mode.

time ... time the execution of a command

limit/unlimit ... change system resource limits


OSX Special - Commands for Disc and File

These commands are OS X specific and are not found in other Unix flavours.

Disc manipulation and maintenance.

disktool ... refresh, eject, mount or unmount

diskutil ... repair, journal discs

drutil ... interact with CD and DVD burners

hdiutil ... manipulate disc images

cd9660.util ... mount ISO 9660 volumes

udf.util ... mount DVD volumes
ufs.util ... mount UFS (Unix File System) volumes
hfs.util ... mount HFS (Standard Apple) volumes

pdisk ... partition table editor
List all partitions of mounted discs with:
sudo pdisk
Top level command (? for help): L

vsdbutil ... read/write enable permissions on HFS+ volumes

ditto ... copy files/sync folders with resource forks
Option -rsrc to copy preserving resource forks

cpMac ... copy with resource forks
Copies resource forks like ditto. This is part of the Developers Tool and not included in a
standard install.

GetFileInfo ... get attributes of HFS+ files

SetFile ... sets attributes of HFS+ files
Sets HFS+ attributes such as Alias, Locked, Invisible

OSX Special - System Commands

These are command line equivalents for System Preferences panes.

softwareupdate ... command line version of S/W update
This is useful to software update via a remote connection to a server.

update_prebinding ...
redo_prebinding ...
When new executables and libraries are added, these commands can be used to update the binding
information so commands launch faster. Normally done as part of an install or software update.

system_profiler ... generate detailed system information
This is a command line equivalent to 'About This Mac, More Info...' in the Apple menu. It
generates textual information to standard output.

scselect ... change network location
This is equivalent to changing the Network Location in System Preferences pane Network

pmset ... set power management parameters (screen, disc etc)

bless ...bless a system folder, optionally make it the boot disc
Can be used to change the startup disc, and to 'register' cloned systems.

nvram ... manipulation of Open Firmware NVRAM variables

ioreg ... display IO reg hierarchy

service ... interface to xinetd services - shell script

These commands query and manipulate information in NetInfo and Directory Services.
(See Tips weeks 91 and 96 for scripts which use NetInfo and Directory Service commands.)

DirectoryService ... DS daemon, part of Open Directory

ni* ... NetInfo command set to query and change NetInfo
(See example usage in Friday's tip)
nicl ... NetInfo Command Line Utility
nidump ... Dump NetInfo information in unix FF format
niload ... Load NetInfo information from unix FF format
niutil ... read and write domain in plain text
nifind ... find a NetInfo directory
nigrep ... performs a regular expression search on NetInfo
nireport ... prints tables from NetInfo

dscl ... directory services command line utility
dscl is preferred over the ni* set of commands

scutil ... interface to configd to manage Network Locations etc

OSX Special - Utility Commands

open ... open a file as if double-clicked

screencapture ... screen and window capture

These commands are for maintaining and checking Mac OS X preference files (those that end in
.plist in preferences folders).

defaults ... read and write .plist files
plutil ... .plist utilities

Miscellaneous commands.

say ... text to speach converter

sips ... scriptable image processing system

appleping ... pings and appletalk network

certtool ... manage TLS/SSL certificates and keychain access
(See 'man openssl')

opendiff ... opens two files and compare

lsbom ... interprets contents of binary bom files

See 'man 5 bom'

otool ... examine object files (=Unix ldd)

Example Usage

These examples are taken from my own notes and have not been tested for a while.

Screen Capture Command

When you use the interactive version (command-shift-4), the space bar will toggle between the
regular marquee selection mode and a new mode to capture just a single window, the Dock or the
menu bar. The cursor changes to a camera icon when you are in this new mode and as you mouse over
different areas of the screen the screen capture target is highlighted for you.

Screen captures can be triggered from the command line (and therefore AppleScript, perl, etc.)
using the new screencapture command found in /usr/sbin/

usage: screencapture [-icmwsWx] [file] [cursor]
-i capture screen interactively, by selection or window
control key - causes screen shot to go to clipboard
space key - toggle between mouse selection and window selection modes
escape key - cancels interactive screen shot
-c force screen capture to go to the clipboard
-m only capture the main monitor, undefined if -i is set
-w only allow window selection mode
-s only allow mouse selection mode
-W start interaction in window selection mode
-x do not play sounds file where to save the screen capture

hdiutil Command

Here is how to create a 50MB drive image, backup data to it, and then burn it onto a CD (directly
in the commandline).

hdiutil create -fs HFS+ -volname Backup -size 50m ~/Desktop/backup.dmg
hdiutil mount ~/Desktop/backup.dmg
ditto -rsrc -V ~/Desktop/datatobackup /Volumes/Backup >> /Volumes/Backup/backup.log
hdiutil unmount /Volumes/Backup
hdiutil burn ~/Desktop/backup.dmg -noverifyburn -noeject

NetInfo Command Set

- Backup a NetInfo database
nidump -r / -t localhost/local > backup.nidump

- Restore a NetInfo database from backup
periodic daily backs-up the NI database to /var/backups/local.nidump

From single user with / mounted rw, move old /var/db/netinfo/local.nidb
Start essential services
/usr/libexec/kextd
/usr/sbin/configd
/sbin/SystemStarter

Create blank NetInfo database and start NetInfo
/usr/libexec/create_nidb
/usr/sbin/netinfod -s local

Load the backup into NetInfo:
/usr/bin/niload -d -r / . < /var/backups/local.nidump

- Re-create NetInfo via OS X first-time setup
rm /var/db/.AppleSetupDone
reboot

Other Commands

bless
Set OS X as a startup
Set OS 9 as a startup
sudo bless -folder9 '/Volumes/Mac OS 9/System Folder' -setOF

defaults
defaults read/write/delete <file> <parameters>
defaults write com.apple.terminal TerminalOpaqueness '0.85';

fsck
In single user mode, to repair HFS disc 1 partition 5
fsck_hfs -y /dev/rdisk1s5

man
man 7 ascii ... ASCII table
man 7 hier ... file system hierarchy
man cmd | col -b ... print manual pages

mount
mount_afp [-o option1[,option2 ...]] afp://[username:password]@rhost[:port]/volume node

mount_cd9660 /dev/disk2s1 /Volumes/cd
umount /Volumes/cd

mount -t hfs -w /dev/disk0s10 /Volumes/Grima

nvram
nvram -p ... to print variables
sudo nvram boot-args="-v" ... set verbose mode on boot

tcpflow
sudo tcpflow -i en1 -c port 80

whois
whois -h whois.arin.net ip-address

Background Information

Bash can be started as a login shell or as a non-login shell. One of the most significant
differences between the two modes is the initialisation scripts sourced when Bash starts up.

A login shell sources the login scripts:

/etc/profile
~/.bash_profile

(If ~/.bash_profile cannot be read, ~/.bash_login is sourced, and if this can't be read,
~/.profile is sourced.)

On exit a login shell sources ~/.bash_logout.

A non-login shell sources the following scripts:

/etc/bashrc
~/.bashrc

NOTE: according to the bash manual, /etc/bashrc is not sourced, but it is.

When Apple's Terminal.app starts a new terminal session it starts a Bash login shell. When X11's
xterm starts a new terminal session it starts a Bash non-login shell.

When you type 'bash' at the command line a non-login shell is started.

Non-interactive shells. When you run a shell script, a new shell is launched to execute the
script. The new shell is a non-interactive (non-login) shell. It does not source any scripts on
startup.

A non-login shell is exited by 'exit', and a login shell by 'logout'. Typing 'exit' in a login
shell will run 'logout'.

Make X11 Xterm Launch Login Shells

From Monday's tip it can be seen that Apple's Terminal.app launches a login shell while X11 xterm
launches a non-login shell. To help rationalise this behaviour and get a consistent environment
we can change X11 xterm to launch a login shell.

Make X11 launch login shells. Edit /etc/X11/xinit/xinitrc and change the line that invokes xterm:
xterm &

to:
xterm -ls &

Alternatively, to affect only your own account, create a file ~/.xinitrc which is a copy of
/etc/X11/xinit/xinitrc, and make the same change.

Now X11 will start a login shell.

Even better. To ensure all xterm windows are login shells too, instead of the above create a file
called ~/.Xdefaults
$ cat ~/.Xdefaults
# Default settings for X Applications
#

# xterm
XTerm*.LoginShell: True

Now xterm will start a login shell whether it is invoked by X11 on startup, or by typing 'xterm
&' on the command line, or by the X11 Applications menu.

Customise the look of an xterm terminal by adding more lines to ~/.Xdefaults. For example:
XTerm*background: black
XTerm*foreground: white
XTerm*backdrop: white
XTerm*rightScrollBar: True
XTerm*ScrollBar: True
xterm*saveLines: 5000
XTerm*VT100*geometry: 100x55+10+10

(The reason X11 xterm launches non-login shells is that in most Unix systems X11 controls the
root window and the correct environment has already been set up. In OS X Aqua controls the root
window. Hence it is better for xterm to startup login shells so that they source the login
scripts.)

Initialisation Scripts

The four different initialisation scripts (see Monday's tip) must be set up correctly to ensure
shells have the correct environment.

First off, /etc/profile and /etc/bashrc are system-wide. Use these scripts to set up the
environment that is common to all users. ~/.bash_profile and ~/.bashrc are specific to each user
and are used to add personal settings to the environment.

Second, /etc/profile and ~/.bash_profile are sourced by login shells and therefore applied once
for each terminal session. Typing 'bash' at the command line runs a non-login shell which sources
/etc/bashrc and ~/.bashrc.

A good strategy is to do the following:

Use /etc/profile and ~/.bash_profile to set environment variables. Environment variables are
inherited by non-login shells and don't need to be setup again.

Use /etc/bashrc and ~/.bashrc to set shell variables and other shell settings. Shell variables
are not inherited by non-login shells and therefore need to be set each time.

Finally, /etc/profile should source /etc/bashrc:
source /etc/bashrc

and ~/.bash_profile should source ~/.bashrc:
source ~/.bashrc

to avoid having to set shell variables in both the login and non-login scripts. Note that these
must be 'sourced' and not simply called like:
/etc/bashrc

Prevent Execution of Initialisation Scripts

To prevent bash from running /etc/profile and ~/.bash_profile when it is invoked as a login shell
use:
bash --noprofile

To prevent a non-login shell from running /etc/bashrc and ~/.bashrc invoke on the command line
as:
bash --norc

To create a login shell from the command line use:
bash --login

To have a 'noprofile' shell option in X11, use the X11 menu bar Applications::Customise... to add
a new item. Call the item something like xterm-np and make the command:
/usr/X11R6/bin/xterm -e /bin/bash --noprofile

Note that we tell xterm to run a command with the -e option. If no option is given it runs bash
in a default manner.

Fink. Beware! If you source the Fink initialisation script, for example:
[ -r /sw/bin/init.sh ]&& source /sw/bin/init.sh

the script checks to see if '/sw/bin' and '/sw/sbin' are already in the path. If not it adds them
to the start of the path. Fink includes its own version of bash which does not read /etc/profile
but reads /sw/etc/profile. This can confuse matters if you try to start a login shell by typing
'bash --login'. To avoid any problems add '/sw/bin' and '/sw/sbin' to the end of your path before
you call '/sw/bin/init.sh', or else make an alias to the correct bash, or else invoke it directly
with '/bin/bash'

Script vs 'source' vs '.'

Executing a script in the usual manner from the command line:
$ name-of-script

executes the script in a new bash shell. If you want a script executed by the current shell you
must 'source' it:
$ source name-of-script

or
$ . name-of-script

Executing an initialisation script like /etc/bashrc instead of sourcing it will not work. When
the new shell ends all its environment and shell variables are destroyed.

If you need to keep track on which initialisation scripts have been run in any one shell
environment, use the following technique.

To /etc/profile add:
declare -x STARTUP="/etc/profile, "

To /etc/bashrc add:
declare -x STARTUP="${STARTUP}/etc/bashrc, "

and similarly to ~/.bash_profile add:
STARTUP="${STARTUP}~/.bash_profile, "

and to ~/.bashrc add:
STARTUP="${STARTUP}~/.bashrc, "

Any other sourced files such as those from Fink should also have similar statements added.

Zcat in now Gzcat

In versions of Mac OS X prior to 10.4 (Tiger), the zcat command was, like other z-commands, part
of the GNU zip tool set. It could be used to directly display gzip compressed files.

But historically, zcat has existed to directly display files compressed by the Unix compress
command (the extension used for this sort of file is .Z). The command was often overwritten by
the GNU version.

In Tiger, the original zcat is preserved, and the GNU version is named gzcat. Therefore, always
use gzcat unless the target file really was created by compress.

$ zcat /var/log/system.log.0.gz
zcat: /var/log/system.log.0.gz.Z: No such file or directory
$ zcat /var/log/system.log.0
zcat: /var/log/system.log.0.Z: No such file or directory
$ gzcat /var/log/system.log.0.gz
...
Jan 1 18:11:36 Sauron kernel[0]: AFP_VFS afpfs_mount: ...
Jan 1 18:11:49 Sauron kernel[0]: AFP_VFS afpfs_unmount: ...
...

Note that in pre Tiger systems gzcat was installed as zcat and gzcat. Use gzcat in scripts for
portability.

Here's a reminder of the many z-variant commands available:

zcat zdiff zfgrep zgrep zmore zprint
zcmp zegrep zforce zless znew

Note that znew can be used to recompress .Z files to .gz files. Also included the very useful
zless, lacking in earlier versions of Mac OS X.

Remember too that bzip2 compressed files have a similar set of utilities.

bzcat bzdiff bzfgrep bzip2 bzless
bzcmp bzegrep bzgrep bzip2recover bzmore

Pico is Nano

The pico editor was a very popular and easy to learn Unix text editor. It was part of the PINE
email toolset, and as such had some critical limitations such as a lack of search and replace. It
has been replaced by the GNU nano editor. Nano is compatible with pico, but offers extra
features.

The official home site for nano, www.nano-editor.org, is a useful resource for nano users. Most
important for users of pre-10.4 releases of Mac OS X, which didnt include nano, the site provides
free downloads of the program. Here are instructions for installing it:

Download the source by clicking the link at the bottom of the screen labeled Get Nano, and then
clicking on the file nano-[versionnumber].tar.gz. The source builds right out of the box in
Panther. Type the following lines into the Terminal window (the version number may differ from
the one shown here).

$ tar xzf nano-1.2.5.tar.gz
$ cd nano-1.2.5/
$ ./configure
# ...much output here....
$ make
$ sudo make install
Password:
# ...output here...

The executable will be installed into /usr/local/bin/nano, and the man pages into
/usr/local/man/. If /usr/local/bin is not in your search path, you will either need to add it, or
else invoke nano using its full pathname.

Syntax Colour in Nano

Nano is capable of syntax highlighting. For example, we can tell it to display HTML tags such as
<head> in blue, and escaped characters such as & in red. Enter the following into the nanorc
file, called .nanorc in your home directory.

#HTML Syntax Highlighting
syntax "HTML" "\.html$"

color blue start="<" end=">"
color red "&[^; ]*;"

The second line says that the HTML syntax is applied to all files who's name ends with .html. The
third says that everything between < and > (possibly across multiple lines) is to be coloured
blue, and the fourth says that everything between & and ; (when they are not separated by white
space or semicolon) is to be colored red. (The text between [ and ] includes a space character
and a tab characters.)

You may specify any of the colors: white, black, red, blue, green, yellow, magenta, cyan; and
prefix any with the word bright. The string to match-and-color is specified using an extended
regular expression.

Try loading an HTML file into a new instance of nano, such as ~/Sites/index.html, and you should
see the syntax highlighting described above in action.

Download the nano source files (see yesterday's tip) and you'll find a file called nanorc.sample.
This file gives examples of what can be placed in the configuration file, and how to write syntax
highlighting instructions.

More New Commands

Tiger includes the following commands that were not in earlier versions of Mac OS X.

1) The Korn shell in /bin/ksh, useful if you ever want to run a Korn shell script (or if you
prefer to use Korn as your interactive shell!)

2) zless, part of the GNU zip toolkit, displays gzip compressed files directly using the less
command. It's actually a very simple wrapper script:
$ cat /usr/bin/zless
#!/bin/sh
PATH="/usr/bin:$PATH"; export PATH
LESSOPEN="|gzip -cdfq %s"; export LESSOPEN
exec less "$@"
$

3) textutil, which is a command line interface to the Cocoa text system. Some examples of using
textutil are:

To convert the HTML file index.html to rich text format, written to index.rtf, type:
$ textutil -convert rtf index.html

To convert the same file to an MS Word document, written to index.doc, type:
$ textutil -convert doc index.html

The -cat option is handy to concatenate all the input files into one big output file.

4) mdfind is a command line Spotlight. It searches through meta data for attribute values, or for
specific attributes with given values.

Check the man pages for textutil and mdfind for full details.

Here's a handy tip to convert postscript output to PDF using Preview.app. In this example we pipe
the raw postfix man page for bash into Preview.app.
$ man -t bash | open -f -a /Applications/Preview.app

Tidy with Tidy

Tidy is a utility to syntax check and tidy HTML source code. Type:
$ tidy -h

to see a help screen.
See sourceforge. Tidy can be installed in earlier versions of Mac OS X from sourceforge and
presumably other ports like Fink and DarwinPorts.

If you're into the vim editor, tidy can be used to great effect to check and tidy HTML source
files directly within vim. Add the following file type plugin for html files.

$ cat ~/.vim/ftplugin/html/tidy.vim
""""""""""""""""""""""""""""""""""""""""""""""
" Adrian's vim filetype plugin for html tags "
""""""""""""""""""""""""""""""""""""""""""""""
"
" Maintainer: Adrian Mayo
" URL: http://www.mayo-family.com/vim/ftplugin/html/tidy.vim
" MAIN URL: http://www.mayo-family.com/vim
"
" This file defines maps to call the html tidy program, see:
" http://tidy.sourgeforge.net/
"
" This file should be placed in ~/.vim/ftplugin/html/ on Unix systems
" and will automatically be sourced for filetype = html.
"

" Map to check html and display error in a new window
"
noremap \err :1<CR>yG :20new<CR>p :setfiletype html<CR> :%!tidy -i >/dev/null<CR>

" Map to tidy html
"
noremap \tidy :%!tidy -i -f /dev/null<CR>

" END
"

The new commands are invoked by typing \err and \tidy in normal mode.

Redirection - Echo to Standard Error

They're of most use in shell scripts, but can be used directly on the command line too.

First off, here's how a script might send an error message to standard error instead of standard
out. Normally, when your script writes any type of message using a command such as 'echo', output
is sent to standard out.

Why would we want to write to standard error instead? Traditionally, Unix commands such as 'ls'
and 'mount' write output to standard out and error messages to standard error. This convention
ensures that if we were to pipe the command's output to another command (or redirect it to a
file) error messages would still displayed on the terminal screen and would not get mixed up with
the 'normal' output. We want our scripts to honour this convention.

As an example, let's write the error message "Can't open the file." to standard error.
$ cat example
#!/bin/bash

# some processing here...
echo "Can't open the file." 1>&2

The key is this after the echo command:
1>&2

which merges standard out (stream 1) into standard error (stream 2), hence causing 'echo' to
write to standard error.

We run it and see the error message:
$ ./example
Can't open the file.
$

To prove the message went to standard error and not standard out, we redirect standard out to the
file 'out' and standard error to the file 'err', displaying the contents of each file:
$ ./example >out 2>err
$ cat out
$ cat err
Can't open the file.
$

Here's an example in which we issue a command and want to report an error if the command failed.
We test the special shell variable '$?': if it's zero then no error occurred, if its non-zero the
command failed. Naturally, we want to send our error message to standard error, not standard out.

Here's how we achieve this in an example where we test the error status returned from the 'ls'
command. To prove the message was sent to standard error and not standard out, we redirect
standard out to the file 'out', but still see the error message on the screen.
$ ls zzz
ls: zzz: No such file or directory
$ if (($?!=0)); then echo "An error occurred" 1>&2; fi > out
An error occurred

Notice the use of '1>&2' to merge standard out into standard error, as in Monday's tip. Notice
also that we took advantage of Bash's integer expressions in:
(($?!=0))

More conventionally we would have written:
$ if [ $? != 0 ]; then echo "An error occurred" 1>&2; fi >out

Redirection - Redirect and Pipe Both

Monday's tip showed how we can make our scripts write error messages to standard error instead of
standard out, just like 'real' Unix commands. Today, we'll look at how to negate this convention
by merging the standard error from a command back into its standard out.

For example, suppose we wish to send both the standard out and standard error of 'ls' to the same
file. We use the special redirection '&>'. First, we'll test an 'ls' command line that lists
'file' (which fails) and the directory 'test' that contains 'file':
$ ls test file
ls: file: No such file or directory
test:
file
$

Next, we'll redirect standard out to 'out':
$ ls test file > out

$

Finally, we'll redirect both to the same file using the special redirection '&>':
$ ls test file &> out
$

Normally, when we pipe the output of one command to another, only standard out is piped. If you
need to pipe both standard out and standard error, first merge standard error into standard out,
then pipe standard out as normal. Here's an example using the same 'ls' command as above. We pipe
to the 'awk' command, which then prints its input lines with "Piped: " added to the beginning.

First, we pipe only standard out:
$ ls test file | awk '{print "Piped: " $0}'
Piped: test:
Piped: file
$

Next, we pipe both standard out and standard error.
$ ls test file 2>&1 | awk '{print "Piped: " $0}'
Piped: ls: file: No such file or directory
Piped: test:
Piped: file
$

Combine Redirections

Because of the order in which Bash processes a command line and redirection syntax, you must be
careful combining the merging of standard out and standard err (1>&2 or 2>&1) with redirection.

This example uses an 'ls' command line that lists 'file' (which fails) and the directory 'test'
that contains 'file':

$ ls test file
test:
file

We merge standard out into standard error and redirect standard error (hopefully this is now both
out and error) to the file 'err'.

Our first attempt fails:
$ ls test file 1>&2 2>err
test:
file
$

Reversing the order of the redirect and the merge will yield the desired behaviour:
$ ls test file 2>err 1>&2
$ cat err
test:
file
$

The reason the first attempt fails is that out is merged with error at the time error is still
going to the screen. Only later is error redirected to the file, too late for out to be
redirected too. (I've always though that the two syntaxes work the opposite way around to what
seems natural.)

To throw away either or both of standard out and standard error, redirect to the special file
/dev/null. For example, if we like only good news:
$ ls test file 2>/dev/null
test:
file
$

Redirect to a Variable

It's often useful to capture the output of a command into a shell variable. For example, this
command returns the device identifier of the device mounted at '/Volumes/pw'
$ df | awk '/\/Volumes\/pw/ {print $1}'
/dev/disk3s2
$

We capture the output from this command into a shell variable called 'device' by typing:
$ device="$(df | awk '/\/Volumes\/pw/ {print $1}')"
$ echo $device
/dev/disk3s2
$

(We might use $device in a 'mount' command later in the script.)

However, only standard out is captured, as shown in this next example where we form an invalid
'awk' command.
$ device="$(df | awk '/\/Volumes\/pw {print $1}')"
awk: non-terminated regular expression \/Volumes\... at source line 1
context is
>>> <<<

$ echo $device

$

To capture standard error too, we simply merge it with standard out using the syntax '2>&1'.
$ device="$(df | awk '/\/Volumes\/pw {print $1}' 2>&1)"
$ echo $device
awk: non-terminated regular expression \/Volumes\... at source line 1 context is >>> <<<
$

Note that other shells such as 'sh' and 'tcsh' use a different syntax for command substitution -
`command` instead of $(command):
$ device="`df | awk '/\/Volumes\/pw/ {print $1}'`"

Mysql - Grab Mysql

This week's tips show you how to install and set up the MySQL Database server, and how to
integrate it with the Apache-PHP module. We'll look at installing the latest stable release of
MySQL - 5.0.

First, download the binary distribution from MySQL.

Choose the appropriate Mac OS X Binary - 10.3 (Power PC), 10.4 (Power PC), or 10.4 (x86). I've
tried both the standard and the 64 bit versions for 10.4 Power PC. However, although the 64 bit
version works, I could not compile PHP against that distribution. I've had no problems with the
non-64 bit version running on an iMac G5.

You have a second choice too, between a Mac-style Package installer and a Unix style Tar package
(found towards the bottom of the HTML page in the 'special' section). These instructions assume
10.4 Power PC Tar package.

We'll install MySQL in the standard location of /usr/local. Type the following commands:

To become the root user (give your admin password) type:
$ sudo -s
Password:
#

To install MySQL (the name of the Tar archive will be different for other distributions; also,
I've assumed the tar file was downloaded to ~/Desktop) type:
# cd /usr/local
# tar -xzf ~/Desktop/mysql-standard-5.0.18-osx10.4-powerpc.tar.gz
# ln -s mysql-standard-4.1.16-apple-darwin8.2.0-powerpc mysql

This uncompresses the archive and makes a symbolic link (not a copy) called mysql to the actual
installation.

Next, apply the correct owner and permissions to the install:
# cd mysql
# chown -R root:mysql .

It's a good idea to add the path of the MySQL binaries to your PATH variable so you don't have to
type the full path name to execute a MySQL command. (This assumes you are using the Bash shell.
Other shells vary.) The path is '/usr/local/mysql/bin'. A typical PATH variable would be set in
/etc/profile and look like this:
declare -x PATH="\
/usr/local/bin:/usr/bin:/bin:\
/usr/local/sbin:/usr/sbin:/sbin:\
/usr/X11R6/bin:\
/usr/local/mysql/bin:\
"

The statement is broken into six lines. It's ok to break it like this in the file, provided '\'
is added to the end of each broken line. The line pertinent to MySQL is '/usr/local/mysql/bin:\'.

Tuesday's tip will show you how to create the MySQL database and test the installation. The
database will be created in a more useful location that the default /usr/local/mysql/data.

Mysql - Create a Database

The database directory, by default, is created under the actual installation in
/usr/local/mysql/data. It's better to create it elsewhere, perhaps under /var where most server
generated data, such as incoming mail and log files, is stored.

To become the root user (give your admin password) type:
$ sudo -s
Password:
#

Create the database directory /var/mysql-data (you can place it at another location if you wish).
Type:
# mkdir /var/mysql-data
# chown mysql:mysql /var/mysql-data
# chmod 770 /var/mysql-data

Create the MySQL configuration file, /etc/my.cnf, which tells mysql where the database it held.
This is what it should contain:
# cat /etc/my.cnf
[mysqld]
datadir=/var/mysql-data

Next, create the necessary administrative databases by typing:
# /usr/local/mysql/scripts/mysql_install_db --user=mysql --datadir=/var/mysql-data

This should create all the necessary files with owners mysql:mysql.

Test the installation - start the server by typing:
# mysqld_safe --user=mysql &
[1] 19718
# Starting mysqld daemon with databases from /var/mysql-data

(If mysqld_safe cannot be found, see Monday's tip to add the MySQL executable directory to your
PATH. Once you have done so, start a new terminal session. Alternatively, type the full path name
to this and other MySQL commands - eg /usr/local/mysql/bin/mysqld_safe.)

Check that the three default databases are there

# mysqlshow
+--------------------+
| Databases |
+--------------------+
| information_schema |
| mysql |
| test |
+--------------------+

MOST IMPORTANT: enable the root password by typing:
# mysqladmin -u root password my-password
# mysqladmin -u root -h hostname.domain.com password my-password

Replace 'hostname.domain.com' with your host's name (eg 'mymac.local'). You might get away with
typing just the first line. Obviously, replace my-password with your chosen MySQL root password.
(Don't confuse the Unix root user with the MySQL root user - they are entirely separate.)

NOTE: From this point on, it's necessary to give the option -p to most MySQL commands and type
root's password at the prompt.

As a further test of successful installation, stop the server by typing:

# mysqladmin -u root -p shutdown
Enter password:
STOPPING server from pid file /var/mysql-data/...
060111 00:51:57 mysqld ended

[1]+ Done mysqld_safe --user=mysql
#

Next, we must create a Mac OS X startup item so MySQL will be launched when your Mac starts.
Create a directory by typing:

# mkdir /Library/StartupItems/MySQL

Create two files in the directory, called MySQL and StartupParameters.plist. They are included in
the Mac-style Package installer, but not in the Unix style Tar package.

Change to the appropriate directory:
# cd /Library/StartupItems/MySQL

Create the files with the following contents:

-- MySQL --
#!/bin/sh
#

# Startup Item for MySQL
# ######################

. /etc/rc.common

mysqlctl="/usr/local/mysql/support-files/mysql.server"

StartService ()
{
if [ "${MYSQLCOM:=-NO-}" = "-YES-" ] ; then
echo "Starting MySQL server"
$mysqlctl start
fi
}

StopService ()
{
echo "Stopping MySQL server"
$mysqlctl stop
}

RestartService ()
{
echo "Restarting MySQL server"
$mysqlctl restart
}

if [ -x $mysqlctl ]; then
RunService "$1"
else
echo "Cannot execute $mysqlctl"
fi

-- StartupParameters.plist --
{
Description = "MySQL Startup";
Provides = ("MySQL");
Requires = ("DirectoryServices", "NetworkExtensions");
OrderPreference = "Last";
}

Make sure the script is executable:
# chmod +x MySQL

When you reboot, MySQL should automatically start up.

NOTE: If you copy and paste this code from the OSXFAQ forums, you might have problems because
PHP-BB adds odd invisible characters to the code sections.

Wednesday's tip will show how to add a new database, and how to add a user to the database.

Mysql - Add a Database and a User

Let's create a database called jan. You need not be the Unix root user, but must be the mysql
root user.
$ mysql -u root -p
Enter password:
mysql> create database jan;
Query OK, 1 row affected (0.00 sec)

Then create a user, also but not necessarily called jan, and give that user permissions on the
jan database. Type:
mysql> grant all on jan.* to jan@localhost identified by 'jans-password';
Query OK, 0 rows affected (0.00 sec)

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| jan |
| mysql |
| test |
+--------------------+
4 rows in set (0.01 sec)

To view all users (user information is held in the mysql database) type:
mysql> use mysql;
Database changed
mysql> Select * from user\G
...
mysql> quit
Bye
$

Mysql, PHP and Passwords

The first three tips showed how to install MySQL and how to create a new database and user. If
you wish to use MySQL with Apache-PHP, and you are using Apple's PHP install, then you'll have
problems.

Firstly, Apple's PHP install, at the time of writing, was built to use old-style MySQL password
encryption, whereas MySQL 4.1 and later use the new-style encryption.

To allow PHP to connect to a database, either don't give a password by missing out:
identified by 'password';

when you create a new user. Alternatively, create a new user with an old-style password. To do
this, create the user as given in Wednesday's tip and then type (this example shows a user called
jan):
# mysql -u root -p
Enter password:
mysql> set password for jan@localhost = old_password('jans-password');

mysql> quit
Bye
$

Also, as of the Tiger 10.4.4 update, and until Apple corrects it, the Apache-PHP module was built
in a way that's incompatible with most MySQL installs. This can be corrected by building your own
PHP install, as shown in Friday's tip.

Mysql - Build the Apache PHP Module

As of the 10.4.4 update, and until Apple corrects it, the Apache-PHP module was built in a way
that's incompatible with most MySQL installs. This can be corrected by building your own PHP
install, or by re-installing and re-configuring MySQL. The problem lies in the configuration of
Apple's PHP build, applied when Apple builds the PHP module. In particular, Apple used the
configuration commands:
'--with-mysql=/usr'
'--with-mysql-sock=/var/mysql/mysql.sock'

meaning MySQL is expected to be found at /usr instead of /usr/local, and also the socket is to be
found at /var/mysql instead of /tmp.

You can solve this by moving the MySQL installation and starting the MySQL daemon (in the startup
items) using the option:
--socket=/var/mysql/mysql.sock

Alternatively, re-build PHP. These instruction apply to PHP-4. If you are running 5, you'll
already know how to install PHP-5.

Download the (bzip2) source from: PHP.

Extract it by typing:
$ tar xf php-4.4.1.tar.bz2 --bzip2
$ cd php-4.4.1

(You version may be different to 4.1.1).

Configure it by typing (on a single command line):
$ ./configure --with-mysql=/usr/local/mysql
--with-apxs --mandir=/usr/local/share/man

Or, for a more fully featured PHP, configure by typing:
$ ./configure --prefix=/usr/local
--mandir=/usr/local/share/man --with-apxs
--with-mysql=/usr/local/mysql
--with-mysql-sock=/tmp/mysql.sock
--with-openssl=/usr --with-ldap=/usr
--with-kerberos=/usr --with-zlib-dir=/usr
--with-iodbc=/usr --with-curl=/usr --enable-exif
--enable-ftp --enable-dbx --enable-sockets

Build it by typing:
$ make

Install it by typing:
$ sudo -s
Password:
# mv /usr/libexec/httpd/libphp4.so /usr/libexec/httpd/libphp4.so.orig
# make install

NOTE: Run
./configure --help

for a full list of the configure options.

Restart Apache (if it's already running) so the new PHP module is read.
# apachectl graceful

Install extra packages with PEAR. The install does not include the DB package by default. (Why?)
It can be installed by user root using pear. Type:
# pear install DB

NOTE: Some handy pear commands:
pear # lists all pear commands
pear help command # get help on a specific command
pear config-show # shows the config settings, these
should point to directories
within /usr/local
pear list # list all installed packages

With any luck, you should have an Apache-PHP module that talks to the MySQL 5 installation from
the earlier tips.

Scripting one-liners - Authentication

Suppose you have a script that uses 'sudo' to execute a command as root. When the script executes
sudo, the user must authenticate with a password. If the user fails to authenticate then sudo
will not run the command and we must terminate the script.

If the script is already part-way through execution we might have to undo some of the stages its
already gone through. Ideally, we want to achieve authentication at the start of the scrip, even
though the command that requires it occurs later.

To do this we rely on the fact that a sudo authentication is valid for five minutes. Place this
line at the start of the script:
sudo -p Admin password echo 2> /dev/null || { echo Incorrect ; exit; }

How does it work? Option -p causes sudo prints a prompt. The sudo command just executes a null
echo command. If authentication is not successful, sudo returns failure (or False) and the
following statement (after the || 'OR') operator is executed. This statement prints an error
message and exits.

If authentication is successful, sudo returns success (or True) and the second statement is
skipped.

An OR operator executes its statements only in order to produce a Boolean result, True or False.
It reasons that if the first statement yields True, then the second statement needn't be executed
because it won't affect the overall result - True OR anything = True.

Similarly, the AND operator (&&) doesn't execute its second statement if the first yields False -
False AND anything = False.

Scripting One-Liners - Fancy Printing

Ever wanted to spice up the output from a script, like aligning numbers or zero padding them.
Then consider using 'printf' over echo.

Here's how to print numbers in columns, right justified.

$ num1=4; num2="123"
$ printf "Value: %5d units\n" $num1; printf "Value: %5d units\n" $num2
Value: 4 units
Value: 123 units

Printf prints what's inside its format string. %5d tells printf to look outside the format string
and interpret what it finds as a decimal number and to write it to a column 5 characters wide.

And to left justify:
$ printf "Value: %-5d units\n" $num1; printf "Value: %-5d units\n" $num2
Value: 4 units
Value: 123 units

And to print leading zeros:
$ printf "Value: %05d units\n" $num1; printf "Value: %05d units\n" $num2
Value: 00004 units
Value: 00123 units

You can also change the format stored in a variable:
$ num1=$(printf "%05d" $num1)
$ echo $num1
00004

The printf function can do much than than shown here. It uses the printf library function, which
is documented by:
$ man 3 printf

Scripting One-Liners - More printf

We can take advantage of a very useful feature of printf to print tables of data. Check this
example:

$ printf "<tr>\n <td>%s</td> <td>%s>/td>\n</tr>\n" Apples 7 Pineapples 5 Pears 6
<tr>
<td>Apples</td> <td>7</td>
</tr>
<tr>
<td>Pineapples</td> <td>5</td>
</tr>
<tr>
<td>Pears</td> <td>6</td>
</tr>
$

The printf statement prints what's in its format string. %s tells printf to look outside the
format string for a value to substitute. Because the format string has two occurrences of %s, but
we supply six values, it takes three sweeps at the string.

Using the next trick, we can read the values from a file.

$ echo "Apples 7 Pineapples 5 Pears 6" > fruit-count
$ printf "<tr> <td> %s</td> <td>%s</td>\n</tr>\n" $(cat fruit-count)
<tr>
<td>Apples</td> <td>7</td>
...

Scripting One-Liners - Batch Process Files

Here's a handy way to process every file in a directory:
$ for f in *; do echo "File: $f"; done
File: a file with spaces.txt
File: db-snmpd.conf
File: snmp.conf
File: snmp.txt
File: snmpd.conf
File: treewalk-dot
$

(A more useful example would do something more than simply echo each filename.) It'll even cope
with filenames that contain spaces, as witnessed by the first file in the list.

You can process each field in a file with:
$ for f in $(cat file-to-process); do echo $f; done

Scripting One-Liners - Bulk Redirection

We all know how to redirect output to a file using the redirection symbol '>'.
$ ls > file-list

Here's an example in which we redirect the output of many commands in a shell script, and without
having to redirect the output of each command individually. Create a sub-shell around the
commands who's output you wish to redirect.
$ (
>ls
>df
>cat number-list
>) > out
$

You can use a group-command instead of a sub-shell by replacing
(...) > out

with
{...} > out

Use the same trick to re-direct standard input to a group of commands.
$ cat infile
1234
full
$ { read code; read membership; } < infile
$ echo $code $membership
1234 full
$

Note: use a group-command, not a sub-shell, if you wish to retain the values of variables. A
sub-shell is executed by a new instance of the shell, and its environment (including all
variable) dies when the sub-shell completes.

Spotlight - Reveal the Metadata

I'm sure you're familiar with Apple's Spotlight and its powerful search capabilities. It searches
file content rapidly using a pre-built content index database. It also searches for information
about files, making use of metadata. Metadata is data about data. For example, an image created
by Photoshop contains data that is the image itself, but the filename, date stamp, owners, file
size, and such like form the metadata.

(Note: Spotlight was introduced in Mac OS X Tiger 10.4)

Apple, being a nice Unix player, has opened up the power of Spotlight to the command line. Let's
look at what's available in Terminal land.

The 'mdls' command (metadata ls) shows all metadata attached to a file or directory. For example,
if we mdls a TextEdit file saved in Rich Text format we see:

$ mdls schedule.rtf
schedule.rtf -------------
kMDItemAttributeChangeDate = 2006-02-06 00:59:21 +0000
kMDItemContentCreationDate = 2004-06-03 17:17:17 +0100
kMDItemContentModificationDate = 2006-02-06 00:59:21 +0000
kMDItemContentType = "public.rtf"
kMDItemContentTypeTree = ("public.rtf", "public.text", "public.data", "public.item",
"public.content")
kMDItemDisplayName = "schedule"
kMDItemFSContentChangeDate = 2006-02-06 00:59:21 +0000
kMDItemFSCreationDate = 2004-06-03 17:17:17 +0100
kMDItemFSCreatorCode = 0
kMDItemFSFinderFlags = 16
kMDItemFSInvisible = 0
kMDItemFSIsExtensionHidden = 1
kMDItemFSLabel = 0
kMDItemFSName = "schedule.rtf"
kMDItemFSNodeCount = 0
kMDItemFSOwnerGroupID = 501
kMDItemFSOwnerUserID = 501
kMDItemFSSize = 7607
kMDItemFSTypeCode = 0
kMDItemID = 923169
kMDItemKind = "Rich Text Format (RTF) document"
kMDItemLastUsedDate = 2006-02-06 00:59:21 +0000
kMDItemUsedDates = (
2006-01-14 00:00:00 +0000,
2006-01-20 00:00:00 +0000,
2006-01-21 00:00:00 +0000,
2006-01-23 00:00:00 +0000,
2006-02-05 00:00:00 +0000,
2006-02-06 00:00:00 +0000
)

You'll see all the usual metadata normally displayed by 'ls -l', such as filename name:

kMDItemFSName = "schedule.rtf"

and user and group owner (shown as numeric IDs - the translation to textual names is performed by
ls):


You'll notice other metadata too, such as:

kMDItemDisplayName - the name as shown in the Finder, which often means without the extension.

kMDItemKind - the type of content.

Metadata with the types kMDItemFS* is information from the filesystem. Other metadata types
concern the file contents and is not information that ls reports.

BTW, Spotlight stores its content index and metadata store are held in a directory at the top
level of each file system:

$ sudo ls /.Spotlight-V100/
.journalHistoryLog ContentIndex.db _exclusions.plist store.db
.store.db _IndexPolicy.plist _rules.plist

Each file system has its own Spotlight store ensuring that this information travels with the
files to which it belongs.

Tomorrow, we'll look at what metadata is available for other types of file content.

Spotlight - Pictures and Songs

Today's tip looks at file content metadata. Different types of file content have different types
of metadata. Let's pass a photoshop file to mdls:

$ mdls heart.psd
heart.psd -------------
kMDItemAttributeChangeDate = 2006-01-03 00:21:09 +0000
kMDItemBitsPerSample = 40
kMDItemColorSpace = "RGB"
kMDItemContentCreationDate = 2004-02-06 02:12:09 +0000
kMDItemContentModificationDate = 2004-02-06 02:12:09 +0000
kMDItemContentType = "com.adobe.photoshop-image"
kMDItemContentTypeTree = (
"com.adobe.photoshop-image",
"public.image",
"public.data",
"public.item",
"public.content"
)
kMDItemCreator = "Adobe Photoshop 7.0"
kMDItemDisplayName = "heart"
kMDItemFSContentChangeDate = 2004-02-06 02:12:09 +0000
kMDItemFSCreationDate = 2004-02-06 02:12:09 +0000
kMDItemFSCreatorCode = 943868237
kMDItemFSFinderFlags = 1024
kMDItemFSInvisible = 0
kMDItemFSIsExtensionHidden = 1
kMDItemFSLabel = 0
kMDItemFSName = "jj1.psd"
kMDItemFSNodeCount = 0
kMDItemFSSize = 2204679
kMDItemFSTypeCode = 943870035
kMDItemHasAlphaChannel = 1
kMDItemID = 923966
kMDItemKind = "Adobe Photoshop file"
kMDItemLastUsedDate = 2006-01-03 00:21:09 +0000
kMDItemLayerNames = (jan, hair)
kMDItemOrientation = 0
kMDItemPixelHeight = 718
kMDItemPixelWidth = 450
kMDItemResolutionHeightDPI = 72.00006
kMDItemResolutionWidthDPI = 72.00006
kMDItemUsedDates = (2005-11-02 12:18:02 +0000, 2006-01-03 00:00:00 +0000)

The last few items show metadata describing the image size, resolution, and even on what
photoshop layers it contains.

Try passing a song file to mdls. The metadata includes the genre, title, and bit rate.

...
kMDItemMediaTypes = (Sound)
kMDItemMusicalGenre = "Alternative & Punk"
kMDItemStreamable = 0
kMDItemTitle = "Like It Or Leave It"
kMDItemTotalBitRate = 191384
...

To extract a specific metadata item, pass the item name as an argument to the name option:
$ mdls -name kMDItemAuthors "12 DARE.m4a"
12 DARE.m4a -------------
kMDItemAuthors = (Gorillaz)
$

At this point, you might be tempted to form a 'find/xargs/grep' command line to search through,
say, all music for songs a particular Genre. But Apple has provided a much better way of doing
this...a command line spotlight.

Tomorrow's tip will look at spotlighting on the command line.

A tip to save you life: "Remember to buy her a card!"

Spotlight - Spotlight from the Command Line

So far we've seen how to examine file metadata using the mdls command. To search the file system
for specific content or metadata, use mdfind. Here's an example to search for the text "DOCSIS
MIB" (it's a cable TV term):
$ mdfind "DOCSIS MIB"
/Users/saruman/1dot1/Miri/emulator/design/figure-1.psd
/Users/saruman/Sites/adrian/data-base/rfc-index.txt
/Users/saruman/1dot1/Miri/emulator/proposals/Emulator.swx
/Users/saruman/1dot1/Miri/emulator/design/Emulator.pdf

These results should match those found by pressing Command-Space and entering the same term into
the Spotlight bar.

Here's another example:
$ mdfind "miranda"
/Users/Shared/Books/Folders/Shakespear/COMEDY/TEMPEST.PDF

Or:
$ mdfind "kiss me, kate"
/Users/Shared/Books/Folders/Shakespear/COMEDY/LOVESLAB.PDF
/Users/Shared/Books/Folders/Shakespear/COMEDY/MEASURE.PDF
/Users/Shared/Books/Folders/Shakespear/COMEDY/SHREW.PDF
/Users/Shared/Books/Folders/Shakespear/COMEDY/TEMPEST.PDF

Why does this search produce more results than just SHREW.PDF? Because Spotlight is looking for
kiss AND me AND kate

(it ignores the comma) but not necessarily consecutively. If all three words appear somewhere in
a document, that document will match the search.

We can also look for specific metadata. Tuesday's tip showed us that songs include the metadata
item
kMDItemMusicalGenre

Let's search for all songs that are 'Hip Hop/Rap':
$ mdfind "kMDItemMusicalGenre == 'Hip Hop/Rap'"
/Users/sharing/Music/iTunes/iTunes Music/Compilations/Encore/01 Curtains Up.m4a
/Users/sharing/Music/iTunes/iTunes Music/Compilations/Encore/02 Evil Deeds.m4a
/Users/sharing/Music/iTunes/iTunes Music/Compilations/Encore/03 Never Enough.m4a
/Users/sharing/Music/iTunes/iTunes Music/Compilations/Encore/04 Yellow Brick Road.m4a
/Users/sharing/Music/iTunes/iTunes Music/Compilations/Encore/05 Like Toy Soldiers.m4a

Use mdls to discover other types of metadata for other content types.

Tomorrow's tip will show how to perform more complex searches using mdfind.

Spotlight - Boolean Expressions

Yesterday, we saw how to use mdfind to find files that contain particular text, or have
particular metadata. Today we'll look at how we can be more exacting about what words a file
should, or should not, contain.

To illustrate, we have three simple text files:
$ cat file1.txt
Kiss me
$ cat file2.txt
Kate
$ cat file3.txt
Kiss me, kate

Let's look for files that contain the word "kiss":
$ mdfind -onlyin ~/splotlight-test "kiss"
/Users/saruman/splotlight-test/file1.txt

Note use of the option -onlyin to limit searches to files rooted in a particular directory
hierarchy. You must specify the full path name of the directory.

And files that contain the word "kate":
$ mdfind -onlyin ~/splotlight-test "kate"

If we specify both terms, Spotlight will return those files that contain ALL the terms - in this
example "kiss" AND "kate".
$ mdfind -onlyin ~/splotlight-test "kiss kate"

To test for "kiss" OR "kate", we separate the terms with '|'.
$ mdfind -onlyin ~/splotlight-test "kiss|kate"

There must be no white space around the '|' symbol.

What about not kissing kate?
$ mdfind -onlyin ~/splotlight-test "kiss(-kate)"

Again, make sure there is no white space between the terms.

And finally, "kiss" or "kate", but not "me".
$ mdfind -onlyin ~/splotlight-test "kiss|kate(-me)"

Tomorrow, we'll combine mdfind with regular Unix commands such as find, grep, and xargs.

Spotlight - Piping its Output

Just like the regular Unix find, the list of files returned by mdfind can be piped to other Unix
commands for further processing.

The next command will filter the results from find leaving only files that have the extension
".pdf".
$ mdfind "kiss kate" | grep -i "\.PDF$"

We escaped the dot because it has a special meaning to regular expression, and used the dollar
symbol to anchor matching text to the end of the term (i.e. matching filenames must end in .pdf,
not just have .pdf in the middle).

To open all the files in Preview type:
$ mdfind -onlyin /Users/shared/books "kiss kate" | xargs open

To make the search term more selective, use grep and regular expressions. Here's a simple
example:
$ cat file1.txt
Kiss me $ cat file3.txt
Kiss me, kate $ mdfind -onlyin /Users/saruman/splotlight-test "kiss me,"

The comma is ignored by spotlight. Let's use grep:
$ mdfind -onlyin /Users/saruman/splotlight-test "kiss me," | xargs grep -i "Kiss me,"
/Users/saruman/splotlight-test/file3.txt:Kiss me, kate
grep: /Users/saruman/splotlight-test/file: No such file or directory
grep: 4.txt: No such file or directory
$

(Tip: Use grep to make the search term case sensitive.)

The command choked on the filename "file 4.txt" because it contains a space. Specify option -0
(zero) to both mdfind and xargs to fix this.
$ mdfind -0 -onlyin /Users/saruman/splotlight-test "kiss me," | xargs -0 grep -i "Kiss me,"
/Users/saruman/splotlight-test/file3.txt:Kiss me, kate
/Users/saruman/splotlight-test/file 4.txt:Kiss me, kate
$

This is the same trick as used with "find | xargs" as is described here.

1 I 44 Lbfpsvivzz 8 Zvqs 9

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

1 I 44 Lbfpsvivzz 8 Zvqs 9

Uploaded by

Copyright:

Available Formats

Unix_tricks.

txt Page 1 of 271

++ Basic UNIX Tutorials

Device Driver Tutorial (http://docs.sun.com/app/docs/doc/817-5789) (SunOS)

How to set up sar (http://thproxy.jinr.ru/file-archive/doc/sun/sarhowto.htm)

Escaping is also used to reference a filename that includes spaces.

Pick and mix these.

tr \\r \\n <x >x.new

sudo ditto -rsrc /Users/ /Volumes/Backup/2002.10.16/Users

defaults read com.apple.terminal

/dev/disk0s11 on /Users (local)

Q. What is the ASCII code for 'A'?

Files f1.txt and f2.txt differ

This Line not in File Two

find . -name "file*"

@monthly once a month == "0 0 1 * *"

This technique can be used to avoid replacing the original file:

Remove the extension:

We all know of the venerable 'ping' utility.

10 dnvrco1wcx2-pos10-0.wcg.net (64.200.210.66) 169.951 ms 170.472 ms 170.171 ms

Destination Gateway Flags Refs Use Netif Expire

The following answer is not authoritative:

For private files:

This applies especially to commands like:

Filenames With Spaces

Use cut, copy, and paste:

mv ./dirname/file2 name ./dirname/file2name

To repair the boot volume type:

For example, make the following directory structure:

this is line one

alias tlog 'tail -f -n35 /var/log/\!:1.log'

httpd) apachectl graceful;;

level, and its own configuration. (See Tuesday.)

# messages shouldn't go to terminals or publically-readable

by '#' for root.

Open a folder into a Finder window:

Search and Replace

code and data sections).

(text previously copied from a GUI application)

There is no manual page, so use:

to recognise the '|' as a token and therefore as the pipe symbol.

Change Your Shell

create a file called mounts:

A handler assigned to a sub-shell is not inherited by the parent shell.

3 < 2 < 1 ! You jest.

if [ "$password" != "" ]; then

MATCHED: the second line

the word witch, and some of witch

fs_usage must be run as root.

<b><i>Hello in bold italic</i></b>

bash: aaa: No such file or directory

# The script must be run by root

echo "This is the user's primary group - not added"

None of these attempt work:

*.text : head: *.text: No such file or directory

echo " otherwise a warning is issued if the group to"

read -p "Type a to abort: " ans

echo "User $user removed from group $group"

It uses the previous two scripts.

List all files created since the last restart:

$ find . -name "file*" | xargs ls -l

echo -n "^[[6t;"; sleep 5; echo -n "^[[5t;"

.text : head: .text: No such file or directory