Fundamentals

of
UNIX

Fundamentals of 1
 Index

Fundamentals of 2
 Chapter 1

UNIX Operating System

Fundamentals of 3
 Objectives
In this session, you learn about:
• The functions of OS
• The history of Unix
• The features of UNIX
• The Unix architecture
• Process management
• CPU scheduling
• Memory management
• File management

Fundamentals of 4
 Operating System (OS)

• OS is a system software

• OS can be defined as an organized collection of software

consisting of procedures for operating a computer

• OS provides an environment for execution of programs

• OS acts as an interface between the user and the

hardware of the computer system.

Fundamentals of 5
 Operating System

• Operating system interacts with user in two ways

• Operating system commands
Enables user to interact directly with the operating system.

• Operating system calls

Provides an interface to a running program and the
operating system. System calls in UNIX are written in C.

Fundamentals of 6
 History of UNIX
• Ken Thompson of AT&T Bell Laboratories designed UNIX
in late 1960s

• Two versions of UNIX that emerged are AT&T Unix and

BSD Unix

• In 1989, AT&T and Sun Microsystems joined together and

developed system V release 4 (SVR4)

• Two of the main standards mainly in use are POSIX

(Portable Operating System Interface) and X/open
standard. In 1988, MIT formed Xconsortium developed
vendor-neutral Xwindow System.

Fundamentals of 7
 What is Linux?

• An open-source UNIX like operating system

• Initially created by Linus Torvalds for PC architecture

• Ports exist for Alpha and Sparc processors

• Developer community world-wide contribute to its enhancement

and growth

Fundamentals of 8
 Features of UNIX
• Multi-user, multitasking, timesharing

• Portability

• Modularity

• File structure

• Security

• Strong networking support & advanced graphics

Fundamentals of 9
 Layered Architecture

... banner
ls
cp kernel sort
comp
sh
shell hardware
as
who
ld a.out

vi date
ed wc
grep

Fundamentals of 10
 UNIX System Architecture

• Unix system follows a layered approach. It has four layers

• The innermost layer is the hardware layer

• In the second layer, the kernel is placed

• The utilities and other application programs form the third layer

• Fourth layer is the one with which the user actually interacts.

Fundamentals of 11
 Kernel

• Kernel is that part of the OS which directly makes interface

with the hardware system.
• Actions:
• Provides mechanism for creating and deleting processes
• Provides processor scheduling, memory, and I/O
management
• Provides inter-process communication.

Fundamentals of 12
 The Shell
• A utility program that comes with the UNIX system.
• Features of Shell are:
• Interactive Processing
• Background Processing
• I/O Redirection
• Pipes
• Shell Scripts
• Shell Variables
• Programming Constructs

Fundamentals of 13
 Process Management

• A process is a program in execution

• Several processes can be executed simultaneously in a

UNIX system.

• A process is generally created using the “fork( )”

system call.

• The process that invokes the “fork( )” system call is the

parent process, and the newly created process is called
the child process.

Fundamentals of 14
 CPU Scheduling

• Unix uses round-robin scheduling to support its multi-user

and time-sharing feature.

• Round-robin fashion of scheduling is considered to be the

oldest, simplest and widely used algorithm.

• Every process is given a time slice (10-100 millisec.)

Fundamentals of 15
 Memory Management

• Virtual memory

• Swap area
• Demand paging

Fundamentals of 16
 File Management

• UNIX uses a hierarchical file system with “/” as its root.

• Every non-leaf node of the tree is called as a directory file.

• Every leaf node can either be a file, or an empty directory

Fundamentals of 17
 File System

dev bin tmp home etc var lib usr

spool
inittab bin src
sh
console ls passwd
user1
lp0
user2

Fundamentals of 18
 File System

• File system is the structure in which files are stored on disk

• File in UNIX is sequence of bytes organized in the form of

blocks

• The size of each block is 512 bytes (depends on architecture)

• Block size can be decided while creating the file system

structure

Fundamentals of 19
 File System Structure
Type of the file
Link counter gle
Uid, gid, size Sin irect Double
ind indirect
Date and time of Creation
Boot Block Date and time of access
Date and time of
Super Block modification
:
ple
Tri rect
i
: Ind
Inode Block
Data Block Address of datablock

Address of datablock
:
:

Address of the addr block

Address of the addr block

Address of the addr block

Fundamentals of 20
 Common UNIX Flavours
BSD: Berkeley, BSD

Solaris: Sun Microsystems, Sys 5/BSD

Ultrix: Digital Equipment Corporation, BSD

OSF 1: Digital Equipment Corporation, BSD/sys 5

HPUX: Hewlett-Packard, Sys 5

AIX: IBM, Sys 5 / BSD

IRIX: Silicon Graphics, Sys 5

GNU/Linux: GNU, BSD/Posix

Fundamentals of 21
 Types of UNIX Users

• Broad classification of users

• root (most privileged)
• Non-root (less privileged)

• Group
• UNIX allows user IDs to be grouped
• A single user ID can be member of multiple groups

• Differentiating users with respect to file access

• Owner
• Group
• Others

Fundamentals of 22
 Working With UNIX

• User logs in with a valid user ID

• User logs out to terminate the login session

Fundamentals of 23
 Summary
In this session, you learned about …
• The functions of OS
• The History of Unix
• The features of Unix
• The Unix Architecture
• Process management
• CPU Scheduling
• Memory management
• File management

Fundamentals of 24
 Chapter 2

UNIX Commands

Fundamentals of 25
 Objectives

In this session, you will learn to:

• Use the basic Unix commands
• pwd
• date
• who
• ls
• man
• Use “man” pages

Fundamentals of 26
 Simple Commands

• pwd
• Displays the current working directory.
• date
• Displays the current date and time

Fundamentals of 27
 Simple Commands

• who
• Displays the names of all the users who have
currently logged in
• who am i
• Displays the name of the current user.

Fundamentals of 28
 Listing the Directory Contents

• ls
Syntax :ls [options] [file….]
options: -l list in long format
-a list all files including those
beginning with a dot
-i list inode no of file in first column
-s reports disk blocks occupied by file
-R recursively list all sub directories
-F mark type of each file
-C display files in columns

Fundamentals of 29
 Meta Characters

Meta Purpose Example

Characters
* Match with one or more characters or none $ ls –l *.c file*
? Match with any single character $ ls –l file?
[…] Match with any single character within the $ ls –l file[abc]
brackets
; Command separator $ cat file1; cat file2
| Pipe two commands $ cat abc | wc
() Group commands $ (echo “==== x.c ====”; cat x.c) > out
Useful when the output of thecommand group
has to be redirected
`command` Execute the command enclosed within back count=`expr $count + 1`
quotes. Useful when the output of a command
into a variable in a shell script assuming count has value3, this
increments the value of count
‘string’ Quote all characters with no substitution echo ‘expr $count + 1’
(ex. no special meaning for $) displays expr $count + 1
“string” Quote all characters with substitution. echo “expr $count + 1”
The characters $,\ (back slash) and back quote displays expr 3 + 1
have special meaning. assuming the variable count has value 3

Fundamentals of 30
 Listing the Directory Contents
$ ls –l

total 6
-rwxr-xr-x 1 user1 projA 12373 Dec 15 14:45 a.out
drwxr-xr-x 2 user2 projD 4096 Dec 22 14:00 awkpro
-rw-r--r-- 1 user1 projA 12831 Dec 12 13:59 c
-rw------- 1 user1 projA 61440 Dec 15 11:16 core
-rw-r--r-- 1 user3 projC 255 Dec 20 14:29 cs

File access File size Date & time of File name

permissions User id
in bytes modification
File type Group id
Link count

Fundamentals of 31
 Getting Help on Commands
• The Unix manual, usually called man pages, is available on-
line to explain the usage of the Unix system and commands.

Syntax:
• man [options] command_name

Common Options
-k keyword list command synopsis line for all keyword matches
-M path path to man pages
-a show all matching man pages (SVR4)

• info command_name - help for commands

• help -–command_name– gives command synatx

Fundamentals of 32
 Summary
In this session, you have learned to …
• use the basic Unix commands like
• pwd
• date
• who
• ls
• man
• use “man” pages

Fundamentals of 33
 Chapter 3

Files & Directories

Fundamentals of 34
 Objectives

• In this session, you will learn to:

• set file permissions using the chmod command
• use directory-related commands namely mkdir, rmdir,
cd commands
• use file-related commands namely cp, mv, rm
commands
• access advanced file permissions using commands
umask, suid, sgid, linking files, stickybit
• create and edit files using the vi editor

Fundamentals of 35
 File Access Permissions
• Refers to the permissions associated with a file with respect to
the following

• Permission Levels
• User (owner) (u)

• Group (wheel, staff, daemon, etc.) (g)

• World (guest, anonymous and all other users) (o)

• Permission Settings
• Read (r)

• Write (w)

• Execute (x)

Fundamentals of 36
 File Access Permissions

• No read permission does not allow the user to:

• List the contents of directory

• Remove the directory

• No Write permission does not allow the user to :

• copy files to the directory

• remove files from the directory

• rename files in the directory

• make a subdirectory

• remove a subdirectory from the directory

• move files to, and from the directory

Fundamentals of 37
 File Access Permissions

• No execute permission does not allow the user to:

• display the contents of a directory file from within the

directory

• change to the directory

• display a file in the directory

• copy a file to, or from the directory

Fundamentals of 38
 Changing Permissions - chmod

• chmod u+x file_name

Syntax:
chmod <category> <operation> <permission> <filename(s)>
or
chmod <octal number> filename

Octal Number
4 - for read
2 - for write
1 - for execution

$ chmod 744 xyz

this sets read, write and execute permissions for owner, read
permission for group and others

Fundamentals of 39
 Directory Creation
Command Syntax
mkdir [OPTION] DIRECTORY
$ mkdir <path>/<directory>
$ mkdir –m <directory>
$ mkdir –p <directory1>/<directory2>/<directory3>

Example:
$ mkdir project1
This creates a directory project1 under current directory

Note: Write and execute permissions are needed for the

directory in which user wants to create a directory

Fundamentals of 40
 Directory Removal
rmdir command removes directory
Syntax
– rmdir <directory name>
Example
Removes project1 directory in the current directory
– rmdir project1
Remove multiple directories
rmdir pos1 pos2
Remove the directory recursively
rmdir –p dir1/dir2/dir3

rmdir removes a directory if it is empty and is not the current

directory

Fundamentals of 41
 Command - cd
cd command is used to change the directory

• cd - take to the home directory

• cd .. - takes to the parent directory
• cd / - takes to the root directory

Fundamentals of 42
 File-Related Commands

File Operation Command

Copying a file cp

Moving a file mv

Removing a file rm

Displaying a file cat

and concatenating files

Fundamentals of 43
 Command - cp

Used to copy files across directories

Syntax
cp <source file> <new file name>

Example
cp file1 file2

Fundamentals of 44
 Command - cp

Options to cp
• -p
• Copies the file and preserves the following attributes
• owner id
• group id
• permissions
• last modification time
• -r
• recursive copy; copy subdirectories under the directory if
any
• -i
• interactive; prompts for confirmation before overwriting
the target file, if it already exists

Fundamentals of 45
 Command - mv

Used to move a file, or rename a file

Preserves the following details

• owner id
• group id
• permissions
• Last modification time

-f suppresses all prompting (forces overwriting of target)

-i prompts before overwriting destination file

Fundamentals of 46
 Command - rm

Used to remove a file

• Syntax : rm file(s)

-f suppresses all prompting

-i prompts before deleting destination file

-r will recursively remove the file from a directory (can be

used to delete a directory along with the content )

Caution: Use “i” option along with “r” to get notified on

deletion

Fundamentals of 47
 Command – chown & chgrp
$ ls –l
-rwxr-xr-x 1 user1 training 12373 Dec 15 14:45 a.out
-rwxr-xr-x 3 user1 faculty 4096 Dec 24 11:56 awkpro
$chown user2 a.out
$ls –l
-rwxr-xr-x 1 user2 training 12373 Dec 15 14:45 a.out
-rwxr-xr-x 3 user1 faculty 4096 Dec 24 11:56 awkpro
$ chgrp training awkpro
$ls –l
-rwxr-xr-x 1 user2 training 12373 Dec 15 14:45 a.out
-rwxr-xr-x 3 user1 training 4096 Dec 24 11:56 awkpro

Fundamentals of 48
 Command - umask
umask value is used to set the default permission of a file and
directory while creating

umask command is used to see the default mask for the file
permission

Default umask value will be set in the system environment file

like /etc/profile

umask 022 will set a mask of 022 for the current session
– The file permission after setting this umask value will be
644
– And the directory permission will be 755

Fundamentals of 49
 Command - ln
Linking files

• Hard Link (in the same filesystem)

• $ ln /usr/bin/clear /usr/bin/cls

• Hard link uses the same inode number

• Soft Link (in different filesystems also used to link

directories)
• $ ln –s /usr/bin/clear /home/user1/cls

Fundamentals of 50
 Special Permission Bits
• Set user ID (SUID)
• This means that if the SUID bit is set for any application then
your user ID would be set as that of the owner of application/file
rather than the current user, while running that application

• “set user ID” bit can be set in one of the two ways:
• chmod u+s <filename>

• chmod 4755 <filename>

• The leftmost octal number 4 indicates “set user ID” bit to be set, other octal
digits indicate regular file permissions. This is meaningful for executable
files only.

Fundamentals of 51
 Special Permission Bits
• Set group id (SGID)
• Just like SUID, setting the SGID bit for a file sets your group
ID to the file's group while the file is executing
• SGID set to a directory ; all files created in the directory will
belong to the named group
• “set group ID” bit can be set in one of the two ways:
• chmod g+s <filename>

• chmod 2755 <filename>

• The leftmost octal number 2 indicates “set group ID” bit to be set, other
octal digits indicate regular file permissions. This is meaningful for
executable files only.

Fundamentals of 52
 Special Permission Bits
• Sticky bit (SVTX)
• Typically set to a directory that is shareable
• Any user can create a file in such sharable directory
• Only owner of the file or super user (root) can remove a file
from the directory

• “sticky” bit can be set in one of the two ways:

• chmod +t <directoryname>

• chmod 1555 <directoryname>

• The leftmost octal number 1 indicates “sticky” bit to be set, other octal
digits indicate regular file permissions.

Fundamentals of 53
 Vi Editor
• vi is a visual editor used to create and edit text files.
• A screen-oriented text editor
• Included with most UNIX system distributions
• Command driven
• Categories of commands include
• Cursor movement
• Editing commands
• Search and replace commands

• The vi editor is invoked by the following command:

$ vi filename

Fundamentals of 54
 Navigation

Backspace Space

h j k l

the quick brown fox the quick brown fox

w w w $
the quick brown fox the quick brown fox

2 w ^
the quick brown fox

b b b

Fundamentals of 55
 Editing Commands
• Text insertion / replacement

• i - inserts text to the left of the cursor

• a - inserts text to the right of the cursor
• I - inserts text at the beginning of the line
• A - appends text at end of the line
• o - opens line below
• O - opens line above
• R - replaces text from cursor to right
• s - replaces a single character with any number of
characters
• S - replaces entire line

Fundamentals of 56
 Editing Commands

• Deletion

• x - to delete character at cursor position

• 3x - to delete 3 characters at cursor position
• dw - to delete word
• 2dw - to delete 2 word
• dd - to delete a line
• 2dd - to delete 2 lines

Fundamentals of 57
 Editing Commands
• Yanking
• Y - copy line into buffer
• 3Y - copy 3 lines into buffer
• p - copy buffer below cursor
• P- copy buffer above cursor

• Save and quit

• :w - to save
• :w! - to name a file (:w! filename -> save as)
• :x - save and quit
• :q - cancel changes
• :q! - cancel and quit

Fundamentals of 58
 Search & Replace Commands
The following commands are applicable for vi editor in Linux

/pat searches for the pattern pat and places cursor

where pattern occurs.

/ repeat last search

:%s/old/new/g to change every occurrence in the whole file.

:#,#s/old/new/g where #,# are replaced with the numbers of

the two lines.

Fundamentals of 59
 Summary
• In this session, you have learned how to …
• use file permissions using the chmod command
• use directory-related commands namely mkdir, rmdir,
cd commands
• use file-related commands namely cp, mv, rm
commands
• access advanced file permissions using commands
umask, suid, sgid, linking the files, stickybit
• create and edit files using the vi editor

Fundamentals of 60
 Chapter 4

UNIX Utilities

Fundamentals of 61
 Objectives

In this session, you will learn how to:

• use the Unix utilities such as
• cat, echo, touch, more, file, wc, cmp, comm, find
• employ redirection operators
• use filters such as
• sort, grep, cut, head, tail, tr, and paste
• use communication commands
• telnet, ftp
• use backup commands
• zip/gzip and tar

Fundamentals of 62
 cat
• cat command takes the input from the keyboard, and sends the
output to the monitor

• We can redirect the input and output using the redirection operators

$ cat > file1

Type the content here
press <ctrl d>
$ cat file1
Displays the content of the file
$cat >> file1
This will append standard input to the content of file1

Fundamentals of 63
 touch
• touch is used to change the time stamp of the file

Syntax:
touch [options] file

• Options:
• -a to change the access time
• -m to change the modification time
• -c no create if not exists

• touch <file> will change the time of change of the file if the file exists

• If the file does not exist, it will create a file of zero byte size.

Fundamentals of 64
 echo & read
• echo command is used to print output to the screen
echo “This is an example”
This is an example

x=10
echo $x
10

• read command allows to read input from user and assign it to

the variable specified.
read x

Fundamentals of 65
 General Purpose Utilities

• more
• Allows user to view one page-full of information at a time.
• file
• Used to display the type of the file
• tty
• Prints the terminal’s name

Fundamentals of 66
 General Purpose Utilities

• wc
• A filter used to count the number of lines, words, and characters in a
disk file or from the standard input.
• -l - displays the number of lines
• -w - displays the number of words
• -c - displays the number of characters

Fundamentals of 67
 General Purpose Utilities

• cmp
• Returns the offset and the line number of the first position where the
two files differ.

• comm
• col1 - unique lines of first file
• col2 - unique lines of second file
• col3 - common lines

Fundamentals of 68
 General Purpose Utilities

• diff
• Indicate the differences between the files
• a Lines added
• d Lines deleted
• c Lines changed

Fundamentals of 69
 find
• Lets user to search set of files and directories based on various criteria
• Syntax: find [path...] [expression]
• [path]
• where to search
• [expression]
– What type of file to search (specified with –type option)
– What action to be applied (–exec, –print, etc.)
– Name of the files (specified as part of –name option, enclosed in “ “)
• Example
find . –name “*.c” -print

lists all files with .c extension from the current dir & its subdirectories

Fundamentals of 70
 find

• Finding files on the basis of file size

• – size [+ –]n[bc]

n represents size in bytes (c) or blocks (b) of 512 bytes

find . –size 1000c lists all files that are exactly 1000 bytes in size
find . –size +1000c lists all files that are more than 1000 bytes in size
find . –size –1000c lists all files that are less than 1000 bytes in size

Fundamentals of 71
 find

• Finding files on the basis of access time (atime) or modified

time (mtime)
• – atime [+-]n
• – mtime [+-]n

n represents number of days ( actually 24 * n hours)

find . –atime 2 lists files accessed exactly 2 days ago

find . –atime +2 lists files accessed more than 2 days ago

find / –mtime –2 lists files modified less than 2 days ago

Fundamentals of 72
 find

• Applying a command on files matching the criteria with –exec

and –ok options
• – exec command {} \;

command is command to be applied on the matching files (does not prompt user)

find . -name “*.dat” –exec ls –l {} \;

Long listing of all files with .dat extension in the current and its subdirectories
• -ok command {} \;

Functionality is similar to –exec, but prompts user before applying the command
on the file matching the criteria.

Fundamentals of 73
 pr

• pr
• Used to display a file in a format to be printed.
• Breaks up a file into pages with a header, text and footer
area

• Options
• -l to alter the length of the file
• -h to set the header
• -t to suppress the header and the footer
• -n to set the line number

Fundamentals of 74
 Standard Files

• Standard Input file

• Keyboard, file descriptor is 0

• Standard Output file

• Monitor, file descriptor is 1

• Standard Error file

• Monitor, file descriptor is 2

Fundamentals of 75
 I/O Redirection

< file redirect standard input from file

> file redirect standard output to file
2> file redirect standard error to file
2>&1 merge standard error with standard output
$ cat > abc

$ ls –l > outfile

$ cat xyz abc > outfile 2> errfile

$ cat xyz abc > outfile 2>&1

Fundamentals of 76
 Filters

• Filters are programs that takes its input from the standard
input file, process it, and sends it to the standard output file.

• Commonly used filter commands

• sort
• grep
• cut
• head
• tail
• paste

Fundamentals of 77
 sort

Sorts the contents of the given file based on the first char of
each line.
-n numeric sort (comparison made
according to strings numeric value)
-r reverse sort
-t specify delimiter for fields
+num specify sorting field numbers
+num [-num] to specify the range

Fundamentals of 78
 grep

• grep -Global Regular Expression Printer is used for

searching regular expressions
• Syntax
• grep <options> <pattern> <filename(s)>

Fundamentals of 79
 grep options

-c displays count of the number of occurrences

-n displays line numbers along with the lines

-v displays all lines except lines matching pattern

-i Ignores case for matching

Fundamentals of 80
 Patterns
* - matches 0 or more characters

[^pqr] - Matches a single character which is not p ,q or r

^pqr -Matches pqr at the beginning of the line

pqr$ -Matches pqr at the end of the line

“.” - Matches any one character

\ - ignores the special meaning. grep “New\[abc\]” filename

Fundamentals of 81
 Filter Command - head

Displays the first n lines of the file

$ head -3 file1

Fundamentals of 82
 Filter Command - tail

Displays the last n lines of a file

$ tail -3 file1

Can also specify the line number from which the data has to be
displayed till the end of file

$ tail +5 file1

Fundamentals of 83
 Filter command - tr

tr - translate filter used to translate a given set of characters

Example :
tr [a-z] [A-Z] < filename

This converts standard input read from lower case to upper case.

option -s can be used to squeeze the repeated characters.

Fundamentals of 84
 Filter command - tr

Useful options for tr

• -s char
Squeeze multiple contiguous occurrences of the character into single char

• -d char
Remove the character

Fundamentals of 85
 Command Piping

• Allows the output (only the standard output) of a command

to be sent as input to another command.

• Multiple pipes may appear in one command line.

Example:

$ cat * | wc

$ cat fil1 | head | wc -l

Fundamentals of 86
 Filter Command – tee

• tee command allows the normal output to the standard

output, as well as to a file

• Useful to capture intermediate output of a long command

pipeline for further processing, or debugging purpose.

• Example
• who | tee userlist
• cat - | tee file1 | wc -l

Fundamentals of 87
 Filter Command – cut

Used to extract specified columns of a text

Option remark
-c used to extract characters
-d Delimiter for fields
-f Field no.
Examples
$ cut -c2-5 file1
$ cut -d “|” -f2,3 file1

Fundamentals of 88
 Filter Command – paste

Paste is used to fix two cut portions of the file vertically

-s Pastes the contents of file2 below file1

-d Specify delimiter

$ paste -d”|” file1 file2

Fundamentals of 89
Connecting to Remote Systems - telnet

telnet hostname
or
telnet <ip address>

Fundamentals of 90
 ftp

• Ftp is a file transfer program

• Provides necessary user interface to the standard File Transfer
Protocol
• Allows users to transfer files to and from a remote host

• Syntax
$ ftp hostname

Fundamentals of 91
 ftp - commands

• Ftp program supports the following commands

• get receive file from host

• mget receive multiple files from host
• put send file to host
• mput send multiple files from host

Fundamentals of 92
 ftp - commands

• Ftp program supports the following commands as well

• ls list directory of host

• cd change directory on the host
• lcd change directory on the local machine

• To set transfer format

• ascii set to ascii mode
• binary set to binary mode

Fundamentals of 93
 ftp - commands

• Progress indication of transfer

• hash command

• Quitting ftp session

• bye command

Fundamentals of 94
 Compression Utilities

gzip,
Usage is very similar to compress and pack utilities in Unix:
gzip [-vc] filename

where -v displays the compression ratio.

-c sends the compressed output to standard output and leaves
the original file intact.

gunzip
gunzip can uncompress files originally compressed with compress.

Fundamentals of 95
 Tape Archive - tar

• Tar is an archiving utility to store and retrieve files from an

archive, known as tarfile.

• Though archives are created on a tape, it is common to have

them as disk files as well.

– tar c|t|x [vf destination] source...

Fundamentals of 96
 Tape Archive - tar

Examples:
Create a new tar file containing all .dat files (assuming a.dat, b.dat and c.dat
exist)
$ tar –cf mytar *.dat

Fundamentals of 97
 Summary
• In this session, you have learned to:
• use the Unix Utilities like
• cat, echo, touch, more, file, wc, cmp, comm, find
• employ redirection operators
• use filters like
• sort, grep, cut, head, tail, tr, and paste
• communication commands
• telnet, ftp
• backup commands
• zip/gzip and tar

Fundamentals of 98
Chapter 5

Process

Fundamentals of 99
 Objectives
In this session, you will learn to:
• Use process-related commands like
• ps, kill, sleep
• Start a background process
• Use background and foreground-related commands like
• bg, fg, jobs , nice , nohup

Fundamentals of 100
 Processes

• Process - a program in execution

• When program is executed, a new process is created

• The process is alive till the execution of the program is

complete

• Each process is identified by a number called pid

Fundamentals of 101
 Login shell

As soon as the user logs in, a process is created which executes the login
shell.

Login shell is set for each login in /etc/passwd file.

Fundamentals of 102
 ps

• The ps command is used to display the characteristics of a process

• It fetches the pid, tty, time, and the command which has started the
process.
• -f lists the pid of the parent process also.
• -u lists the processes of a given user
• -a lists the processes of all the users
• -e lists all the processes including the system
processes

Fundamentals of 103
 Background Process

• Enables the user to do more than one task at a time.

• If the command terminates with an ampersand (&), UNIX executes the
command in the background
• Shell returns by displaying the process ID (PID) and job id of the
process

Fundamentals of 104
 Background Process

• nohup
– Lets processes to continue to run even after logout
– The output of the command is sent to nohup.out if not redirected

$ nohup command args

$ nohup sort emp.lst &

[1] 21356
nohup: appending output to `nohup.out'

Fundamentals of 105
 Background Process

• wait command
– can be used when a process has to wait for the output of a
background process
– The wait command, can be used to let the shell wait for all
background processes terminate.
$ wait
– It is possible to wait for completion of one specific process as well.

Fundamentals of 106
 Controlling Background Processes

• jobs
• List the background process
• fg % <job id>
• Runs a process in the foreground
• bg %<job id>
• Runs a process in the background

Fundamentals of 107
 Process priority

• nice
• Used to reduce the priority of jobs

Fundamentals of 108
 The kill Command
• kill: Kills or terminates a process

• kill command send a signal to the process

• The default signal is 15 ( SIGTERM)

• kill -9 (SIGKILL)
• Terminates the process abruptly

Fundamentals of 109
 Summary
• In this session, you learned to:
• Define a process
• Use process-related commands like
• ps, kill, sleep
• Start a background process
• Use background and foreground-related commands like
• bg, fg, jobs

Fundamentals of 110
 Chapter 6

UNIX Shell Programming

Fundamentals of 111
 Objectives
• In this session, you will learn to:
• Use Shell variables
• Write scripts to process positional parameters
• Use “test” command
• Use “if” construct
• Use “for” loop
• Use “while” loop
• Use “case” construct
• Define and use functions
• Debug shell scripts

Fundamentals of 112
 Flavours of the Unix shell

• Bourne shell sh

• C shell csh

• Korn shell ksh

• Bourne again shell bash

(shell distributed with linux)

Fundamentals of 113
 Command processing

• Displays the shell prompt and reads the command typed by the
user.

• Interprets the command and classifies it as an internal (built-in),

or an external command.

• If it is NOT a built-in command, searches for the command in the

PATH-specified directories, and executes that command if it is
found.

Fundamentals of 114
 Shell Features

Parent shell process $ vi test.c

(bash) command typed by
fork user
Child shell process
Exec of “vi test.c”
(bash) User
Mode
Kernel
Mode

Fundamentals of 115
 Additional Shell Features

• Each shell, apart from the basic features, provides

additional features such as:

• Maintaining command history (C, korn and bash)

• Renaming (aliasing) a command (C, korn, bash)

• Command editing (C, korn and bash)

• Programming language (all shells)

Fundamentals of 116
 History

• some UNIX shells support command history

• facility to keeps track of commands that were executed
• facility to rerun previously executed commands
• bash shell supports the following

!! recall the last command and execute it.

!num execute nth command where n is the

the num specified after !

Fundamentals of 117
 alias

• alias can be used to give new name to an existing command

• A better name that represents a single command or a sequence of
commands to be executed, often with appropriate options
• alias is an internal command

alias newname=command

$ alias l=‘ls –l’

The unalias command cancels previously defined alias.

Fundamentals of 118
 File name substitution

• When the user enters a command string, the shell parses the string
into following components:
• Command (the first part of the string, till the first space char)
• Command arguments (the subsequent parts of the string)

• For example, given the command-string “ls –l *.c”, this string

contains the “ls” command and two arguments “-l” and “*.c”.

Fundamentals of 119
 File name substitution

• In arguments of a command, the shell recognizes certain

characters – such as *, ?, [ ], and - as special characters
and expands these characters into a filename list before
executing the command.

• To see how this works, enter following commands while in

/bin directory

$ ls a*
$ ls ??

Fundamentals of 120
 Shell Programming
• Allows
• Defining and referencing variables
• Logic control structures such as if, for, while, case
• Input and output

Fundamentals of 121
 Shell Variables
• A variable is a name associated with a data value, and it offers a
symbolic way to represent and manipulate data variables in the
shell. They are classified as follows
• user-defined variables
• environment variables
• predefined variables

• value assigned to the variable can then be referred to by preceding

the variable name with a $ sign.

Fundamentals of 122
 Shell Variables

• The shell provides the facility to define normal, and

environment variables.

• A normal variable can be only used in the shell where it is

defined.

• An environment variable can be used in the shell where it

is defined, plus any child shells invoked from that shell.

Fundamentals of 123
 Using Normal Variables
To define a normal variable, use the following syntax:
• variable_name=value

Examples:
x=10
textline_1=‘This line was entered by $USER’
textline_2=“This line was entered by $USER”
allusers=`who`
usercount=`who | wc –l`

Fundamentals of 124
 Using Normal Variables

• Once variables are defined, one can use the echo command
to display the value of each variable:
echo $x
echo $textline_1
echo $textline_2
echo $allusers
echo $usercount

Fundamentals of 125
 Using Environment Variables

• To define an environment variable, use following syntax:

variable_name=value
export variable_name

• Examples:
• $ x=10; export x
• $ allusers=`who` ; export allusers

Fundamentals of 126
 Built-in environment variables

• PATH • MAIL
• BASH_ENV • USER
• HOME • LOGNAME
• PWD • PS1
• SHELL • PS2
• TERM

Fundamentals of 127
 Sample Shell Script
#! /bin/bash
# The above line has a special meaning. It must be the
# first line of the script. It says that the commands in
# this shell script should be executed by the bash
# shell (/bin/bash).
# ---------------------------------------------------------------
echo “Hello $USER….”
echo “Welcome to programming shell scripts..”
# ---------------------------------------------------------------

Fundamentals of 128
 Executing Shell Scripts

There are two ways of executing a shell script:

By passing the shell script name as an argument to the shell.

For example:

sh script1.sh

If the shell script is assigned execute permission, it can be

executed using it’s name. For example:

./script1.sh

Fundamentals of 129
 Passing Parameters to Scripts
• parameter can be passed to a shell script

• parameters are specified after the name of the shell script

when invoking the script.

• Within the shell script, parameters are referenced using the

predefined variables $1 through $9.

• In case of more than 9 parameters, other parameters can be

accessed by shifting.

Fundamentals of 130
 Built-in variables
• Following are built-in variables supported
• $0, $1…$9 - positional arguments
• $* - all arguments
• $@ - all arguments
• $? - exit status of previous command executed
• $$ - PID of the current process
• $! - PID of the last background process

Fundamentals of 131
 Passing Parameters to Scripts

• Consider following shell script:

----------------------script2.sh--------------------------
echo “Total parameters entered: $#”
echo “First parameter is : $1”
echo “The parameters are: $*”
shift
echo “First parameter is : $1”
------------------------------------------------------------

• Execute the above script using the “script2.sh these are the
parameters” command.

Fundamentals of 132
 Passing Parameters to Scripts

• The shell parameters are passed as strings.

• to pass a string containing multiple words as a single

parameter, it must be enclosed within quotes.

• For example,

$ ./script2.sh “this string is a single parameter”

Fundamentals of 133
 Doing Arithmetic Operations

• Arithmetic operations within a shell script can be

performed using expr command.

• Example,
x=10
y=5
number_1 = `expr $x + $y`
number_2 = `expr $x - $y`
number_3 = `expr $x / $y`
number_4 = `expr $x \* $y`
number_5 = `expr $x % $y`

Fundamentals of 134
 Using the test Command

The general syntax of test command is:

test expression

• The expression can be formed using a combination of shell

variables and the operators supported by the test
command. These operators provide facility to compare
numbers, string and logical values, file types and file
access modes.

Fundamentals of 135
 Using the test Command

To compare two integers using test following operators are

available:

-eq (equal to)

-ne (not equal to)
-lt (less than)
-le (less than or equal to)
-gt (greater than)
-ge (greater than or equal to)

Fundamentals of 136
 Using the test Command

• General syntax

test expression
or
[ expression ]

test integer1 operator integer2

OR
[ integer1 operator integer2 ]

Fundamentals of 137
 Using the test Command

To compare two strings using the test command, following

operators are available:

• string1 = string2 (equal to, please note it is a single =)

• string1 != string2 (not equal to)
• string1 (string is not NULL)
• -n string1 (string is not NULL and exists)
• -z string1 (string is NULL and exists)

Fundamentals of 138
 Using the test Command

The syntax for this string comparison is:

test string1 operator string2

OR
[ string1 operator string2 ]
OR
test operator string
OR
[ operator string ]

Fundamentals of 139
 Using the test Command

To check a file type/access permissions using the test

command, following operators are available:

• -s file (file is not empty and exists)

• -f file (Ordinary file and exists)
• -d file (file is a directory and exists)
• -r file (file is readable and exists)
• -w file (file is write-able and exists)
• -x file (file is executable and exists)

Fundamentals of 140
 Using the test Command
To check a file type/access permissions using the test
command, following operators are available:

• -b file (file is a block device and exists)

• -c file (file is a character device and exists)
• -p file (file is a named pipe and exists)
• -g file (file has sticky bit set)
• -u file (file has setuid bit set)
• -t file_des (file descriptor is standard output)

Fundamentals of 141
 Combining Conditions

• It is possible to combine conditions by using following

operators:

• -a (logical AND operator)

• -o (logical OR operator)
• ! (logical NOT operator)

Fundamentals of 142
 Combining Conditions
The syntax for this is:

test expression_1 –a expression _2,

OR
[ expression _1 –a expression _2 ]

test expression_1 –o expression _2,

OR
[ expression_1 –o expression_2 ]

test ! expression _1
OR
[ ! expression _1 ]

Fundamentals of 143
 Condition Checking in Scripts

Bash shell provides the if command to test if a condition is

true. The general format of this command is:

if condition
then
command
fi

The condition is typically formed using the test command.

Fundamentals of 144
 Example

# to check if the current directory is the same as your home

directory
curdir=`pwd`
if test “$curdir” != “$HOME”
then
echo your home dir is not the same as your pesent
working directory
else
echo $HOME is your current directory
fi

Fundamentals of 145
 Checking Multiple Conditions

The complex form of if statement is as follows:

if condition_1
then
command
elif condition_2
then
command
else
command
fi

Fundamentals of 146
 Using for Loop

The Bash shell provides a for loop.

Example:
The syntax of this loop is:
for i in 1 2 3 4 5
for variable in list do
do echo -n $i \* $i = " "
command echo `expr $i \* $i `
…
done
command
done

Fundamentals of 147
 Example

----------------------script.sh--------------------------
#! /bin/sh
usernames=`who | cut –d “ “ –f1`
echo “Total users logged in = $#usernames”
#
for user in ${usernames}
do
echo $user
done
------------------------------------------------------------

Fundamentals of 148
 Using while Loop

The Bash shell provides a while loop. The syntax of this loop
is:

while condition
do
command
…
command
done

Fundamentals of 149
 Example

Shell script checks for a blank/non blank string

eg: read nam
while [ -z “$nam” ]
do
read nam
done
echo the string is $nam

the above piece of code keeps accepting string variable nam

until it is non zero in length.

Fundamentals of 150
 Example
Shell script to compute factorial of a given number

#!/bin/bash
n=$1
if [ $n -eq 0 ]; then
fact=0
else
fact=1
while [ $n –ne 0 ]
do
fact=`expr $fact \* $n`
n=`expr $n – 1`
done
fi
echo $fact

Fundamentals of 151
 The case Statement
The structure of case statement
case value in
pattern1)
command
command;;
pattern2)
command
command;;
patternn)
command;;
esac

Fundamentals of 152
 Example

#!/bin/bash (contd.)
echo enter 2 nos
read num1 case $choice in
read num2 1) res=`expr $num1 + $num2`
echo “enter 1 (for addition) or echo result is $res;;
2 (for subtraction) 1) res=`expr $num1 - $num2`
read choice echo result is $res;;
*) echo invalid input;;
esac

Fundamentals of 153
 Example
#!/bin/bash
read number
case $number
1) echo 1st
break;;
2) echo 2nd
break;;
3) echo 3rd
break;;
*) echo ${number}th
break;;
esac

Fundamentals of 154
 Functions

Shell functions are a way to group commands for later

execution using a single name for the group. They are
executed just like a "regular" command.

Shell functions are executed in the current shell context; no

new process is created to interpret them. Functions are
declared using this syntax:

[ function ] name () { command-list; }

Fundamentals of 155
 Functions

• Shell functions can accept arguments

• Arguments are passed in the same way as given to
commands
• Functions refer to arguments using $1, $2 etc., similar to
the way shell scripts refer to command line arguments

Fundamentals of 156
 Functions

Function to convert standard input into upper case

toupper()
{
tr “[a-z]” “[A-Z]”
}

This function can be used as

$ cat abc | toupper

Fundamentals of 157
 Debugging Shell Scripts

• Two options help in debugging shell scripts

– “-v” (verbose) option:
causes shell to print the lines of the script as they are read.

$ bash –v script-file

– “-x” (verbose) option:

prints commands and their arguments as they are executed.

$ bash –x script-file

Fundamentals of 158
 Programming in C vs Shell

Comparison between
• A solution in C
• A shell solution written like a C program
• A proper “shell/unix” solution
e.g:
The problem is to count the no of lines in a file ( the file is
called the_file)

Fundamentals of 159
A solution in C A shell solution
#include <stdio.h> count=0
void main(void) while read line
{ do
int lcount=0; count=`expr $count + 1`
FILE *infile; done < the_file
char line[500];
infile=fopen("the_file","r"); echo Number of lines is $count
while(!feof(infile))
{ Solution using existing
fgets(line,500,infile); commands
lcount ++;
}
$ wc –l the_file
printf("no of lines is %d\n",lcount);
}
Fundamentals of 160
 Summary
• In this session, you have learned to:
• Use Shell variables
• Write scripts to process positional parameters
• Use “test” command
• Use “if” construct
• Use “for” loop
• Use “while” loop
• Use “case” construct
• Define and use functions
• Debug shell scripts

Fundamentals of 161
Advanced Scripting in Unix

Egrep, SED, AWK

Fundamentals of 162
 Advanced Filters
• egrep, fgrep
• sed
• awk

Fundamentals of 163
 egrep , fgrep
• Egrep
– Extended Grep used to search for more than one patterns at a time
– used as egrep ‘pattern1|pattern2’ filename
– Options used
• n - prints the line numbers
• c - count of lines matching the pattern
• v - inverses the output of the search
• fgrep - similar to the egrep command
– Alternate patterns are seperated by a new line character.
– Usage:
• $fgrep ‘a
• >b ‘ filename

Fundamentals of 164
 SED
 sed is a stream editor

 Reading lines using sed

Examples:
sed ‘4q’ file read first 4 lines of the file and quit
sed –n ‘1,4p’ file print line 1 to 4
sed –n ‘1,2!p’ file print the expect the first two lines
sed ‘G’ file adds a blank line after each line
sed ‘4G’ file adds a blank line after 4th line.

Fundamentals of 165
 “ sed- substitution ”

 “s” for substitution

Syntax:
– sed 's/old/new/' <file_ip >file_op

– There are four parts to this substitute command:

• s Substitute command
• /../../ Delimiter
• old Regular Expression Pattern String
• New Replacement string

Fundamentals of 166
 “sed- substitution ”
 If you want to change a pathname that contains a slash, you could use
the backslash to quote the slash
– sed 's/\/usr\/local\/bin/\/common\/bin/' <file_ip >file_op

– sed 's_/usr/local/bin_/common/bin_' <file_ip >file_op

# substitute (find & replace) "foo" with "bar" on each line

sed 's/foo/bar/' # replaces only 1st instance in a line
sed 's/foo/bar/4' # replaces only 4th instance in a line
sed 's/foo/bar/g' # replaces ALL instances in a line

# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'

# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
sed '/baz/!s/foo/bar/g'

Fundamentals of 167
 “sed –n /p /w”

sed ‘/root/p’ /etc/passwd (note line with “root” written twice)

 "sed -n," argument will not, by default, print unmodified lines.
sed –n ‘/root/p’ /etc/passwd (note difference w.r.t above command)

 When the "-n" option is used, the "p" flag will cause the modified line to
be printed.
• sed –n ‘/root/p’ /etc/passwd (acts as grep)

 With /w filename you can specify a file that will receive the modified data
• sed –n ‘1,4pw /tmp/op_file’ /in_file (copies first four lines from in_file to op_file)

Fundamentals of 168
 “sed multiple commands”
 Below command replaces “red” with “RED” and “white” with “WHITE”
• sed –n ‘s/red/RED/gp’ test | sed ‘s/white/WHITE/g’

• sed –e ‘s/red/RED/p’ –e ‘s/white/WHITE/q’ test

 sed –f scriptname:
Syntax: sed –f sedscript < ip_file

 If you have (multiple) large number of sed commands you can create
sed command file and execute with –f option
Example: Edit a file “sedcmd” and add the lines below
s/red/RED/g
s/white/WHITE/g
and execute it as follows
sed –f sedcmd <test_ip >test_op

Fundamentals of 169
 “sed”
 you could remove the encrypted password from the password file
( “:” acts as delimiter for input stream)
sed 's/[^:]*//2' </etc/passwd >/etc/password.new

 If you wanted to add a colon after the 80th character in each line,
you could type ( “.” for every character)
sed 's/./&:/80' <file >new

 If you want remove the comment lines

sed ‘/^#/d’ filename

 To count number of lines in a file

sed –n ‘$=‘ filename

 With “&” you can put the string you found in the replacement
string
sed -n 's/pattern/&/p' <file ( acts as grep )

Fundamentals of 170
 “quoting sed in script”
 Example: Reverse the order of the lines in the file and removes
blank lines
#! /bin/ksh
sed ‘
# 1!G;h;$!d
/./!d’ < filename

Fundamentals of 171
 “sed adding records”
 Records can be appended, inserted, deleted as below
Example1:
sed ‘a\
LINE 1 ADDED\
LINE2 ADDED’ test

Example2:
sed `/orange/ a\
ABOVE LINE HAS orange IN IT ‘ test

sed ‘3i\
Insert a line at line 3\
‘ test

Fundamentals of 172
 “sed deleting records”
Example:
sed ‘2d’ file: delete 2nd line of the file
sed ‘1,4d’ file: delete first 4 lines of the file
sed ‘1,4!d’ file: delete all other lines except first four
sed ‘s/^[ \t]*//’: del leading spaces and tabs at the front of each line
sed ‘s/[ \t]*$//’: del trailing spaces and tabs at the end of each line

Fundamentals of 173
 AWK

Fundamentals of 174
 Review of Awk Principles
• Awk’s purpose: to give Unix a general purpose
programming language that handles text (strings) as easily
as numbers
– This makes Awk one of the most powerful of the Unix utilities
• Awk process fields while ed/sed process lines
• nawk (new awk) is the new standard for Awk
– Designed to facilitate large awk programs
• Awk gets it’s input from
– files
– redirection and pipes
– directly from standard input

Fundamentals of 175
 History
• Originally designed/implemented in 1977 by Al Aho, Peter
Weinberger, and Brian Kernigan
– In part as an experiment to see how grep and sed could be
generalized to deal with numbers as well as text
– Originally intended for very short programs
– But people started using it and the programs kept getting bigger
and bigger!
• In 1985, new awk, or nawk, was written to add
enhancements to facilitate larger program development
– Major new feature is user defined functions

Fundamentals of 176
 nawk
• Other enhancements in nawk include:
– Dynamic regular expressions
• Text substitution and pattern matching functions
– Additional built-in functions and variables
– New operators and statements
– Input from more than one file
– Access to command line arguments
• nawk also improved error messages which makes
debugging considerably easier under nawk than awk
• On most systems, nawk has replaced awk
– On ours, both exist

Fundamentals of 177
 Running an AWK Program
• There are several ways to run an Awk program
– awk ‘program’ input_file(s)
• program and input files are provided as command-line arguments
– awk ‘program’
• program is a command-line argument; input is taken from standard
input (yes, awk is a filter!)
– awk -f program_file_name input_files
• program is read from a file

Fundamentals of 178
 Awk as a Filter
• Since Awk is a filter, you can also use pipes with other
filters to massage its output even further
• Suppose you want to print the data for each employee
along with their pay and have it sorted in order of
increasing pay

awk ‘{ printf(“%6.2f %s\n”, $2 * $3, $0) }’ emp.data | sort

Fundamentals of 179
 Errors
• If you make an error, Awk will provide a diagnostic error message
awk '$3 == 0 [ print $1 }' emp.data
awk: syntax error near line 1
awk: bailing out near line 1
• Or if you are using nawk
nawk '$3 == 0 [ print $1 }' emp.data
nawk: syntax error at source line 1
context is
$3 == 0 >>> [ <<<
1 extra }
1 extra [
nawk: bailing out at source line 1
1 extra }
1 extra [

Fundamentals of 180
 Structure of an AWK Program
• An Awk program consists of:
– An optional BEGIN segment BEGIN{action}
• For processing to execute prior to
reading input pattern {action}
– pattern - action pairs
pattern {action}
• Processing for input data
• For each pattern matched, the .
corresponding action is taken
– An optional END segment .
• Processing after end of input data
.
pattern { action}
END {action}

Fundamentals of 181
 BEGIN and END
• Special pattern BEGIN matches before the first input line
is read; END matches after the last input line has been read
• This allows for initial and wrap-up processing
BEGIN { print “NAME RATE HOURS”; print “” }
{ print }
END { print “total number of employees is”, NR }

Fundamentals of 182
 Pattern-Action Pairs

• Both are optional, but one or the other is required

– Default pattern is match every record
– Default action is print record
• Patterns
– BEGIN and END
– expressions
• $3 < 100
• $4 == “Asia”
– string-matching
• /regex/ - /^.*$/
• string - abc
– matches the first occurrence of regex or string in the record

Fundamentals of 183
 Selection

• Awk patterns are good for selecting specific lines from the
input for further processing
• Selection by Comparison
– $2 >=5 { print }
• Selection by Computation
– $2 * $3 > 50 { printf(“%6.2f for %s\n”, $2 * $3, $1) }
• Selection by Text Content
– $1 == “Susie”
– /Susie/
• Combinations of Patterns
– $2 >= 4 || $3 >= 20

Fundamentals of 184
 Data Validation
• Validating data is a common operation
• Awk is excellent at data validation
– NF != 3 { print $0, “number of fields not equal to 3” }
– $2 < 3.35 { print $0, “rate is below minimum wage” }
– $2 > 10 { print $0, “rate exceeds $10 per hour” }
– $3 < 0 { print $0, “negative hours worked” }
– $3 > 60 { print $0, “too many hours worked” }

Fundamentals of 185
 Regular Expressions in Awk
• Awk uses the same regular expressions we’ve been using
– ^$ beginning of/end of field
– . any character
– [abcd] character class
– [^abcd] negated character class
– [a-z] range of characters
– (regex1|regex2) alternation
– * zero or more occurrences of preceding expression
– + one or more occurrences of preceding expression
– ? zero or one occurrence of preceding expression

Fundamentals of 186
 Awk Variables
• $0, $1, $2, … ,$NF
• NR - Number of records read
• FNR - Number of records read from current file
• NF - Number of fields in current record
• FILENAME - name of current input file
• FS - Field separator, space or TAB by default
• OFS - Output field separator, space by default
• ARGC/ARGV - Argument Count, Argument Value array
– Used to get arguments from the command line

Fundamentals of 187
 Arrays
• Awk provides arrays for storing groups of related data
values
# reverse - print input in reverse order by line
{ line[NR] = $0 } # remember each line
END { i = NR # print lines in reverse order
while (i > 0) {
print line[i]
i=i-1
}
}

Fundamentals of 188
 Operators
• = assignment operator; sets a variable equal to a value or
string
• == equality operator; returns TRUE is both sides are equal
• != inverse equality operator
• && logical AND
• || logical OR
• ! logical NOT
• <, >, <=, >= relational operators
• +, -, /, *, %, ^
• String concatenation

Fundamentals of 189
 Control Flow Statements
• Awk provides several control flow statements for making decisions
and writing loops
• If-Else
if (expression is true or non-zero){
statement1
}
else {
statement2
}
where statement1 and/or statement2 can be multiple statements enclosed in
curly braces { }s
– the else and associated statement2 are optional

Fundamentals of 190
 Loop Control
• While
while (expression is true or non-zero) {
statement1
}
• For
for(expression1; expression2; expression3) {
statement1
}
• Do While
do {
statement1
}
while (expression)

Fundamentals of 191
 Computing with AWK
• Counting is easy to do with Awk
$3 > 15 { emp = emp + 1}
END { print emp, “employees worked more than 15 hrs”}
• Computing Sums and Averages is also simple
{ pay = pay + $2 * $3 }
END { print NR, “employees”
print “total pay is”, pay
print “average pay is”, pay/NR
}

Fundamentals of 192
 Handling Text
• One major advantage of Awk is its ability to handle strings
as easily as many languages handle numbers
• Awk variables can hold strings of characters as well as
numbers, and Awk conveniently translates back and forth
as needed
• This program finds the employee who is paid the most per
hour
$2 > maxrate { maxrate = $2; maxemp = $1 }
END { print “highest hourly rate:”, maxrate, “for”, maxemp }

Fundamentals of 193
 String Concatenation
• String Concatenation
– New strings can be created by combining old ones
{ names = names $1 “ “ }
END { print names }
• Printing the Last Input Line
– Although NR retains its value after the last input line has been
read, $0 does not
{ last = $0 }
END { print last }

Fundamentals of 194
 Command Line Arguments
• Accessed via built-ins ARGC and ARGV
• ARGC is set to the number of command line arguments
• ARGV[ ] contains each of the arguments
– For the command line
– awk ‘script’ filename
• ARGC == 2
• ARGV[0] == “awk”
• ARGV[1] == “filename
• the script is not considered an argument

Fundamentals of 195
 ARGC/ARGV in Action
#argv.awk – get a cmd line argument and display
BEGIN {if(ARGC != 2)
{print "Not enough arguments!"}
else
{print "Good evening,", ARGV[1]}
}

Fundamentals of 196
BEGIN {if(ARGC != 3)
{print "Not enough arguments!"
print "Usage is awk -f script in_file field_separator"
exit}
else
{FS=ARGV[2]
delete ARGV[2]}
}
$1 ~ /..3/ {print $1 "'s name in real life is", $5; ++nr}
END {print; print "There are", nr, "students registered in your class."}

Fundamentals of 197
 getline
• How do you get input into your awk script other than on
the command line?
• The getline function provides input capabilities
• getline is used to read input from either the current input or
from a file or pipe
• getline returns 1 if a record was present, 0 if an end-of-file
was encountered, and –1 if some error occurred

Fundamentals of 198
 getline Function
Expression Sets

getline $0, NF, NR, FNR

getline var var, NR, FNR

getline <"file" $0, NF

getline var <"file" var

"cmd" | getline $0, NF

"cmd" | getline var var

Fundamentals of 199
 getline from stdin
#getline.awk - demonstrate the getline function
BEGIN {print "What is your first name and major? "
while (getline > 0)
print "Hi", $1 ", your major is", $2 "."
}

Fundamentals of 200
 getline From a File
#getline1.awk - demo getline with a file
BEGIN {while (getline <"emp.data" >0)
print $0}

Fundamentals of 201
 getline From a Pipe
#getline2.awk - show using getline with a pipe
BEGIN {{while ("who" | getline)
nr++}
print "There are", nr, "people logged on clyde right
now."}

Fundamentals of 202
 Simple Output From AWK
• Printing Every Line
– If an action has no pattern, the action is performed for all input lines
• { print } will print all input lines on stdout
• { print $0 } will do the same thing
• Printing Certain Fields
– Multiple items can be printed on the same output line with a single print
statement
– { print $1, $3 }
– Expressions separated by a comma are, by default, separated by a single
space when output

Fundamentals of 203
 Formatted Output
• printf provides formatted output
• Syntax is printf(“format string”, var1, var2, ….)
• Format specifiers
– %c – single character
– %d - number
– %f - floating point number
– %s - string
– \n - NEWLINE
– \t - TAB
• Format modifiers
– - left justify in column
– n column width
– .n number of decimal places to print

Fundamentals of 204
 printf Examples
• printf(“I have %d %s\n”, how_many, animal_type)
– format a number (%d) followed by a string (%s)
• printf(“%-10s has $%6.2f in their account\n”, name, amount)
– prints a left justified string in a 10 character wide field and a float with 2
decimal places in a six character wide field
• printf(“%10s %-4.2f %-6d\n”, name, interest_rate, account_number >
"account_rates")
– prints a right justified string in a 10 character wide field, a left justified
float with 2 decimal places in a 4 digit wide field and a left justified
decimal number in a 6 digit wide field to a file
• printf(“\t%d\t%d\t%6.2f\t%s\n”, id_no, age, balance, name >>
"account")
– appends a TAB separated number, number, 6.2 float and a string to a file

Fundamentals of 205
 Built-In Functions

• Arithmetic
– sin, cos, atan, exp, int, log, rand, sqrt
• String
– length, substitution, find substrings, split strings
• Output
– print, printf, print and printf to file
• Special
– system - executes a Unix command
• system(“clear”) to clear the screen
• Note double quotes around the Unix command
– exit - stop reading input and go immediately to the END pattern-action pair if it
exists, otherwise exit the script

Fundamentals of 206
 Built-In String Functions
Function Description

gsub(r, s) substitute s for r globally in $0, return number of

substitutions made

gsub(r, s, t) substitute s for r globally in string t, return

number of substitutions made

index(s, t) return first position of string t in s, or 0 if t is not

present

length(s) return number of characters in s

match(s, r) test whether s contains a substring matched by r,

return index or 0

sprint(fmt, expr-list) return expr-list formatted according to format

string fmt

Fundamentals of 207
 Built-In String Functions

Function Description

split(s, a) split s into array a on FS, return number of fields

split(s, a, fs) split s into array a on field separator fs, return number of
fields

sub(r, s) substitute s for the leftmost longest substring of $0 matched

by r
sub(r, s, t) substitute s for the leftmost longest substring of t matched
by r

substr(s, p) return suffix of s starting at position p

substr(s, p, n) return substring of s of length n starting at position p

Fundamentals of 208
 UNIX Bibliography

UNIX in a Nutshell for BSD 4.3: A Desktop Quick Reference For

Berkeley (O'Reilly & Associates, Inc., 1990, ISBN 0-937175-
20-X).

UNIX in a Nutshell: A Desktop Quick Reference for System V &

Solaris 2.0 (O'Reilly & Associates, Inc., 1992, ISBN 0-56592-
001-5).

The UNIX Programming Environment, Brian W. Kernighan &

Rob Pike (Prentice Hall, 1984).

Fundamentals of 209
Thank You

Fundamentals of 210