ES-eEnabling

Utilities for
2005-2006 Strategy
Debugging in Unix

Srinivas Adyapak

Wipro confidential
1
 Common Utilities for Developers in Unix

•Common Utilities required by Developers to improve their code and enhance their productivity
• These utilities will be available in most of the Linux installations.
Debugging  cscope, ctags – Code browsing
 strace (truss in Solaris) – Debugging a running process
 pstack - Stack trace of a running process
 pmap - Memory used by a process
 nm - Symbol names in an object or exe
 ddd – A GUI for Command line debugging

Process & Task  ps - Snapshot of process status

Management  top - Display task details (usage of memory, cpu..)
 time - Code performance
 kill – terminate a process

File Management  lsof - Open files and Processes opening the files
 df, du – available disk space, disk usage
Networking  netstat - displays the contents of various network related data structures
 ifconfig – configuring a network device
 ping - Sends ICMP ECHO_REQUEST packets to network hosts

• Refer to man pages for more details on each of these utilities

Wipro confidential 2
 Common Utilities for Developers in Unix

Other utilities  sar – Displays the CPU activity

 vmstat - reports virtual memory statistics of process, virtual memory, disk,
trap, and CPU activity
 iostat - reports terminal and disk I/O activity and CPU utilization

• Refer to man pages for more details on each of these utilities

Wipro confidential 3
 cscope
• Instead of doing a “grep” on a source code of few million lines, it is easier to
examine source code thro’ a code browsing tool such as cscope.

•Useful in Code browsing – locating variables, function calls, functions, macros

in files
• Human-readable cross-reference can be got by for ctags eg. ctags -x *.c

• Not limited to C –> C++, Python, Perl, hardware designed languages such
as Verilog.

• Builds the cross-references on all the source files to examine – includes .c, .h
file extensions
• cscope vsftpd-2.0.5/* - builds the cross-references for all files under the
directory vsftpd-2.0.5
• cscope rebuilds the cross-reference only if a source file has changed or
the list of source files is different.
• Can directly navigate to the occurrence of the source.
Wipro confidential 4
 cscope
 Cscope version 15.5 Press the ? key for help

 Find this C symbol:

 Find this global definition:
 Find functions called by this function:
 Find functions calling this function:
 Find this text string:
 Change this text string:
 Find this egrep pattern:
 Find this file:
 Find files #including this file:
 Find all function definitions:
 Find all symbol assignments:

Wipro confidential 5
 cscope
 Functions called by this function: str_netfd_alloc

 File Function Line

 0 netstr.c bug
 1 netstr.c str_empty
 2 netstr.c vsf_sysutil_recv_peek
 3 netstr.c vsf_sysutil_retval_is_error
 4 netstr.c die
 5 netstr.c die
 6 netstr.c vsf_sysutil_read_loop
 7 netstr.c vsf_sysutil_retval_is_error
 8 netstr.c die
 9 netstr.c die
 a netstr.c str_alloc_alt_term
 b netstr.c bug
 c netstr.c vsf_sysutil_read_loop
 d netstr.c vsf_sysutil_retval_is_error
 e netstr.c die
31 bug("poor buffer accounting in str_netfd_alloc");
36 str_empty(p_str);
39 retval = vsf_sysutil_recv_peek(fd, p_readpos, left);
40 if (vsf_sysutil_retval_is_error(retval))
42 die("vsf_sysutil_recv_peek");
46 die("vsf_sysutil_recv_peek: no data");
55 retval = vsf_sysutil_read_loop(fd, p_readpos, i + 1);
56 if (vsf_sysutil_retval_is_error(retval) ||
59 die("vsf_sysutil_read_loop");
63 die("missing terminator in str_netfd_alloc");
65 str_alloc_alt_term(p_str, p_readbuf, term);
72 bug("bytes_read > left in str_netfd_alloc");
75 retval = vsf_sysutil_read_loop(fd, p_readpos, bytes_read);
76 if (vsf_sysutil_retval_is_error(retval) ||
79 die("vsf_sysutil_read_loop");

 Find this C symbol:

 Find this global definition:
 Find functions called by this function: str_netfd_alloc
 Find functions calling this function:
 Find this text string:
 Change this text string:
 Find this egrep pattern:
 Find this file:
 Find files #including this file:
 Find all function definitions:
 Find all symbol assignments:

Wipro confidential 6
 cscope
 Functions calling this function: str_netfd_alloc

 File Function Line

 0 readwrite.c ftp_getline 94 str_netfd_alloc(

 Find this C symbol:

 Find this global definition:
 Find functions called by this function:
 Find functions calling this function: str_netfd_alloc
 Find this text string:
 Change this text string:
 Find this egrep pattern:
 Find this file:
 Find files #including this file:
 Find all function definitions:
 Find all symbol assignments:

Wipro confidential 7
 strace
 Traces the system calls used by program

• reports whether they pass or fail

• It even can follow forks.

 Attach to processes and begins tracing

• strace also lets you attach to processes for just-in-time debugging.

 Lets you know what’s happening even if there is no debugger or source code

 Useful in a live environment

Wipro confidential 8
 strace

 Each line in the trace contains the system call name, followed by its arguments
in parentheses and its return value.
• open("/dev/null", O_RDONLY) = 3

 Errors (typically a return value of -1) have the errno symbol and error string
appended.
• open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)

 Usage
• strace –c –p 2155 gives the count time, calls and errors for each system call and
report summary for the process with pid 2155
• strace –o strace.out –p 2155
• strace <program>

Wipro confidential 9
 strace
 For eg. If a normal user tries to delete the services file (owned by root)
• strace rm /etc/services
lstat64("/etc/services", {st_mode=S_IFREG|0644, st_size=362031, ...}) = 0
access("/etc/services", W_OK) = -1 EACCES (Permission denied)
unlink("/etc/services") = -1 EACCES (Permission denied)
write(2, "cannot remove `/etc/services\'", 29) = 29

• strace –o strace.out rm /etc/services will copy the output to strace.out

 strace –c –p 4313 (gcalctool is a calculator started in Linux with pid 4313)

• After few seconds do a Ctrl-C -> with the following output

% time seconds usecs/call calls errors syscall

------ ----------- ----------- --------- --------- ----------------------------------------------------
38.97 0.005311 19 285 poll
29.35 0.004000 68 59 gettimeofday
15.14 0.002063 172 12 writev
11.04 0.001504 7 213 26 read
5.51 0.000751 3 262 1 ioctl
0.00 0.000000 0 150 write
0.00 0.000000 0 1 uname
------ ----------- ----------- --------- --------- -------------------------------------------------------
100.00 0.013629 982 27 total

Wipro confidential 10
 pmap
 pmap reports the memory map of a process
 Amount of memory used by the process
 Details of the map are also available in the /proc

• pmap –x 2352
2352: ./server
Address Kbytes RSS Anon Locked Mode Mapping
00111000 1200 - - - r-x-- libc-2.4.so
0023d000 12 - - - r-x-- libc-2.4.so
00240000 4 - - - rwx-- libc-2.4.so
00241000 12 - - - rwx-- [ anon ]
006dc000 100 - - - r-x-- ld-2.4.so
006f5000 4 - - - r-x-- ld-2.4.so
006f6000 4 - - - rwx-- ld-2.4.so
00769000 4 - - - r-x-- [ anon ]
08048000 4 - - - r-x-- server
08049000 4 - - - rw--- server
b7f5d000 4 - - - rw--- [ anon ]
b7f6e000 4 - - - rw--- [ anon ]
bfe59000 88 - - - rw--- [ stack ]
-------- ------- ------- ------- ----------------------------------------------------------------
total kB 1444 - - -

Wipro confidential 11
 pstack
 pstack

• prints the stack trace of the running process – quite useful along with strace.
• Attaches to the active process given on the command line
• The output is read bottom up.

Wipro confidential 12
 nm
 Used to retrieve information on symbol names inside an object file or
executable file.
 Helps in resolving problems due to name conflicts
• If there are poorly defined headers tracking down the offending module becomes
easier using nm.

Wipro confidential 13
 ps
 ps

• powerful debugging tool

• Report a snapshot of the current processes
• use the –o option with ps and get many details such as virtual memory, cpu usage, current state
and lot more
• Most of the information that ps produces is available from the /proc filesystem
ps –ef will list the following
UID PID PPID C STIME TTY TIME CMD

root 2184 1635 0 09:00 ? 00:00:00 sshd: guest [priv]

guest 2187 2184 0 09:00 ? 00:00:01 sshd: guest@pts/2
guest 2188 2187 0 09:01 pts/2 00:00:00 -bash
guest 2281 1 0 09:11 ? 00:00:00 /usr/libexec/gam_server
guest 2352 2188 0 09:19 pts/2 00:00:00 ./server

Wipro confidential 14
 ps
 ps

ps -eo start,etime,state,pid,cmd

STARTED ELAPSED S PID CMD

08:54:15 27:26 S 1 init [5]
09:00:46 20:55 S 2184 sshd: guest [priv]
09:00:53 20:48 S 2187 sshd: guest@pts/2
09:00:59 20:42 S 2188 -bash
09:11:03 10:38 S 2281 /usr/libexec/gam_server
09:19:13 02:28 S 2352 ./server

Wipro confidential 15
 top
 top is a simple text based system monitoring tool

 Displays system summary and details, status of individual tasks (memory, cpu, pid and
more)

 The top line prints the current time, uptime since the last reboot, users logged in, and the
load average.

 The memory line displays the

• Total physical RAM available on the system,
• Amount of usage, free, shared, along with the amount of ram in buffers.

 Most importantly it displays per process the

• Amount of physical memory consumed
• percentage of the available processor time a process is taking
• total amount of processor time the process has had

Wipro confidential 16
 top
top - 10:57:09 up 2:02, 3 users, load average: 0.07, 0.12, 0.60
Tasks: 96 total, 1 running, 95 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3% us, 2.4% sy, 0.0% ni, 97.2% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 126252k total, 118640k used, 7612k free, 4148k buffers
Swap: 1048568k total, 16084k used, 1032484k free, 52500k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

22513 guest 16 0 2124 1008 796 R 1.6 0.8 0:05.98 top
1 root 16 0 1988 652 564 S 0.0 0.5 0:03.00 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
3 root RT 0 0 0 0 S 0.0 0.0 0:00.02 watchdog/0
4 root 10 -5 0 0 0 S 0.0 0.0 0:00.09 events/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.04 khelper
6 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
8 root 10 -5 0 0 0 S 0.0 0.0 0:00.29 kblockd/0

Wipro confidential 17
 lsof
 Reports a list of all
• open files and
• processes that opened them.

 Open files in the system include

• Disk files
• Pipes
• Network sockets and
• Devices opened by all processes.

 glsof is a GUI for lsof

Wipro confidential 18
 lsof
 To list the internet ports/files opened by guest user
• lsof –i | grep guest
• lsof –u guest –a -i

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

sshd 2168 guest 3u IPv4 7562 TCP 10.116.3.80:ssh->m2-14956.wipro.com:writesrv (ESTABLISHED)
sshd 2605 guest 3u IPv4 9888 TCP 10.116.3.80:ssh->m2-14956.wipro.com:direcpc-si (ESTABLISHED)
server 3191 guest 3u IPv4 12473 TCP *:8171 (LISTEN)
server-test 3198 guest 0u IPv4 12508 TCP *:8181 (LISTEN)
telnet 3529 guest 3u IPv4 16885 TCP 10.116.3.80:53112->efcg.wipro.com:telnet (ESTABLISHED)

 Examining a process (pid 3198)

• lsof –p 3198
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
server-test 3198 guest cwd DIR 253,0 4096 948823 /home/guest/testserver/webpages
server-test 3198 guest rtd DIR 253,0 4096 2 /
server-test 3198 guest txt REG 253,0 10971 948808 /home/guest/testserver/server-test
server-test 3198 guest mem REG 253,0 121396 1275458 /lib/ld-2.4.so
server-test 3198 guest mem REG 253,0 1528292 1275459 /lib/libc-2.4.so
server-test 3198 guest mem REG 0,0 0 [vdso] (stat: No such file or directory)
server-test 3198 guest 0u IPv4 12508 TCP *:8181 (LISTEN)

Wipro confidential 19
 lsof
 Sometimes you need to track down the user or process that's blocking you from
unmounting a disk.
• Umount /opt will give -> umount: /opt: device is busy
• kill `lsof -t /opt` will kill all the users on the /opt

 List all processes that have files open in /tmp

• lsof +D /tmp

Wipro confidential 20
 netstat
 Netstat gets the information about the status of network connections

 Displays the kernel routing tables

 Retrieve information on Network interfaces

• netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.116.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 10.116.3.1 0.0.0.0 UG 0 0 0 eth0

Wipro confidential 21
 netstat
 Processes accepting connections for TCP
• netstat -tpln

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1353 0.0.0.0:* LISTEN 2352/server
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:50000 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:50002 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:52340 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN -

Wipro confidential 22
 ifconfig
 Allows to configure network interfaces (bring interfaces up or down)
 Displays the currently active and inactive network interfaces.
 Supports a variety of address families and hardware types
• Address famililes – inet (ipv4), inet6(ipv6), unix ..
• Hardware types – ethernet, x.25, frame-relay ..

• ifconfig eth0 - View the network settings on the first Ethernet adapter installed in the computer.
eth0 Link encap:Ethernet HWaddr 00:03:FF:E3:08:1C
inet addr:10.116.3.80 Bcast:10.116.3.255 Mask:255.255.255.0
inet6 addr: fe80::203:ffff:fee3:81c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1640 errors:0 dropped:0 overruns:0 frame:0
TX packets:434 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:163039 (159.2 KiB) TX bytes:108893 (106.3 KiB)
Interrupt:11 Base address:0x6000

Wipro confidential 23
 sar
 Collect, report, or save system activity information.
 Displays the CPU activity
 Memory usage can be monitored
 Disk I/O activity
 Number of context switches - a good indication of how much time a process is
wasting
• sar –u -> cpu utilisation
Linux 2.6.15-1.2054_FC5 (localhost.localdomain) 12/05/2006

10:10:02 AM CPU %user %nice %system %iowait %idle

10:20:01 AM all 14.20 22.05 57.01 6.74 0.00
10:30:02 AM all 16.73 21.21 59.62 2.44 0.00
Average: all 15.47 21.63 58.32 4.59 0.00

Wipro confidential 24
 sar
 sar –r -> prints the memory usage
10:10:02 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
10:20:01 AM 3500 122752 97.23 4108 45604 1032052 16516 1.58 1952
10:30:02 AM 2904 123348 97.70 2572 49116 1032052 16516 1.58 980
Average: 3202 123050 97.46 3340 47360 1032052 16516 1.58 1466

 sar –b -> prints the disk i/o

10:10:02 AM tps rtps wtps bread/s bwrtn/s

10:20:01 AM 52.00 34.31 17.69 911.39 229.92
10:30:02 AM 24.33 10.30 14.03 91.34 188.15
10:40:01 AM 22.05 11.81 10.24 137.21 136.69
Average: 33.22 19.08 14.14 389.49 186.85

 sar –w -> number of context switches/sec

10:10:02 AM cswch/s
10:20:01 AM 1749.10
10:30:02 AM 1997.61
10:40:01 AM 1269.78
Average: 1688.47

Wipro confidential 25
 iostat
 Reports the CPU statistics and Input/Output statistics for devices and partitions
 Can be used with sar and vmstat
 CPU utilization gives the percentage of idle time with/with out any outstanding
I/Os requests (iowait, idle).
 Blocks read and written is given for device utilization

avg-cpu: %user %nice %system %iowait %idle

7.56 1.43 11.43 2.79 76.80

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

hda 5.79 188.11 38.92 284468 58862
dm-0 11.79 186.26 38.90 281674 58832
dm-1 0.04 0.33 0.00 504 0

Wipro confidential 26
 ddd
 DDD – Data Display Debugger is a popular GUI for command line
debugger
 Debugs executable binaries using GDB, DBX command line debuggers
 Debugs programs in
• Java (DDD with JDB Java debugger)
• Python (DDD with PYDB, Python debugger)
• Perl (Perl debugger)
 Advantages
• User need not know much about the underlying command line debugger used with
DDD
• Available freely

Wipro confidential 27