You are on page 1of 100
18.11 Configuration le case study 601 ifdef( COMMERCIAL CONFIG’, “define(confCLIENT_KEY,, ocal/certs/key. per) define{ confEIGHT_BIT_ HANDLING’, ‘mimify}) define( con{LDAP_DEFAULT. SPEC’, -h‘Idap sendmail com Idap2-sendmailcom" “b "de-tendmailde=com" -p 1389)) define( confREFUSE_LA, 93} definel confRUN_AS USER’, ‘mailnull!) ifdef( COMMERCIAL CONFIG’, “define(con{SERVER CERT, ‘/localcerts/cert per’) ifde{( COMMERCIAL CONFIG, ‘define(confSERVER KEY", "local/certs/key per!) define(confT0_ IDENT, “Os) define( confTO_QUEUEWARN, "24) ifdeffconfPOP 0’, °,“define(canfPOP_70', "300 FEATURE( accept nqualified_senders}) FEATURE(accept_unresolvable domains’) FEATURE(allmasquerade) FEATURE(always_add_domain’) FEATURE(domaintable) FEATURE(\dap routing, Idap -1 -v mailHost -k Idap -1 -v mailhost -k (&(cbjectclass=maiiRecipient)(mail=%.0)(mailAlternateAddress=%0))) “dap “1 -v mail -k (@(objectclase-mailRecipientmailalternateaddress-%40) ‘passthru) FEATURE(mailertable} FEATURE( masquerade entire domain) FEATURE( masquerade_envelope’) FEATURE(relay_entire domain’) FEATURE(use_cw- file) MAILER(ocal}) MAILER(smntp)) LOCAL, RULESETS Stocal check rept RS: & $ufvenfy) ROK $# 0K ‘The master machine routes incoming mail othe correct internal server and serves as, the smart relay host for outgoing mail. Because ofthe following two lines, FEATURE accept_unqualified_senders) FEATURE(accept_unresolvable_domains) all incoming mail is accepted, even mail from unqualified senders and unresolvable ‘domains. This way, potential customers who have sendmail or DNS misconfigured «an till get through. These rules undo the defaults that catch lots of spam with forged. headers. dent is turned off (timeout set to 0) to speed up delivery of incoming mal. ‘This master mail machine fist checks incoming messages fr certain types of MIME attachments that are frequently used by viruses (INPUT _MAIL, FILTER statement). ‘The mime-filter called there contains lines such as ~sanniv.doc:~ error-Your email was not accepted by Sendmail, it appears to be infected with the Melissa-X virus. sivbs= errorFor gecurity and virus protection reasons, Sendmail does ‘not accept messages with VBS files attached, Please retransmit your rmeseage without the VBS fle, 602 Seepage 507 for more els about miter (Chapter 18 ~ Electronic Mail MIME attachments of type .vba, ot, exe, .com, .eg, and so on are rejected, but a full virus scan is not done here because it would slow the processing of incoming ‘mail. The master uses LDAP (with a site-specific schema) to look up the recipient of, each message and route it to the correct internal IMAP/POP server: Ifthe recipient is not found in the LDAP database, the mail is sent to an internal master machine (the MAIL, HUB statement) for further processing. Both the IMAP/POP servers and the internal master machine do ful virus scanning before delivering a message to a users mailbox. Cutgoing mails also routed through this master machine by SMART_HOST state- ‘ments on the client machines. To send a message through the sendmail.com mail servers, hosts outside the sendmail.com domain must present a certificate signed by the sendmail.com certificate authority. Employees visiting a customer site can relay email to a third party through sendmail.com with this mechanism, but others can- not. This convention authenticates each user and prevents forged email from tran- siting sendmail.com, After accepting email destined for the Internet, the master machine passes it to the ‘SMART_HOST for virus scanning. The master mail machine is not to0 busy to do this virus scanning itself, bur ifthe scanning were done there, users sending mail would have to wait forthe scanning to complete before their message was realy sent. Queueing it for the virus-scanning machine keeps the users happy—their messages seem to zip off instantaneously. ‘The LOCAL CONFIG rules atthe end of the config file ate where header checking for various viruses and known spammers is usually put. Good examples can be found in the knecht.me example file inthe sendmail distribution. We have included a sample below. During the summer of 2001, the destructive SieCam worm was circulating wildly. The following fragment from the knechtae file in the sendmail distribution catches it. SirCam is one of the first nastygrams to have random headers. The usual tools to catch it would have failed, except that its authors made an error that differentiates a SirCam message from a real Outlook Express message. The message's content is quite regular (it asks for your advice on the enclosed attachment) and would therefore be a candidate for the new libmilter filtering abilities in version 8.12. Without product liability guarantees in the software world, it seems the only solution to all these Mi- crosoft viruses and worms is to dump Windows and install Linux everywhere! LOCAL, RULESETS KSirCamWormMarker regex -f -aSUSPECT multipart/mixed:boundary= “+-OutlookExpress_message boundary Content-Type: $>CheckContentType ScheekContentType RS 5: $(SirComWormMarker $1 $) RSUSPECT S¥error $:"5S3 Possible virus, see http: ‘fw symantec com /aveenter/venc/data/w32.srcam.worm@mm btm" 18.12 18:12 Secuity and sendmail 603 HContent-Disposition:$>CheckCon sntDigposition ScheekContentDisposition BS, $a OK BS. | Se $0 OK BS $Sterror §:°953 Ilogal Content-Disposition” Clients at sendmai.com have no spam control in their contig files. The reason is that all mail coming into the site comes through the external mail hub and an inter- nal hub and the spam is winnowed there. Some ofthe features and other constructs inthis example are not covered in our configuration section, but you can find docu- ‘mentation on them in the effREADME file ‘SECURITY AND SENDMAIL ‘With the explosive growth ofthe Internet, programs such as sendmail that accept arbitrary user-supplied input and deliver it to local users, files, or shells have fre= ‘quently provided an avenue of attack for hackers. sendmail, along with DNS and ‘even IPs flirting with authentication and encryption asa built-in solution to some ‘of these fundamental security issues. Recent softening of the export laws of the United States regarding encryption freed ‘sendmail tobe shipped with builtin hooks for encryption. Versions 8.11 and later support both SMTP authentication and encryption with TLS, Transport Layer Se- ‘curity (previously known as SSL, the Secure Socket Layer). sendmail uses the term ‘TLS in this context and has implemented it as an extension, STARTTLS, to the SMTP protocol. TLS brought with it six new configuration options for cetificate files and. key files. New actions for acess database matches can require that authentication rust have succeeded. In this section, we describe the evolution of sendmail’ permissions model, owner- ships, and privacy protection. We then brielly discuss TLS and SASL the Simple Au- thentication and Security Layer) and their use with sendmail, ‘sendmail has gradually tightened up its security overtime, and itis now very picky about fle permissions before it believes the contents of, say, forward or aliases file, Although this tightening of security has generally been welcome, its sometimes necessary to relax the tough new policies. To this end, sendmail introduced the DontBlameSendmail option, so named in hopes that the name will suggest to sysadmins that what they are doing is considered unsafe ‘This option has many possible values—55 at last count. The default is safe, For a complete list of values, see docfop/op.ps in the sendmail distribution, The values are not listed inthe second edition ofthe O'Reilly sendmail book, but will surely be in the third. Or just leave the option set to safe. ‘Ownerships ‘Three user accounts are important in the sendmail universe: the DefaultUser, the ‘TrustedUser, and the RunAsUser Wail ane 604 (Chapter 18 ~ Electronic Mail By default, all of sendmail’s mailers run as the DefaultUser unless the mailers flags specify otherwise. Ifa user mailnull, sendmail, or daemon exists in the fete/passwd file, DefaultUser will be that. Otherwise, it defaults to UID 1 and GID 1. We recom- ‘mend the use of the mailnull account and a mailnull group. Add itt /ete/passwd With a star asthe password, no valid shell, no home directory, and a default group of ‘mailnll. You'll have to add the mailnull entry to the /ete/group file too. The mail- null account should not own any files. sendmail is not running as root, the mail- ers must be setuid. IfRunAsUser is set, sendmail ignores the value of DefaultUser and does every- thing as RunasUser. Ifyou are eunning sendmail setgid (to smmsp), then the sub- mission sendmail just passes messages to the real sendmail through SMTP. Te real sendmail does not have its setuid bit set, but it runs as root from the startup fies. sendmail’ TrustedUser can own maps and alias files. The TrustedUser is allowed to start the daemon or rebuild the aliases file. This facility exists mosly to support GUT interfaces to sendmail that need to provide limited administrative conteol to certain users Ifyou set TrustedUser, be sure to guard the account that it points to, because this account can easily be exploited to gain oot acces. The TrustedUser is different from the TRUSTED USERS class, which determines who can rewrite the From line of messages.” ‘The RunAsUlser is the UID that sendmail runs under after opening its socket con- nection to port 25. Ports numbered less than 1,024 can be opened only by the supe- user; therefore, sendmail must initially run as root. However, after performing this operation, sendmail can switch toa different UID. Such a switch reduces the risk of| damage or access if sendmail is tricked into doing something bad. Don't use the RunAsUser feature on machines that support user accounts of other services; it is ‘meant for use on firewalls or bastion hosts only. By default, sendmail does not switch identities and continues to run as root. Ifyou change the RunAsUser to something other than root, you must change several other things as well. The RunAsUser must own the mail queue, be able to read all maps and include files, be able to run programs, etc. Expect to spend a few hours finding all the file and directory ownerships that must be changed Permissions File and directory permissions are important to sendmail security. Use the settings listed in Table 18.14 tobe safe sendmail does not read files that have lax permissions (for example, file that are _group- of world-weitable or that live in group- or world-vritable directories). Some of sendmail’ rigor with regard to ownerships and permissions was motivated by ‘The TRUSTED _USERSfeatureisypclly sed support malig list software. For example fyouuse Majordome, you must add the "msjordom” se othe TRUSTED USERS clas. Th user soot and daemon ae the default members ofthe cats 18:12 Secuity and sendmail 605 Table 18.14 Owner and permissions for sendmail-related directories ath ‘Owner ‘Mode _Whatitcontains ‘War/spoolidientmqueue snmp 70 Nail queue fornia submissions* Nar/spool/mqueue —RunAsUser_ 700 Mall queue directory 1. Iva,var/spool root 755. Pathtomqueue Jetcimail/* ‘TrustedUser 644 Maps the configfile aliases Jetcimall ‘Trustedliser 755 Parentdlrectory for maps ete root 755 Path toma dtectory {Veron 12 and ter ‘operating systems that let users give ther files away with chown (those derived from system V, mostly), Linux systems by default have a sane version of chown and do not allow file give- aways. However, an #ifdef in the code (CAP_CHOWN) can be set to give System V semantics to chown. You would then have to rebuild the kernel. But this behavior is evil; don't coerce your sensible Linux chown to behave inthe broken System V way. In particular, sendmail is very picky about the complete path to any alias file or for- ‘waed file. Ths pickiness sometimes clashes with the way sites like to manage Major: /procisys/net/ipvajtep fin timeout to change TCP’s FIN timeout value ‘SENDMAIL STATISTICS, TESTING, AND DEBUGGING ‘sendmail can collect statistics on the number and size of messages ithas handled, You display this data with the mailstats command, which organizes the data by mailer. sendmail’ STATUS FILE option (in the OSTYPE file) specifies the name of the fle in which statistics should be kept. The existence of the specified file turns on the accounting function. ‘The default location for sendmail’ statistics fle is fete/mail/statisics, but some vendors cal the file Sendmail.st and put it in var/log. The totals shown by mai stats are cumulative since the creation of the statistics file Ifyou want periodic sta- Listes, you can rotate and reinitialize the file from exon, os (Chapter 18 ~ Electronic Mail Here isan example: 5 mailstats Statistics from Tue Aug 1 02:13:30 2006 ‘M- magefr bytes from magsto bytes to mgere)_megedis Mailer 422 2k 63 ASK 0 0 esmtp 7 a oK 1B 28K a 0 relay est 472K a ox a 0 tocal tT 66 “ork 18K a ° cc 66 a a the mail statistics ile is world-readable, you don't need to be root fo run mailstats. Six values are shown: messages and kilobytes received (msg, bytes from), mes sages and kilobytes sent (msgsto, bytes_to), messages rejected (msgsre}), and mes- sages discarded (msgsci). The fist column isa number identifying the mailer, and the last column lists the name of the mailer. The T row is total messages and bytes, and the Crow is connections, These values include both local and relayed ml Testing and debugging smit-based configurations are to some extent pretested. You probably won't need to do low-level debugging ifyou use them. One thing the debugging flags cannot testis your design, While researching this chapter, we found errors in several ofthe config tration files and designs that we examined. The errors ranged from invoking a fea- ture without the prerequisite macto (eg. using masquerade envelope without having turned on masquerading with MASQUERADE_ AS) to total conflict between the design of the sendmail configuration and the firewall that controlled whether and under what conditions mail was allowed i, You cannot design a mail system in a vacuum. You must be synchronized with (or at least not be in conflict with) your DNS MX records and your firewall policy. sendmail provides one ofthe world’s richest sets of debugging aids, with debug flags that are not simple Booleans or even integers but are two-dimensional quantities x, where x chooses the topic and y chooses the amount of information to display. A value ofO gives no debugging, and 127 wastes many tees if you print the output, ‘Topics range from 0 to 99; currently, about 80 are defined. ‘The file Sendmail TRACEFLAGS in the distribution lists the values in use and the files and functions in which they are used, All debugging support i at the level of the raw config file. in many cases it’s helpful to look atthe sendmail source along with the debug output. If sendmail is invoked with a-dx.y lag, debugging output comes tothe screen (stan dard error) Table 18.18 shows several important values of xand some suggested val- ues fory, Be careful if you turn on debugging for a sendmail running asa daemon (bd) because the debug output may end up interjected into the SMTP dialogue and cause odd failures when sendmail talks to remote hosts. 18.14 sendmail statistics, testing, and debugging a7 Table 18.18 Debugging topics chechsenda i ‘vale fom ww harencom. Topic Meaning and suggestions (0 Shows compile lags and system identity (ry y= 1 or 10) 8 Shows DNS name esolution (try =8) 11 Traces delivery (shows mailer invocations) 12. Shows cabto-remote name translation 17 Lists aK hosts a 21 Traces tering ues (use 2 forme detail = 27. Showsalasing and forwarding ey y=4) eS 44 Shows file open attempts in case things are failing (y = 4) = 60 Shows database map lookups . Gene Kim and Rob Kolstad have weitten a Perl script called checksendmail that in- vvokes sendmail in address test mode on a file of test addresses that you supply: It ‘compares the resulls to those expected. This script lets you test new versions of the configuration file against a testsuite of your site’ typical addresses tobe sure you ‘haven't inadvertently broken anything that used to work. Verbose delivery Many user agents that invoke sendmail on the command line accept a-v flag, which is passed to sendmail and makes it display the steps taken to deliver the message. ‘The example below uses the mail command. The words in bold were typed as input to the user agent, and the rest is sendmails verbose output. S$ mail -v trent@toadranch.com Subject: just testing, please ignore bi ce tyent@toadranch.com... Connecting to coyate toadranch.com. via esmtp. 220 coyote toadranch.com ESMTP Sendmail §.11.0/8.11.0; Tue, 7 Aug 2001 20: ‘0851 -0600 ose EHLO anchor.cs.colorada.edu s0-coyete.toadranch com Hello anchor.cs colorado.edu [128.138.242.1], pleased to meet you $0- ENHANCEDSTATUSCODES 0-EXPN 0-XUSR 0-AUTH DIGEST-MDS CRAM-MDS 0 HELP >s> MAIL From:... Sender ok ose RCPT To:etr 250 241 os DATA 394 Enter mail, end with * 250 2.00 £7826pi03229 Message accepted for delivery trent@toadranch com... Sent ({7828p103229 Message accepted for delivery) Closing connect >>> QUIT 221 2.00 cayo .dranch.com>... Recipient ok a line by itself tetoadranch.com closing connection ‘The sendmail on anchor connected to the sendmail on toadranch.com. Each ma- chine used the ESMTP protocol to negotiate the exchange of the message. You can make direct use of SMTP when debugging the mail system. To initiate an SMTP session, use sendmail -bs or telnet to TCP port 25. By default, this is the port on which sendmail listens when run in daemon (-bd) mode; sendmail uses port 587 when running as the mail submission agent. Table 18.19 shows the most impor- tant SMTP commands. SMTP commands ‘com Function HELO hostname Identifies the connecting host if speaking SMTP EHLO hastname Identifies the connecting host if speaking ESMTP [MAILFrom:reypath Initiates a mail transaction (envelope sender) RCPTTo: fudpath* Identifies envelope recipent(s) VRE address Veifes that adres is valid (deliverable) EXPN address Shows expansion of aliases and forward mappings DATA Begins the message body? our Ends the exchange and closes the connection SET Resets the state ofthe connection HELP Printsa summary of SMTP commands 1 There canbe multiple RCPT commands for amessage You eminat the body by enternga daton fs on ne ‘The whole language has only 14 commands, so its quite easy to learn and use. Its not case sensitive. The specification for SMTP can be found in RFC2821. Most transport agents, including sendmail, speak both SMTP and ESMTP; smap is the lone exception these days, Unfortunately, many firewalls boxes that provide at filtering do not speak ESMTP. ESMTP speakers start conversations with the EHLO command instead of HELO. Ifthe process atthe other end understands and responds with an OK, then the participants See Chapter 10for ‘more information about lg 18.14 sendmail statistics, testing, and debugging a9 negotiate supported extensions and arrive ata lowest common denominator for the ‘exchange. Ifan error is eturned, then the ESMTP speaker falls back to SMTP. ‘Queue monitoring ‘You can use the mailg command (which is equivalent to sendmail -bp) to view the status of queued messages. Messages are “queued” while they are being delivered or when delivery has been attempted but has failed. ‘mailg prints a human-readable summary ofthe files in var/spool/mqueue at any ‘given moment. The output is useful for determining why a message may have been delayed. iit appears that a mail backlog is developing, you can monitor the status of KSULKAHBO37374 279 Fri jun 20 15-46. crandy@atrust.com> (Deferred: Name server: k2wieless.com. host name lookup fa) kesujom72023575 2485 Fri jun 30 13:13 MAILER-DAEMON, (reply: read error from mx4.level3.com) ‘Aug 18 22:41:33 nova postfix/cleanup: OE4AS3688: message-i «<20040818204132,GA11444@e0.ethz.ch> ‘Aug 18 22:41:33 nova postfix/qmgr: OF4A93688: from=cdws@ee.ethz cho, size=S77,nrept=1 (queue active) ‘Aug 18 22:41:33 nova postfix/smtp: OF4A93688: to-cevides ethz.ch> relay=tardis.ee eth ch{129.132.2.217] delay-0 statusesent (250 Ok: queued as 1540405208) ‘Aug 18 22:41:33 nova postfix/qmgr: 0493688: removed ‘As you can see, the interesting information is spread over many lines. Note that the identifier 0E4A93688 is common to every line: Postfix assigns a queue ID as soon as ‘a message enters the mail system and never changes it. Therefore, when searching the logs for the history of a message, first concentrate on determining the message's ‘queue ID. Once you know that, it’s easy to grep the logs forall the relevant entries, Postfix is good at logging helpful messages about problems that it notices. However, its sometimes difficult to spot the important lines among the thousands of normal status messages. This is a good place to consider using some ofthe tools discussed in the section Condensing log files to useful information, which starts on page 220, 638 (Chapter 18 ~ Electronic Mail Looking at the queue Another place to look for problems is the mail queue. As in the sendmail system, a smailg command prints the contents of a queue. You can use ito see ifand why a message has become stuck. Another helpful tool i the qshape script thats shipped with recent Postfix versions. shows summary statistics about the contents ofa queue. The output looks lke this # qshape deferred T 5 10-20 40 0 160 320 640 1280 1280+ TOTAL72 00073 3 212 2 4 expncom 34 00000 0 03 0 2 chinabankph 5 00011 1 2.0 0 @ probhelperbi2 3 000000 00 0 3 ‘qshape summarizes the given queue (hete, the deferred queue) sorted by recipient domain, The columns report the number of minutes the relevant messages have been in the queue. For example, you can see that 49 messages bound for expn.com have been in the queue longer than 1280 minutes. All the destinations in this example are suggestive of messages having been sent from vacation scripts in response to spam. ‘qshape can also summarize by sender domain with thes ag, Soft-bouncing Ifsoft_bounce is set to yes, Postfix sends temporary error messages whenever it would normally send permanent error messages such as “user unknown” or “relay- ing denied.” Ths is a great testing feature; it les you monitor the disposition of mes- sages afte a configuration change without the risk of permanently losing legitimate email. Anything you reject will eventually come back for another try. Don't forget to turn off this feature when you are done testing, however, Otherwise, you will have to deal with every rejected message over and over again. Testing access control ‘The easiest way to test access control restrictions isto try to send a message from an outside host and see what happens. This is good basic test, but it doesn't cover special conditions such as mail from a specific domain in which you have no login, Postfix 2.1 introduced an extension to the SMTP protocol called XCLIENT that sim- tulates submissions from another place. This feature is disabled by default, but with the following configuration line in main.cf, you can enable it for connections origi- nating from localhost smtpd authorized xclient hosts = localhost A testing session might look something like this: 5 telnet localhost 25 ‘Trying 127.001. Connected to localhost. Escape character is" 18.17 Recommended eading 639 220 tardis.ce.ethz.ch ESMTP Postfix XCLIENT NAME=mail.cs.colorado.edu ADDR=192.168.1.1 250 Ok HELO mailes.colorado.ed 0 tardis ee ethzch ‘MAIL FROM: 0 Ok RCPT TO: 554 cdavid@colorado.edu>: Relay access denied a 18.17 RECOMMENDED READING 5 CCostatss, BRYAN, and ERIC ALLMAN. sendmail (3d Editon). Sebastopol, CA : ‘O'Reilly Media, 2002. ‘This book is the definitive vome—1,200 pages’ worth. It includes a tutorial as well as ‘a complete reference section. The book reads well in the open-to-a-random-page mode, which we consider an important feature fora reference book. It has a good index too. CLAYTON, RICHARD. "Good Practice for Combating Unsolicited Bull Email.” RIPE/Demon Internet. 2000, www.ripe.net/ripe/docs/ripe-206.html ‘This document is aimed at ISPs. thas los of policy information and some good links to technical subjects SCHWARTZ, ALAN. SpamAssassin. Sebastopol, CA: O'Reilly Media, 2005, ‘SCHWARTZ, ALAN, AND PAULA FERGUSON. Managing Mailing Lists. Sebastopol, CA: ‘O'Reilly Media, 1998. HazeL, PHIL. The Exim Smip MailServer: Official Guide for Release 4. Cambridge, UK: User Interface Technologies, Ltd, 2003 Exim documentation and information can also be found at www.exim.org, ‘The man page for sendmail describes its command-line arguments, See Sendmail ‘An Internetwork Mail Router, by Eric Allman, for an overview. Installation instructions and a good description of the configuration fle are covered in Sendmail Installation and Operation Guide, which can be found in the doclop subdirectory of the sendmail distribution. This document is quite complete, and in conjunction with the README fle in the ef directory, it gives a good nuts-and-bolts, view of the sendmail system, ‘wwrw-sendmailorg, www-sendmail org/~ca, and www.sendmail.org/~gshapiro all contain sendmail-related documents, HOWTOs, and tutorials. RFC2822, which supersedes RFC822, describes the syntax of messages and ad- dresses ina networked mail system, and RFC1123 describes host requirements, ‘These are, in a sense, the functional specifications to which sendmail was built, 18.18 (Chapter 18 ~ Electronic Mail RFC2821, which supersedes RFC821, defines the Simple Mail Transport Protocol (SMTP), and RFCs 1869, 1870, 1891, and 1985 extend it to ESMTP. REC974 describes MX records in the Domain Name System and theit relationship to ‘mail routing. Other mai -related RFCs include C1731 - IMAP4 Authentication Mechanisms + RECI733 - Distributed Electronic Mail Models in IMAP + REC2033 - Local Mail Transfer Protocol + REC2076~ Common Internet Message Headers : ailbox Names for Common Services, Roles and Functions + REC2505 ~ Anti-Spam Recommendations for SMTP MTAS + REC2635 - DON'T SPEW: Guidelines for Mass Unsolicited Mailings" + REC2821 - Simple Mail Transfer Protocol + REC2822 - Internet Message Format + RECI05 ~ SMTP Service Extension for Indicating Message Submitters + RECH06 - Sender ID: Authenticating E-Mail + RECHO8 - SPF for Authorizing Use of Domains in E-Mail, Version 1 + RECH409 ~ Message Submission for Mail RFCs 2821 (SMTP) and 2822 (Internet Message Format) tidy up some of the most commonly referred-to email RFCs; they supersede RFCs 821, 822, 974, and 1869. RECs 2821 and 2822 were fitst published in April 2001 and are proposed standards. EXERCISES E18.1 [sendmail specific] Briefly list the differences and similarities between ge nericstable and virtusertable. In what situations would you use each? E18.2__ [sendmail specific] Compare the use of /ete/mail/aliases with the use ‘of an LDAP server to store mail aliases, What are the advantages and. disadvantages ofeach? E183. Briefly explain the difference between a mail user agent (MUA), a delivery agent (DA), and an access agent (AA). Then explain the difference between a ‘mail transport agent (MTA) and a mail submission agent (MSA). E184 [sendmail specific] Whats smesh, and why should you use i instead of bin/sh? If smrsh sin use at your site, what programs ae allowed to rrunas the progeam mailer? Are any of them dangerously insecure? E185. [sendmail specific] Writea small /ete/mailfaliases file that demon- strates three different types of aliases, Talk briefly about what each line does and why it could be useful 35, Taleparaphrsed 18.18 Exercises oa se EIB.6 oe EIS.7 E188 vk E189 Write a brief description ofthe following email header. What path did the email ake? To whom was itaddressed, and to whom was it delivered? How long did i take the email to go from the sender tothe destination? From clements@boulderlabe.com Fri Dec 28 17:06:57 2001 Return-Path: Received: from boulder Colorado.EDU (boulder Colorado.£DU (128.138.240.1)) by ucsub.colorado.edu (8.11.6/8.11.2/TTS-5.0/student) ‘with ESMTP id—BTD6vE10618 for ; Fri, 28 Dec 2001 17:06:57-0700 (MST) Received: from mail boulderlabs.com (mail boulderlabs.com (206.168 112.48) by boulder.Colorado EDU (€.10.1/8.10.1/UnixOps+Hesiod (Boulder)) with ESMTP id #8TO6ULI2184; Fri, 28 Dee 2001 17:06:56 -0700 (MST) Received: from ath.boulderlabs com (epe-24-221-212-162.co sprintbbd net (24.221,212.169)) by mail boulderlabs.com (@11.6/8.11.6) with ESMTP 1 {87060929214 for cbooklistaboulderlabs.com>; Fri, 28 Dec 2001 117-06:50 -0700 (MST) (envelope-from clements@mail houlderlabs.com) From: David Clements Received: from clements@localhost) by ath boulderlabs.com (€11.6/8.114) id f8T05m201470 for booklistBboulderlabs.com; Fri 28 Dec 2001 17:06:48 -0700 (MST) (envelope-from clements) Date: Fri, 28 Dec 2001 17:06:48 -0700 (UST) Message-Id: <200112290006.fBT06ma01470@ath.boulderlabs com> ‘To: boalist@boulderlabs.com, Subject: Book Questions [sendmail specific Lis the prefixes fo files in the mail queue directory and explain what each one means, Why is it important to delete some {queue files but very wrong to delete others? How can some of the pre- fixes be used to debug sendmail configuration mistakes? Look at the mailq on your campus mail server. Is there any cruftin the directory? Are there any messages with no control files or control files with no messages? What is the oldest message in the queue? (Requires root access.) [sendmail spectic] Explain the purpose of each of the following m4 ‘macros Ifthe macro includes a file, provide a short description of what the contents ofthe file should be. a) VERSIONID b) OsTYPE ©) DOMAIN ) MAILER ©) FEATURE Exercises are continued on the next page. Joke Yoke (Chapter 18 ~ Electronic Mail E1810 EIS E1812 E1813 BELG [Explain what an MX record is. Why are MX records important for mail delivery? Give an example in which a misconfigured MX record might ‘make mail undeliverable. ‘What are the implications of being blacklisted on sbl-xbLspamhaus.org ‘or a similar spam black hole list? Outline some techniques used to stay offsuch lists. your site allows procmail and if you have permission from your local sysadmin group, set up your personal procmail configuration fie to il- lustrate how procmail can compromise security Explore the current MTA configuration at your site. What are some of the special features of the MTA that are in use? Can you find any prob- Jems with the configuration? In what ways could the configuration be ‘made better? Find a piece of spam in your mailbox and inspect the headers. Report, any signs thatthe mail has been forged. Then run some ofthe tools ‘mentioned in this chapter, such as SpamCop or SpamAssassin, and re- port their findings, How did you do at recognizing faked headers? Sub- ‘it the spam and your conclusions about the sender, the validity of the listed hosts, and anything else that looks out of place. 19 Network Management and Debugging Because networks increase the number ofinterdependencies among machines, they tend to magnify problems. As the saying goes, “Networking is when you cant get any ‘work done because of the failure of a machine you have never even heard of” Network management isthe art and science of keeping a network health: It gener- ally includes the following tasks: Fault detection for networks, gate Schemes for notifying an administrator of problems General monitoring, to balance load and plan expansion Documentation and visualization ofthe network ‘+ Administration of network devices from a central ste ys, and critical servers ‘Ona single network segment, its generally not worthwhile to establish formal pro- cedures for network management, Just test the network thoroughly after installation, and check it occasionally tobe sure that its load isnot excessive. When it breaks, fix it As your network grows, management procedures should become more automated. ‘On annetwork consisting of several different subnets joined with switches or routers, ‘you may want to start automating management tasks with shell scripts and simple programs, Ifyou have a WAN or a complex local network, consider installing a ded. icated network management station, In some cases, your organization’ reliability needs dictate the sophistication of your network management system. A problem with the network can bring all work to a standstill Four site cannot tolerate dowatime, it may well be worthwhile to ob- tain and install a high-end enterprise network management system, 19.1 Chapter 19 — Network Management and Debugging Unfortunately, even the best network management system cannot prevent all flutes. Wis critical 1o have a well-documented network and a high-quality staff available to handle the inevitable collapses. NETWORK TROUBLESHOOTING Several good tools are available for debugging a network at the TCP/IP layer. Most sive low-level information, so you must understand the main ideas of TCP/IP and routing in order to use the debugging tools. On the other hand, network issues can also stem from problems with higher-level protocols such as DNS, NFS,and HTTP. You might want to read through Chapter 12, ‘TCPAP Networking, and Chapter 13, Routing, before tackling this chapter. In this section, we start with some general troubleshooting strategy: We then cover several essential tools, including ping, traceroute, netstat, tepdump, and Wire- shark. We don't discuss the arp command in this chapter, though it, t00, is useful debugging tool—see page 296 for more information, Before you attack your network, consider these principles: ‘+ Make one change at atime, and test each change to make sure that it had the effect you intended. Back out any changes that have an undesired effect. + Document the situation as it was before you got involved, and document every change you make along the way. + Start at one “end?” ofa system or network and work through the systems critical components until you reach the problem. For example, you might start by looking at the network configuration on a client, work your way up to the physical connections, investigate the network hardware, and finally, check the server's physical connections and software configuration. + Communicate regularly. Most network problems involve or affect lots of different people: users, ISPs, system administrators, telco engineers, net- work administrators, etc. Clear, consistent communication prevents you. from hindering one another's efforts to solve the problem, + Work as a team. Years of experience show that people make fewer stupid iistakes if they have a peer helping out. + Use thelayers ofthe network to negotiate the problem. Start a the “top” or “bottom” and work yout way through the protocol stack. ‘This last point deserves a it more discussion. As described on page 275, the archi- tecture of TCP/IP defines several layers of abstraction at which components of the network can function. For example, HTTP depends on TCP, TCP depends om IP, IP depends on the Ethernet protocol, and the Ethernet protocol depends on the integ- rity ofthe network cable. You can dramatically reduce the amount of time spent de- bugging a problem ifyou first figure out which layer is misbehaving 192 ping: check to seefahostis alive 64s Ask yourself questions like these as you work up (or down) the stack: + Do you have physical connectivity and a link light? + Isyour interface configured properly? + Do your ARP tables show other hosts? + Can you ping the localhost address (127.0.0.1)2 + Can you ping other local hosts by 1? address? + ISDNS working properly? + Can you ping other local hosts by hostname? + Can you ping hosts on another network? + Do high-level services ike web and SSH servers work? ‘Once you've identified where the problem les, takea step back and consider the effect your subsequent tests and prospective fixes will have on other services and hosts. 19.2. PING: CHECK TO SEE IF A HOST IS ALIVE ‘The ping command is embarrassingly simple, but in many situations itis all you need. It sends an ICMP ECHO_REQUEST packet to a target host and waits to see if the host answers back. Despite its simplicity, ping is one ofthe workhorses of net- ‘work debugging. ‘You can use ping to check the status of individual hosts and to test segments of the network. Routing tables, physical networks, and gateways ae all involved in process- ing a ping, so the network must be more or less working for ping to succeed. If ping ddoest't work, you can be pretty sure that nothing more sophisticated will work either However, this rule does not apply to networks that block ICMP echo requests with a firewall. Make sure that a firewall isn't interfering with your debugging before you conclude thatthe target host is ignoring a ping. You might consider disabling a med- dlesome firewall for a short period of time to facilitate debugging. pping runs in an infinite loop unless you supply a packet count argument. Once you've had your fil of pinging, type the interrupt character (usually ) to get out. Here's an example: 5 ping beast PING beast (10:1.146): 56 bytes of data 64 bytes from beast (10.1.1 46): iemp_se 64 bytes from beast (10.1.1 4): imp_st 64 bytes from beast (10.11.46): icmp_seq=2 nc beast ping statistics 43 packets transmitted, 3 received, 0% packet loss, time 2026me rt min/avg/man/mdey = 45,450/61.202/88,731/18.481 ms 1. your machine hangs aboot ine, boos very slowly, or hangs on inbound SSH connections, DNS shouldbe you prime suspect. Chapter 19 — Network Management and Debugging ‘The output for beast shows the hosts IP address, the ICMP sequence number ofeach response packet, and the round trip travel time. The most obvious thing thatthe out put above tells you is that the server beast is alive and connected to the network. Ona healthy network, ping can allow you to determine ifa host is down. Conversely, when a remote host is known to be up and in good working order, ping can give you useful information about the health of the network. Ping packets are routed by the usual IP mechanisms, and a successful round trip means that all networks and gate- ways lying between the source and destination are working correctly, at least toa first approximation, ‘The ICMP sequence number is a particularly valuable piece of information, Discon- tinuities in the sequence indicate dropped packets; they're normally accompanied bya message for each missing packer. Despite the fact that IP does not guarantee the delivery of packets, a healthy network should drop very few of them. Lost-packet problems are important to track down because they tend to be masked by higher- level protocols. The network may appeat to function correctly, but it willbe slower than it ought tobe, not only because ofthe retransmitted packets but also because of| the protocol overhead needed to detect and manage them, ‘To track down the cause of disappearing packets, first run traceroute (see the next section) to discover the route that packets are taking tothe target host. Then ping the intermediate gateways in sequence to discover which link is dropping packets. To pin ddown the problem, you need to senda statistically significant number of packets. The network fault generally lies on the link between thelast gateway that you can ping without significant loss of packets and the gateway beyond it. ‘The round trip time reported by ping gives you insight into the overall performance ofa path through a network. Moderate variations in round trip time do not usually indicate problems. Packets may occasionally be delayed by tens or hundreds of mil- liseconds for no apparent reason; thats just the way IP works. You should expect to seea faitly consistent round trp time for the majority of packets, with occasional lapses. Many of today’s routers implement rate-limited or lower-priority responses to ICMP packets, which means that a router may delay responding to your ping if it is already dealing with alot of other trafic. ‘The ping program can send echo request packets of any size, so by using a packet larger than the MTU of the network (1,500 bytes for Ethernet), you can force frag- ‘mentation, This practice helps you identify media errors or other low-level issues such as problems with a congested network or VPN. To specify the desired packet size in bytes, use the -s flag. S ping -s 1500 cuinfo.comnell edu Use the ping command with the following caveats in mind. First itis hard to distinguish the failure of network from the failure of a server with only the ping command. In an environment where ping tests normally work, 19.3 Seepage 396 for more Information about revere DNS lookups 19.3 traceroute: trace P packets a7 failed ping just tells you that something is wrong, (Network firewalls sometimes in- tentionally block ICMP packets.) Second, a successful ping does not guarantee much about the target machines state Echo request packets are handled within the IP protocol stack and do not require a server process to be running on the probed host. A response guarantees only that a machine is powered on and has not experienced a kernel panic, You'll ned higher- level methods to verily the availabilty of individual services such as HT'TP and DNS. ‘TRACEROUTE: TRACE IP PACKETS. traceroute, originally written by Van Jacobson, uncovers the sequence of gateways through which an IP packet travels to teach its destination, All madern operating systems come with some version of traceroute, The syntax is simply traceroute hostname ‘There ae a variety f options, most of which are not important in daily use. As usual, the hostname can be specified with eithera DNS name of an IP address. The output is simply alist of hosts, starting with the first gateway and ending atthe destination, Forexample,a traceroute from the host jaguar tothe host nubark produces the following output: 5 traceroute nubarke traceroute to nubark (182.1682.10), 30 hops max, 38 byte packets 1 lab-gw (172168254) 0.840 ms 0.693 me 0671 ms 2 dmz-pw (192.168.1254) 4642 ms 4.582 ms 4674 ms 3 nubark (182.168.210) 7.959 ms 5.949 ms 5.508 ms From this output we can tll that jaguar is exactly three hops away ftom nubark, and ‘we can see which gateways are involved in the connection. The round trip time for ‘each gateway isalso shown—three samples for each hop are measured and displayed. ‘A typical teaceroute between Internet hosts often includes more than 15 hops. traceroute works by setting the time-to-liv field (TTL, actually “hop count to live”) ‘of an outbound packet to an artificially low number. As packets arrive ata gateway, their TTL is decreased. When a gateway decreases the TTL to 0it discards the packet and sends an ICMP “time exceeded” message back to the originating host ‘The first three traceroute packets have their TTL set to 1. The first gateway to see ssuch a packet (lab-gw in this case) determines that the TTL has been exceeded and notifies jaguar ofthe dropped packet by sending back an ICMP message. The sender's IP address in the header ofthe error packet identifies the gateway; traceroute looks up this address in DNS to find the gateway’s hostname. ‘To identify the second-hop gateway, traceroute sends out a second round of packets ‘with TTL fields set to 2. The first gateway routes the packets and decreases their TTL. by 1 Atthe second gateway the packets are then dropped and ICMP error messages Chapter 19 — Network Management and Debugging are generated as before. This process continues until the TTL is equal to the number cof hops to the destination host and the packets reach their destination successfully. “Most routers send their ICMP messages from the interface “closest” to your host. It you run traceroute backwards from the destination host, you will probably see dif- ferent IP addresses being used to identify the same set of routers. You might also see completely diferent paths; this configuration is known as “asymmetric routing.” Since traceroute sends three packets for each value ofthe TTL field, you may some: times observe an interesting artifact. an intervening gateway multiplexes traffic across several routes, the packets might be returned by different hosts; in this case, traceroute simply prints them al. Let’ look at a more interesting example from a host at colorado.edu to xor.com: rupertsbergS traceroute xor.com traceroute to xor.com (192.225.23.1), 30 hops max, 38 byte packets 1 cs-gw3-faculty.cs.colorado.edu (128.138.736.3) 1362 ms 2144 ms 2.76 ms 2. ce-gw-dmz.ce colorado edu (128 138243 193) 2720 me 4.378 ms 5052 me 42 engr-ce Colorade£DU (128.138.8014) $587 ms 2454 ms 2773 ms 4 butengr Colorado ZDU (128.138.80.201) 2.743 me 5.643 ms 2.772 me 5 custm-gw.Colorado.EDU (128.138.80.2) $587 ms 2784 ms 2777 ms 6 204.131.625 (208.131.626) 5.585 ms 3.464 ms 2761 ms 7 border-from-BRAN.coop net (199.45.124.81) 5.593 me 6423 ms $521 me @ core-gweth-25.coop net (199.45.197.14) 53806 ms + 19202 ms 9 xor.com (192.22523.1) 16838 me 15372 ms 11.204 ms ‘This output shows that packets must traverse five internal gateways before leaving the colorado.edu network (cs-gw3-faculty to cuatm-gw). The next-hop gateway on the BRAN network (204.131.62.6) doesnt have a name in DNS. After two hops in coop.nel, we arrive at xorcom, Athop 8, we see star in place of one ofthe round trip times. This notation means that no response (error packet) was received in response to the probe. In this case, the cause is probably congestion, but that is not the only possibility. traceroute re- lies on low-priority ICMP packets, which many routers are smart enough to drop in preference to “real” trafic. A few stars shouldn't send you into a panic. Ifyou see stars in all the round trip time fields for a given gateway, no “time ex- ceeded” messages are arriving from that machine. Perhaps the gateway is simply down. Sometimes, a gateway or firewall is configured to silently discard packets with expired TTLs. In this case, you can still see through the silent host to the gate- ways beyond. Another possibilty is thatthe gateway’s error packets are slow to re- turn and that traceroute has stopped waiting for them by the time they arive Some firewalls block ICMP “time exceeded” messages entirely. Ione such firewall lies along the path, you wont get information about any ofthe gateways beyond it. However, you can still determine the total number of hops to the destination be- cause the probe packets eventually get al the way there. Also, some firewalls may 19.4 194 netstat: gt network tats 649 block the outbound UDP datagrams that traceroute sends to trigger the ICMP re- sponses, This problem causes traceroute to report no useful information at all ‘Aslow link does not necessarily indicate a malfunction, Some physical networks have ‘a naturally high latency; 802.11 wireless networks are a good example. Sluggishness ‘ean also bea sign of congestion on the receiving network, especially ifthe network uses a CSMA/CD technology that makes repeated attempts to transmit a packet Ethernet is one example). Inconsistent round trip times would support such a hy: pothesis, since collisions increase the randomness ofthe network's behavior. Sometimes, you may see the notation IN instead of a star or round trp time. It indi ‘cates that the current gateway sent back a “network unreachable” error, meaning that it doesn't know how to route your packet. Other possibilities include 'H for “host unreachable” and {P for “protocol unteachable” A gateway that returns any of these ‘error messages is usualy the last hop you can get to, That host usually has a outing, problem (possibly caused by a broken link) ether its static routes are wrong or dy- namie protocols have failed to propagate a usable route to the destination, eto traceroute doesn't seem to be working for you (or is working noticeably slowly), it may be timing out while trying to resolve the hostnames of gateways by using DNS IfDNS is broken on the host you are tracing from, use traceroute -n to request nu- meric output. This option prevents the use of DNS; it may be the only way to get traceroute to function on a crippled network. NETSTAT: GET NETWORK STATISTICS netstat collects a wealth of information about the state of your computers network- ing software, including interface statistics, routing information, and connection ta- bles. There isn’t really a unilying theme to the diferent sets of output, except that they all relate to the network. Think of netstat asthe “kitchen sink” of network tools—it exposes a variety of network information that doesn't fit anywhere else Here, we discuss the five most common uses of netstat Inspecting interface configuration information Monitoring the status of network connections dentifying listening network services Examining the routing table Viewing operational statistics for various network protocols Inspecting interface configuration information netstat -i displays information about the configuration and state of each ofthe hosts network interfaces. You can run netstat -i asa good way to familiarize yourself with ‘anew machine’ network setup. Add the -e option for additional details 650 Chapter 19 — Network Management and Debugging For example 5S netstat -i-e Kernel Interface table ethO Link encap:Ethemet HWaddr o0028219:C8:82 inet addr192.168 2.1 Beast-192.1682.295 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500. Metric-1 RX packets:1121827 errors’ dropped.0 overruns frameo ‘TX packets:1138477 errors-0 dropped:0 averruns.0 cartier collisions:0 txquewelen:100 Interrupt Base address-Oxef00 ethl Link encapsEthemet HWaddr 0002.82:19:¢6:86 inet addr192.168 1.13. Beast:192.168.1.255 Mask-295.255,255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric-1 RX packets/67543 errors:0 dropped:0 overruns:0 frame-0 ‘TX packets:69652 errors:0 dropped:0 overruns: carrien0 collisions:0 txqueuelen:100 Interrupts Base address-Oxed0o lo Link encap:Local Loopback inet addr127.0.01 Mask255000 UP LOOPBACK RUNNING MTU-3924 Metric: RX. packets:210572 errors:0 dropped:o overruns.0 frame:0 ‘TX packets:310572 errore:0 dropped:0 overruns‘ carrier0| collsions:0 txqueuelen:d ‘This host has two network interfaces: one for regular traffic plus a second connec- tion for system management named ethl. RX packets and TX packets report the ‘number of packets that have been received and transmitted on each interface since the machine was booted. Many different types of errors ate counted in the error buckets, and it is normal for a few to show up. Errors should be less than 1% of the associated packets. IFyour error rate is high, compace the rates of several neighboring machines. A large number of errors on a single machine suggests a problem with that machine’ interface or connection. A high error rate everywhere most likely indicates a media or network problem. One of the most common causes ofa high error rate is an Ethernet speed ot duplex mis- ‘match caused by a failure of autosensing or autonegotiation, Collisions suggest a loaded network; ervors often indicate cabling problems. Al- though a collision isa type of erzo, itis counted separately by netstat. The field labeled Collisions reports the numberof collisions that were experienced while packets were being sent* Use this number to calculate the percentage of output pack- ets (TX packets) that result in collisions. Ona properly functioning network, clli- sions should be less than 38 of output packets, and anything over 10% indicates serious congestion problems. Collisions should never occur on a fll-duplex ink that is operating properly. 2. Ths fied has meaning only on CSMA/CD-based networks uch 5 Eernet 194 netstat: gt network tats 6s “Monitoring the status of network connections ‘With no arguments, netstat displays the status of active TCP and UDP ports. Inac- Live (‘istening”) servers wating fr connections arent normally shown; they canbe seen with netstat -" The output looks like this $ netstat a Active Internet connections (servers and established) Proto Recv-Q Send.Q Local Address ForeignAddress state ep 0 0 “ldap = UsTEN ep 0 0 mysql os USTEN ep 0 0 wimape ae USTEN ep 00 suliseh Ahep-32hw:4208 ESTABLISHED tp 0 = 0——sbullimape —ubark:s4195.—-ESTABLISHED tp 0 = 0——sbullthttp——dhep-30hw:2563 ESTABLISHED tp 0 = 0—sbullimape —dhep-1ehw:2851 ESTABLISHED ep 0 0 sheep as USTEN tp 0) 0——sbul7203—baikalmysql «ESTABLISHED. ep 0 0 ssh = USTEN, ‘This example s from the host otter, and it has been severely pruned; for example, UDP and UNIX socket connections are not displayed. The output above shows an inbound SSH connection, two inbound IMAPS connections, one inbound HTTP ‘connection, an outbound MySQL connection, and a bunch of ports listening for other connections ‘Addresses ate shown as hostname.service, where the service isa port number. For ‘well-known services, netstat shows the port symbolically, using the mapping de- fined in the /ete/services file. You can obtain numeric addresses and ports with the “option. As with most network debugging tools, if your DNS is broken, netstat is ppainful to use without the -m flag. Send-Q and Recv-Q show the sizes of the send and receive queues for the connec- tion on the local host; the queue sizes on the other end of a TCP connection might be different. They should tend toward 0 and at least not be consistently nonzero. Of ‘course, ifyou are running netstat over a network terminal, the send queue for your ‘connection may never be 0, ‘The connection state has meaning only for TCP; UDP is a connectionless protocol ‘The most common states youll see are ESTABLISHED for currently active connec: tions, LISTEN for servers waiting for connections (not normally shown without -a), and TIME WAIT for connections in the process of closing. ‘This display is primarily useful for debugging higher-level problems once you have determined that basic networking facilities are working correctly. It ets you verily that servers are set up correctly and facilitates the diagnosis of certain types of mis- ‘communication, particularly with TCP. For example, a connection that stays in state ‘Connections for “UNIX doin sockets” ae als shown, but since they aren't elated to networking ‘we do not discus them here. 652 See Chapter 2for ‘mor information bout kernel taning Seepage 294 for more information about the routing ble Chapter 19 — Network Management and Debugging ‘SYN_SENT identifies a process that is trying to contact a nonexistent or inaccessible network server. I netstat shows a lot of connections in the SYN_WAIT condition, your host proba- bly cannot handle the number of connections being requested. This inadequacy ‘may be due to kernel tuning limitations or even to malicious flooding. Identifying listening network services (One common question in this security-conscious era is “What processes on this ma- chine are listening on the network for incoming connections?” netstat -a shows all the ports that are actively listening (any TCP port in state LISTEN, and potentially any UDP port), but on a busy machine those lines can get lost in the noise of estab- lished TCP connections. Use netstat -1to see only the listening ports. The output format isthe same as for netstat -a You can add the p flag to make netstat identi the specific process associated with each listening port The sample output below shows the common services (sshd, sendmail, and named), followed by an unusual one: S netstat -Ip tp 08 0.00022 0000+ LISTEN 23858/ssha tep «080.0025 0.00.0 LISTEN 10342/sendmail udp 0 © 0 000053 000g 30016/named udp 0 ©0- 0.000362 0000 38721/mudd Here, mudd with PID 38221 is listening on UDP port 962. Depending on your site's policy regarding user-insalled software, you might want to follow up om this one. Examining the routing table netstat r displays the kernel’ routing table, The following sample is from a Red Hat ‘machine with two network interfaces. (The output varies slightly among Linux dis- tributions.) S netstat -rm Kernel IP routing table Destination Gateway Genmask lage MSS Window itt face 19216810 0000 255.255.2550 U0 0 © etho 10250 0900 255.255.2550 U 0 0 0 etht 17000 0900 258000 «U0 0 0 lo 0000 192.168.1254 00.00 uc 00 40 etho Destinations and gateways can be displayed either as hostnames or as IP addresses; the -n flag requests numeric output. ‘The Flags characterize the route: U means up (active), G isa gateway, and His a host route. U, G,and H together indicate a host route that passes through an intermediate gateway. The D flag (not shown) indicates a route resulting from an ICMP redirect. 194 netstat: gt network tats 653 ‘The remaining fields give statistics on the route: the current number of TCP connec- tions using the route, the number of packets sent, and the interface used Use this form of netstat to check the health of your system's routing table. It’s partic. ularly important to verify that the system has a default route and that this route is, correct, The default route is represented by an all-0 destination address (0.0.0.0). t is possible not to have a default route entry, but such a configuration would be highly atypical, Viewing operational statistics for network protocols ‘netstat -s dumps the contents of counters that are scattered throughout the network. code. The output has separate sections for IP, ICMP, TCP, and UDP. Below are pieces ‘of netstat-s output from a typical server; they have been edited to show only the tastiest pieces of information, e 671349985 total packets received 0 forwarded 245 incoming packets discarded 6667912993 incoming packets delivered 589623972 requests sent out 60 dropped because of missing route 203 fragments dropped after timeout 2 2 Be sure to check that packets are not being dropped or discarded. It is acceptable for 4 few incoming packets to be discarded, but a quick rise inthis metric usually indi- ‘cates a memory shortage or some other resource problem, temp: 242023 ICMP messages received 512 input ICMP message failed ICMP input histogram: destination unreachable: 72120, timeout in transit: 973 echo requests: 17135 echo replies: 152195 66049 ICMP messages sent O ICMP messages failed ICMP output histogram: destination unreachable: 48914 echo replies: 17135, In this example, the number of echo requests inthe input section matches the num- ber of echo replies in the output section. Note that “destination unreachable” mes- ‘sages can still be generated even when all packets are apparently forwardable. Bad packets eventually reach a gateway that rejects them, and error messages are then sent back along the gateway chain. 654 19.5 Chapter 19 — Network Management and Debugging Tep ‘4402720 active connections openings 1023085 passive connection openings 50399 failed connection attempts 0 connection resets received 444 connections established 666674854 segments received 85111784 segments eend out 107368 segments retransmited 6 bad segments received, 2047240 resets sent udp: 14395827 packets received 21586 packets to unknown port received 0 packet receive errors 4289260 packets sent Wsa good idea to develop a feel for the normal ranges of these statistics so that you can recognize pathological states. SAR: INSPECT LIVE INTERFACE ACTIVITY One good way to identify network problems is to look at what's happening right now, How many packets were sent in the ast five minutes on a given interface? How ‘many bytes? Are collisions or other errors occurring? You can answer all these ques- tions by watching live interface activity On traditional UNIX systems, netstat -iis the tool of choice for ths role. Unfortu- nately, netstat’ ability to report on live interface activity is broken under Linux. We recommend a completely different took: sar. (We discuss sar from the perspective of| ‘general system monitoring on page 816.) Most distributions don't install sar by de- fault, bu it’ always available as an optional package. ‘To make sar report on interface activity every two seconds for a period of one ‘minute (Le. 30 reports), use the syntax sar -n DEV 230. The DEV argument isa literal keyword, not a placeholder for a device o interface name ‘The output includes instantaneous and average readings of network interface utili- zation in terms of bytes and packets. The sample below is from a Red Hat machine with two physical interfaces. The second physical interface (eth) is clearly not in us. 175043 TFACE rapel/s tupek/s ribyt/s tebyt/s remp/s teemp/s remest/s a7so4s lo 361361 26340 26240 000 0.00 000 a7s045 eth 1856 1186 136043 149433 000 000 0.82 a7so4s etht 000 000 0.00 000 000 000 0.00 ‘The irst two columns state the time at which the data was sampled and the names of the network interfaces. The next two columns show the number of packets received and transmitted, respectively. The rebyt/s and txbyt/s columns are probably the ‘most useful since they show the actual bandwidth in use, The final three columns give statistics on compressed (rxcrnp/s, txcmp/s) and multicast (ramest/s) packets. 19.6 ‘Seepage 355 for mare Information about net work switches, 196 Packet sifers 655 ‘sar-n DEV is especially useful for tracking down the source of errors. ifconfig can alert you tothe existence of problems, bu it cat tell you whether the errors came from a continuous, low-level problem or from a brief but catastrophic event. Ob- serve the network over time under a variety of load conditions to solidify your im- pression of what’ going on. Try running ping witha large packet payload (size) ‘while you watch the output of sar -n DEV. PACKET SNIFFERS tepdump and Wireshark belong to a class of tools known as packet sniffer. They listen to the traffic ona network and record or print packets that meet certain cite- ria specified by the user, For example, all packets sent to or from a particular host of ‘TCP packets related to one particular network connection could be inspected, Packet sniffers ate useful both for solving problems you know about and for discov ‘ering entirely new problems. Isa good idea to take an occasional sniff of your net- ‘work to make sure the trafic is in order. Packet sniffers need to be able to intercept traffic thatthe local machine would not normally receive (or at leas, pay attention to), so the underlying network hardware must allow access to every packet. Broadcast technologies such as Ethernet work fine, as do most other modern local area networks. Since packet sniffers need to see as much ofthe raw network traffic as possible, they ‘can be thwarted by network switches, which by design try to limit the propagation ‘of “unnecessary” packets. However, it can still be informative to try out a sniffer on a switched network. You may discover problems related to broadcast or multicast packets. Depending on your switch vendor, you may he surprised at how much traf- fic you can see. ‘The interface hardwace, in addition to having potential access tall network pack- ets, must transport those packets up to the software layer. Packet addresses are nor- mally checked in hardware, and only broadcast/multicast packets and those ad- dressed to the local host are relayed to the kernel. In “promiscuous mode,” an interface lets the kernel ead all packets on the network, even the ones intended for other hosts Packet sniffers understand many of the packet formats used by standard network services, and they can often print these packets in a human-readable form. This ca- pability makes it easier to track the flow of a conversation between two programs. Sonne sniffers print the ASCII contents of a packet in addition to the packet header and so are useful for investigating high-layer protocols. Since some ofthese proto- cols send information (and even passwords) across the network as cleartext, you ‘must take care not to invade the privacy of your users. Each of our example distributions comes with a packet sniffer. A sniffer must read data from a raw network device, so it must run as root. Although the root limitation serves to decrease the chance that normal users will listen in on your network trafti, eto Chapter 19 — Network Management and Debugging itis really not much of abattier. Some sites choose to remove snifler programs from ‘most hosts to reduce the chance of abuse. Ifnothing else, you should check your systems’ interfaces to be sure they are not running in promiscuous mode without your knoviledge or consent. On Linux systems, an interface in promiscuous mode shows the flag PROMISC in its ifconfig status output. You can also use tools such as PromiScan (available from www-securityfriday.com) to check your network for in- terfaces running in promiscuous mode. ‘tcpdump: king of sniffers tepdump, yet another amazing network tool by Van Jacobson, is included in most Linux distributions. cepdump has long been the industry-standard sniffer; most other network analysis tools read and write trace files in “tepdump format” By default, tepdump tunes in on the first network interface it comes across, Ifit chooses the wrong interface, you can force an interface with the -i flag. DNS is broken or you just don’t want tepdump doing namelookups, use the -n option. This ‘option is important because slow DNS service can cause the filter to statt dropping packets before they can be dealt with by tepdump. The -v flag increases the informa- tion you see about packets, and -vy gives you even more data. Finally tepdump can store packets toa file with the -w flag and can read them back in with the- flag. For example, the following truncated output comes from the machine named rnubark. The filter specification host bull limits the display of packets to those that directly involve the machine bull, either as source or a destination. # sudo tcpdump host bull 412,35:23,519339 bull 41537 > nubarkdomain: A? atrust.com. (28) (DF) 12,35:23,519961 nubarkdomain > bull 41837: A 66,77.122.161 (112) (DF) ‘The rst packet shows the host bull sending a DNS lookup request about atrust.com to nubark. The response is the IP address of the machine associated with that name, Which is 66.77.122.161. Note the time stamp on the left and tepdump’ understand- ing ofthe application-layer protocol (in this case, DNS). The port number on bull is arbitrary and is shown numerically (41537), but since the server port number (53) is well known, tepdump shows its symbolic name ("domain") instead Packet sniffers can produce an overwhelming amount of information—overwhelm- ing not only for you but also for the underlying operating system. To avoid this problem on busy networks, tepdump lets you specify fairly complex filters. For ex- ample, the following filter collects only incoming web traffic from a specific subnet: # sudo tepdump sre net 192.168.1.0/24 and dst port 80 “The tepdump man page contains several good examples of advanced filtering along with a complete listing of primitives. your sitering needs exceed tepdump’ capabilites, consider ngrep which can fe packet acco ingto thee coments, 19:7_Network management protocols 657 Wireshark: visual sniffer you're more inclined to use a point-and-click program for packet sniffing, then ‘Wireshark may be for you. Available under the GNU General Public License from ‘www.vireshark.org, Wireshark isa GTK+ (GIMP tool kit)-based GUI packet sniffer that has more functionality than most commercial sniffing products. You can run ‘Wireshark on your Linux desktop, orf your laptop is still painfully suffering in the dark ages of Windows, you can download binaries for that too, In addition to sniffing packets, Wiresharkhas a couple of features that make it extra ‘handy: One nice feature is that Wireshark can read and write a large number of other packet trace file formats, including (but not limited to) + TCPDUMP «+ NALS Sniffer «+ Sniffer Pro + NetXray + Snoop eto Shomiti Surveyor Microsoft’ Network Monitor Novell’ LANalyzer Cisco Secure IDS iplog ‘The second extra-handy feature is that you can click on one packet in a TCP stream and ask Wireshark to “reassemble” (splice together) the payload data ofall the pack- ets in the stream. This feature is useful if you want to quickly examine the data transferred during a complete TCP conversation, such as a connection carrying an ‘email message across the network.* ‘Wireshark has capture filters, which function identically to tepdump's. Watch out, though—one important gotcha with Wireshark isthe added feature of “display fil- ters? which affect what you see rather than whar’s actually captured by the sniffer. (Oddly, display filters use an entirely different syntax from capture filter. ‘Wireshark isan incredibly powerful analysis tool and is included in almost every networking experts too! kit. Moreover it’ also an invaluable learning ad for those just beginning to explore packet networking. Wireshark’s help menu provides many ‘great examples to get you started. Don't he afraid to experiment! 19.7 NETWORK MANAGEMENT PROTOCOLS Networks have grown rapidly in size and value over the last decade, and along with that growth has come the need for an elficient way to manage them, Commercial vendors and standards organizations have approached this challenge in many dilfer- ‘ent ways. The most significant developments have been the introduction of several standard device management protocols and a glut of high-level products that exploit those protocols ‘5, You can use the teplow uy to perform sr fat on the command lin from a tepdamp race Chapter 19 — Network Management and Debugging Network management protocols standardize the method of probing a device to dis- cover its configuration, health, and network connections. In addition, they allow some of this information to be modilied so that network management can be stan- dardized across different kinds of machinery and performed from a central location, ‘The most common management protocol used with TCP/IP is the Simple Network Management Protocol, SNMP. Despite its name, SNMP is actually quite complex. It defines a hierarchical namespace of management data and a way to read and write the data at each node. It also defines a way for managed servers and devices agents”) to send event notification messages (“traps”) to management stations. ‘The SNMP protocol iselfis simple; most of NMP's complexity les above the proto- collayerin the conventions for constructing the namespace and in the unnecessarily baroque vocabulary that surrounds SNMP like a protective shell. As long as you don't think too hard about its internal mechanics, SNMP is easy to use. Several other standards ate loating around out there. Many of them originate from the Distributed Management Task Force (DMTF), which is responsible for concepts such as WBEM (Web-Based Enterprise Management), DMI (Desktop Management Interface), and the CIM (Conceptual Interface Model). Some ofthese concepts, par- ticularly DMI, have been embraced by several major vendors and may become a use- ful complement to (or even a replacement for) SNMP. Many proprietary management protocols are also afloat out there. Fr now, however, the vast majority of network and Linux system management takes place over SNMP. Since SNMP is only an abstract protocol, you need both a server program (“agent”) and a client (“manager”) to make use of it (Perhaps counterintutively the server side of SNMP represents the thing being managed, and the client side is the man- ager.) Clients range from simple command-line utilities to dedicated management sfations that graphically display networks and faults in eye-popping color. Dedicated network management stations are the primary reason for the existence of ‘management protocols. Most products let you build a topographic model of the net work as well as logical model; the two are presented together on-screen, along with ‘ continuous indication of the status of each component. Just asa chart can reveal the hidden meaning in a page of numbers, a network man- agement station can summarize the state ofa large network in a way that’s easily ac- cepted by a human brain. This kind of executive summary is almost impossible to get any other way: A major advantage of management-by-protocol is that it promotes all kinds of net- work hardware onto a level playing field. Linux systems ate all basically similar, bat routers, switches, and other low-level components are not. With SNMP, they all speak 1 common language and can be probed, reset, and configured from a central loca- tion. I's nice to have one consistent interface to all the network’s hardware, 19.8 198 SNMP: the Simple Network Management Protocol 659 ‘SNMP: THE SIMPLE NETWORK MANAGEMENT PROTOCOL ‘When SNMP first became widely used in the early 1990s, it started a mini gold rush. Hundreds of companies have come out with SNMP management packages. Also, many hardware and software vendors ship an SNMP agent as part of their product. Before we dive into the gritty details of SNMP, we should note that the terminology associated with itis some ofthe most wretched technobabble to be found inthe net ‘working arena. The standard names for SNMP concepts and objects actively lead you away from an understanding of what's going on. The people responsible for this tate ‘of affairs should have their keyboards smashed. ‘SNMP organization ‘SNMP data is arranged ina standardized hierarchy: This enforced organization al- lows the data space to remain both universal and extensible, at least in theory. Large portions are set aside for future expansion, and vendor-specific additions are local- ized to prevent conflicts. The naming hierarchy is made up of “Management Infor: ‘mation Bases” (MIBs), structured text files that describe the data accessible through ‘SNMP. MIBs contain descriptions of specific data variables, which ate referred to with names known as object identifiers, or OIDs. ‘Translated into English, this means that SNMP defines a hierarchical namespace of variables whose values are tied to “interesting” parameters ofthe system. An OID is justa fancy way of naming a specific managed piece of information. ‘The SNMP hierarchy is very much like a filesystem. However, a dot is used as the separator character, and each node is given a number rather than.a name. By conven- tion, nodes are also given text names for ease of reference, but this naming is really justa high-level convenience and nota feature of the hierarchy (its similar in prin- ciple to the mapping of hostnames to IP addresses). Forexample, the OID that refers tothe uptime ofthe system is 1 3.6.1.2.1.1.3. This OID is also known by the human readable name iso.orgdod internet mgmt.mib-2.system.sysUpTime ‘The top levels of the SNMP hierarchy are political artifacts and generally do not con- tain useful data Infact, useful data can currently be found only beneath the OID. iso.org dod internet.mgmt (numerically 1.3.6.1.2) ‘The basic SNMP MIB for TCP/IP (MIB-1) defines access to common management data: information about the system, its interfaces, address translation, and protocol ‘operations (IP, ICMP, TCP, UDP, and others). later and more complete reworking of this MIB (called MIB-II) is defined in RFC1213. Most vendors that provide an. ‘SNMP server support MIB-lL Table 19.1 on the next page presents a sampling of nodes from the MIB-II namespace. Chapter 19 — Network Management and Debugging Table 19.1 Selected O1Ds from MIB-II OID Type _ Contents ‘ystemsysDescr___string System info: vendor, model, OS type, et. systemsyslocation string Physical locaton ofthe machine systemsysContact_ string Contact info forthe machine's owner systemsysName string System name, usualy the full NS name interfaces.ifNumber int Number of network interfaces present interfacesiable table Table of nfobits about each interface ipipForwarding it Tifsystemisa gateway; otherwise ipipAddable table Table of IP addressing data (masks, etc) ipipRouteTable table The system’ routing table iempicmpinnedirects int Number of ICMP redirects received iempicmplnchos int. Number of pings received teptepConnTable table Table of current TCP connections udp.udpTable table Table of UDP sockets with serves listening 2 Relatvetwisoorgdedintemetmamtmib2 In addition to the basic MIB, there are MIBs for various kinds of hardware interfaces and protocols, MIBs for individual vendors, and MIBs for particular hardware prod- ucts. A MIB for you, a MIB for me, catch that MIB behind the tree, A MIB is only a convention about the naming of management data, To be useful, a MIB must be backed up with agent-side code that maps between the SNMP name- space and the device’ actual state. Code for the basic MIB (now MIB-II) comes with the standard Linux agent. Some agents are extensible to include supplemental MIBs, and some are not SNMP protocol operations There are only four basic SNMP operations: get, get-next, set, and trap. Get and set are the basic operations for reading and writing data to a node identified bya specific OID. Get-next steps through a MIB hierarchy and can read the contents of tables as well A trap isan unsolicited, asynchronous notification from server (agent) to client (manager) that reports the occurrence of an interesting event or condition. Several standard traps are defined, including “Te just come up” notifications, eports of fai ure or recovery ofa network link, and announcements of various routing and au- thentication problems. Many other not-so-standard traps are in common use, in- cluding some that simply watch the values of other SNMP variables and fire off message when a specified range is exceeded. The mechanism by which the destina- tions of trap messages are specified depends on the implementation of the agent. Since SNMP messages can potentially modily configuration information, some se- curity mechanism is needed. The simplest version of SNMP security is based on the

You might also like