ProcmailMiniTutorial:

AutomatedMailHandling
byJimDennis,Proprietor,StarshineTechnicalServices
ConvertedtoHTMLbyHeatherStern

procmailisthemailprocessingutilitylanguagewrittenbyStephenvandenBergofGermany.Thisarticleprovidesabit
ofbackgroundfortheintermediateUnixuseronhowtouseprocmail.
Asa"little"language(tousetheacademicterm)procmaillacksmanyofthefeaturesandconstructsoftraditional,
generalpurposelanguages.Ithasno"while"or"for"loops.Howeverit"knows"alotaboutUnixmaildelivery
conventionsandfile/directorypermissionsandinparticularaboutfilelocking.
Althoughitispossibletowriteacustommailfilteringscriptinanyprogramminglanguageusingthefacilitiesinstalled
onmostUnixsystemswe'llshowthatprocmailisthetoolofchoiceamongsysadminsandadvancedUnixusers.
UnixmailsystemsconsistofMTA's(mailtransportagentslikesendmail,smail,qmailmmdfetc),MDA's(delivery
agentslikesendmail,deliver,andprocmail),andMUA's(useragentslikeelm,pine,/bin/mail,mh,Eudora,and
Pegasus).
OnmostUnixsystemsontheInternetsendmailisusedasanintegratedtransportanddeliveryagent.sendmailand
compatibleMTA'shavetheabilitytodispatchmail*through*acustomfilterorprogramthrougheitheroftwo
mechanisms:aliasesand.forwards.
Thealiasesmechanismusesasinglefile(usually/etc/aliasesor/usr/lib/aliases)toredirectmail.Thisfileisowned
andmaintainedbythesystemadministrator.Thereforeyou(asauser)can'tmodifyit.
The".forward"mechanismisdecentralized.Eachuseronasystemcancreateafileintheirhomedirectory
named.forwardandconsistingofanaddress,afilename,oraprogram(filter).Usuallythefile*must*beownedbythe
userorrootand*mustnot*be"writeable"byotherusers(goodversionsofsendmailcheckthesefactorsforsecurity
reasons).
It'salsopossible,withsomeversionsofsendmail,foryoutospecifymultipleaddresses,programs,orfiles,separated
withcommas.Howeverwe'llskipthedetailsofthat.
Youcouldforwardyourmailthroughanyarbitraryprogramwitha.forwardthatconsistedofalinelike:
"|$HOME/bin/your.programandsomearguments"

Notethequotesandthe"pipe"character.Theyarerequired.
"Your.program"couldbeaBourneshellscript,anawkorperlscript,acompiledCprogramoranyothersortoffilter
youwantedtowrite.
However"your.program"wouldhavetobewrittentohandleaplethoraofdetailsabouthowsendmailwouldpassthe
messages(headersandbody)toit,howyouwouldreturnvaluestosendmail,howyou'dhandlefilelocking(incasemail
cameinwhile"your.program"wasstillprocessingone,etc).
That'swhatprocmailgivesus.
WhatI'vediscussedsofaristhegeneralinformationthatappliestoallsendmailcompatibleMTA/MDA's.
So,toensurethatmailispassedtoprocmailforprocessingthefirststepistocreatethe.forwardfile.(Thisissafetodo
beforeyoudoanyconfigurationofprocmailitselfassumingthatthepackage'sbinariesareinstalled).Here'sthe
canonicalexample,pastedfromtheprocmailmanpages:
"|IFS=''&&exec/usr/local/bin/procmailf||exit75#YOUR_USERNAME"

Thisseemsawfullycomplicatedcomparedtomyearlierexample.That'sbecausemyexamplewasflawedfor
simplicity'ssake.

Whatthismessmeanstosendmail(paraphrasingintoEnglish)is:
Pipethemailtothefollowingcommand(s):
Settheshell's"interfieldseperator"(IFS)toaspace,andifthatwentO.K.(&&)executetheprogramnamed
"/usr/local/bin/procmail"
(yoursmayneedtobedifferenttrythecommand'whichprocmail'toseeifit'sonthepathor'locateprocmail'
ifyoursystemmaintainsafilelocatordatabase).
Theprocmailprogramisbeingpassedasetofswitches:"f"whichtellsitto"updatetimestampintheleading
the'From'lineintheheader"
(thislastbitisratherobscureandhastodowithhowmessagesarenormallystoredinyour"incoming"ormailfile
or"spool"asweUnixhacksliketocallit).
Thenextpartofthis.forwardcommandistheBourneshell's"||"operatorwhichisbasicallyacontinuationfrom
the"and"(&&)operatorthatweusedbefore.Itsays"or"(ifthatcommanddidn'tworki.e.itreturnedanyerror)
then"exit"(stopprocessing)andreturnanerrornumber75(whichwepresumeismeaningfultosendmailthe
programthatcalledthiscommand).
Thelastpartofthis.forwardexpressionisacommentwhichaccordingtothemanpages:
"isnotactuallyaparameterthatisrequiredbyprocmail,infact,itwillbediscardedbyshbefore
procmaileverseesititishoweveranecessarykludgeagainstoveroptimisingsendmailprograms:"
YoushouldjustchangethephraseYOUR_NAMEtoyourloginnameonthatsystem.
Thiscomplicatedlinecanbejustpastedintomost.forwardfiles,minimallyeditedandforgotten.
Ifyoudidthisandnothingelseyourmailwouldbasicallybeunaffected.procmailwouldjustlookforitsdefaultrecipe
file(.procmailrc)andfindingnoneitwouldperformitsdefaultactiononeachmessages.Inotherwordsitwould
appendnewmessagesintoyournormalspoolfile.
IfyourISPusesprocmailasitslocaldeliveryagentthenyoucanskipthewholepartofaboutusingthe.forwardfile
oryoucanuseitanyway.
Ineithereventthenextsteptoautomatingyourmailhandlingistocreatea.procmailrcfileinyourhomedirectory.You
couldactuallycallthisfileanythingyouwantedbutthenyou'dhavetoslipthenameexplicitlyintothe.forwardfile
(rightbeforethe"||"operator).Almosteveryonejustusesthedefault.
Nowwecangettoaspecificexample.Sofarallwe'vetalkedaboutithoweverythinggetsroutedtoprocmailwhich
mostlyinvolvessendmailandtheBourneshell'ssyntax.Almostallsendmail'sareconfiguredtouse/bin/sh(theBourne
shell)tointerpretaliasand.forward"pipes."
So,here'saverysimple.procmailrcfile:
:0c:
$HOME/mail.backup

Thisjustappendsanextracopyofallincomingmailtoafilenamed"mail.backup"inyourhomedirectory.
Notethatabunchofenvironmentvariablesarepresetforyou.It'sbeensuggestedthatyoushouldexplicity
setSHELL=/bin/sh(ortheclosestderivativetoBourneShellavailableonyoursystem).I'veneverhadtoworryaboutthat
sincetheshellsIuseonmostsystemsarealreadyBournecompatible.
However,cshandothershellusersshouldtakenotethatalloftheprocmailrecipeexamplesthatI'veeverseenuse
Bournesyntax.
The:0linemarksthebeginningofa"recipe"(procedure,clause,whatever.:0canbefollowedbeanyofanumberof
"flags."Thereisaliterallydizzyingnumberofwaystocombinetheseflags.Theoneflagwe'reusinginthisexampleis
'c'for"copy."
Youmightaskwhytherecipestartswitha:0.Historicallyyouusedtouse:x(wherexwasanumber).Thiswasahintto
procmailthatthenextxlineswereconditionsforthisrecipe.Later,theoptionwasaddedtoprecedeconditionswitha
leadingasterisksotheydidn'thavetobemanuallycounted.:0thencametomeansomethinglike:"countthem
yourself."

Thesecondcolononthislinemarkstheendoftheflagsandthebeginningofthenameforalockfile.Sincenonameis
givenprocmailwillpickoneautomatically.
Thisbitisalittlecomplicated.Mailmightarriveinbursts.Ifanewmessagearriveswhileyourscriptisstillbusy
processingthelastmessageyou'llhavemultiplesendmailprocesses.Eachwillbedealingwithonemessage.Thisisn't
aproblembyitself.Howeverifthetwoprocessesmighttrytowriteintoonefileatthesametimetheyarelikelytoget
jumbledinunpredictableways(theresultwillnotbeaproperlyformattedmailfolder).
Sowehinttoprocmailthatitwillneedthecheckforandcreatealockfile.Inthisparticularcasewedon'tcarewhatthe
nameofthelockfilewouldbe(sincewe'renotgoingtohave*other*programswritingintothebackupfile).Sowe
leavethelastfield(afterthecolon)blank.procmailwillthenselectitsownlockfilename.
Ifweleavethe:offoftherecipeheaderline(ommittingthelastfieldentirely)thennolockfileisused.
Thisisappropriatewheneverweintendtoonlyreadfromthefilesintherecipeorincaseswhereweintendtoonly
writeshort,singlelineentriestoafileinnoparticularorder(likelogfileentries).
Thewayprocmailworksis:
Itreceivesasinglemessagefromsendmail(orsomesendmailcompatibleMTA/MDA).Theremaybeseveralprocmail
processingrunningcurrentlysincenewmessagesmaybecominginfasterthantheyarebeingprocessed.
Itopensitsrecipefile(.procmailrcbydefaultorspecifiedonitscommandline)andparseseachrecipefromthefirstto
thelastuntilamessagehasbeen"delivered"(or"disposedof"asthecasemaybe).
Anyrecipecanbea"disposition"or"delivery"ofthemessage.Assoonasamessageis"delivered"thenprocmailcloses
itsfiles,removesitslocksandexits.
Ifprocmailreachestheendofit'srcfile(andthusalloftheINCLUDE'dfiles)without"disposing"ofthemessage
thanthemessageisappendedtoyourspoolfile(whichlookslikeanormaldeliverytoyouandallofyour"mailuser
agents"likeEudora,elm,etc).
Thisexplainswhyprocmailissoforgivingifyouhave*no*.procmailrc.Itsimplydeliversyourmessagetothespool
becauseithasreachedtheendofallitsrecipes(therewerenone).
The'c'flagcausesarecipetoworkona"copy"ofthemessagemeaningthatanyactionstakenbythatrecipearenot
consideredtobe"dispositions"ofthemessage.
Withoutthe'c'flagthisrecipewouldcatchallincomingmessages,andallyourmailwouldendupinmail.backup.None
ofitwouldgetintoyourspoolfileandnoneoftheotherrecipeswouldbeparsed.
Thenextlineinthissamplerecipeissimplyafilename.Likesendmail'saliasesand.forwardfilesprocmailrecognizes
threesortsofdispositiontoanymessage.Youcanappendittoafile,forwardittosomeothermailaddress,orfilterit
throughaprogram.
Actuallythereisonespecialformof"delivery"or"disposition"thatprocmailhandles.Ifyouprovideitwithadirectory
name(ratherthanafilename)itwilladdthemessagetothatdirectoryasaseparatefile.Thenameofthatfilewillbe
basedonseveralrathercomplicatedfactorsthatyoudon'thavetoworryaboutunlessyouusetheRandMHsystem,or
someotherrelativelyobscureand"exotic"mailagent.
Aprocmailrecipegenerallyconsistsofthreepartsastartline(:0withsomeflags)someconditions(linesstartingwith
a'*'asteriskcharacter)andone"delivery"linewhichcanbefile/directorynameoralinestartingwitha'!'bang
characterora'|'pipecharacter.
Here'sanotherexample:
:0
*^From.*someone.i.dont.like@somewhere.org
/dev/null

Thisisasimpleoneconsistingofnoflags,oneconditionandasimplefiledelivery.Itsimplythrowsawayanymailfrom
"someoneIdon'tlike."(/dev/nullunderUnixisa"bitbucket"abottomlesswellfortossingunwantedoutputDOS
hasasimilarconceptbutit'snotnearlyashandy).

Here'samorecomplexone:
:0
*!^FROM_DAEMON
*!^FROM_MAILER
*!^XLoop:myaddress@myhost.mydomain.org
|$HOME/bin/my.script

Thisconsistsofasetofnegativeconditions(noticethattheconditionsallstartwiththe'!'character).Thismeans:for
anymailthatdidn'tcomefroma"daemon"(someautomatedprocess)anddidn'tcomea"mailer"(someotherautomated
process)andwhichdoesn'tcontainanyheaderlineoftheform:"XLoop:myadd..."senditthroughthescriptinmybin
directory.
Icanputthescriptdirectlyinthercfile(whichiswhatmostprocmailusersdomostofthetime).Thisscriptmightdo
anythingtothemail.Inthiscasewhateveritdoeshadbetterbegoodbecauseprocmailwaywillconsideranysuch
mailtobedeliveredandanyrecipesafterthiswillonlybereachedbymailfromDAEMONs,MAILERsandanymailwiththat
particularXLoop:lineintheheader.
ThesetwoparticularFROM_conditionsareactually"special."Theyarepresetbyprocmailandactuallyrefertoacoupleof
rathercomplicatedregularexpressionsthataretailoredtomatchthesortsofthingsthatarefoundintheheadersofmost
mailfromdaemonsandmailers.
TheXLoop:lineisanormalprocmailcondition.IntheRFC822document(whichdefineswhatemailheadersshould
looklikeontheInternet)anylinestartedwithXisa"custom"header.Thismeansthatanymailprogramthatwantsto
canaddprettymuchanyXlineitwants.
AcommonprocmailidiomistoaddanXLoop:linetotheheaderofanymessagethatwesendoutandtocheckfor
ourownXLoop:linebeforesendingoutanything.Thisistoprotectagainst"mailloops"situationswhereourmail
getsforwardedor"bounced"backtousandweendlesslyrespondtoit.
So,here'sadetailedexampleofhowtouseprocmailtoautomaticallyrespondtomailfromaparticularperson.Westart
withtherecipeheader.
:0

...thenweaddouronecondition(thatthemailappearstobefromthepersoninquestion):
*^FROMharasser@spamhome.com
FROMisa"magic"valueforprocmailitchecksfrom,resentby,andsimilarheaderlines.Youcouldalsouse^From:
whichwouldonlymatchtheheaderline(s)thatstartwiththestring"From:"

The^(hiccupor,moretechnically"caret")isa"regularexpressionanchor"(atechiephrasethatmeans"itspecifies
*where*thepatternmustbefoundinordertomatch."Thereisawholebookonregularexpression(O'Reilly&
Associates)."regexes"permeatemanyUnixutilities,scriptinglanguages,andotherprograms.Thereareslight
differencesin"regex"syntaxforeachapplication.Howeverthemanpagefor'grep'or'egrep'isanexcellentplaceto
learnmore.
Inthiscasethehiccupmeansthatthepatternmustoccuratthebeginningofaline(whichisitsusualmeaningingrep,
ed/sed,awk,andothercontexts).
...andweaddacoupleofconditionstoavoidloopingandtoavoidrespondingtoautomatedsystems
*!^FROM_DAEMON
*!^FROM_MAILER

(Theseareacouplemore"magic"values.Themanpagesshowtheexactregexesthatareassignedtothesekeywords
ifyou'recuriousorneedtotweakaspecialconditionthatissimilartooneortheotherofthese).
...andonemoretopreventsometrickyloop:
*!^XLoop:myaddress@myhost.mydomain.org

(Allofthesepatternsstartwith"bangs"(exclammationpoints)becausetheconditionisthat*no*lineoftheheaderstart
withanyofthesepatterns.The'bang'inthiscase(andmostotherregexcontexts)"negates"or"reverses"themeaningof
thepattern).

...nowweadda"disposition"theautoresponse.
|(formailrk\
A"XLoop:yourname@youraddress.com"\
A"Precendence:junk";\
echo"Pleasedon'tsendmeanymoremail";\
echo"Thisisanautomatedresponse";\
echo"I'llneverseeyourmessage";\
echo"So,GOAWAY")|$SENDMAILtoi

Thisisprettycomplicatedbuthere'showitworks:
Thepipecharactertellsprocmailthatitshouldlaunchaprogramandfeedthemessagetoit.
TheopenparenthesisisaBourneshellconstructthatgroupsasetofcommandsinsuchawayastocombinethe
outputfromallofthemintoone"stream."We'llexplainthismorelater.
The'formail'commandisahandyprogramthatisincludedwiththeprocmailpackage.It"formats"mailheaders
accordingtoitscommandlineswitchesanditsinput.
rktells'formail'toformata"reply"andto"keep"themessagebody.Withtheseswitchesformailexpectsa
headerandbodyasinput.
TheAparameterstellsformailto"add"thenextparameterasaheaderline.Theparametersprovidedto
theAswitchmustbeenclosedinquotessotheshelltreatsthewholestring(spacesandall)assingle
parameters.
Thebackslashesattheendofeachlinetellprocmailmailtotreatthenextlineaspartofthisone.So,allof
thelinesendinginbackslashesarepassedtotheshellasonelongline.
This"trailingbackslash"or"linecontinuation"characterisacommonUnixidiomfoundinanumberof
programminglanguagesandconfigurationfileformats.
Thesemicolonstelltheshelltoexecuteanothercommandtheyallowseveralcommandstobeissuedon
thesamecommandline.
Eachoftheechocommandsshouldbereasonablyselfexplanatory.Wecouldhaveuseda'cat'command
andputourtextintoafileifwewanted.Wecanalsocallotherprogramsherelike'fortune'or'date'and
theiroutputwouldbecombinedwiththerestofthis).
Nowwegettotheclosingparenthesis.Thismarkstheendoftheblockofcommandsthatwecombined.The
outputfromallofthoseisfedintothenextpipewhichstartsthelocalcopyofsendmail(notethatthisis
anothervariablethatprocmailtoughtfullypresetsforus).
Thetswitchonsendmailtellittotakethe"To:"addressfromtheheaderofit'sinput(where'formailr'
putit)andtheoiswitchenablesthesendmail"option"to"ignore"linesthatconsistonlyofa'dot'(don't
worryaboutthedetailsonthat).
Mostofthedifficultyinunderstandingprocmailasnothingtodowithprocmailitself.Theintricaciesofregular
expressions(thosewierdthingsonthe'*'conditionallines)andshellquotingandcommandsyntax,andhowtoformat
areplyheaderthatwillbeacceptabletosendmail(the'formail'and'sendmail'stuff)arethepartsthatrequiresomuch
explanation.
ThebestinfoonmailbotsthatI'vefoundusedtobemaintainedbyNancyMcGough(sp??)attheInfiniteInkwebpages:
http://www.jazzie.com/ii/
MoreinformationaboutprocmailcanbefoundinEraEriksson's"MiniFAQ."athttp://www.iki.fi/~era/procmail/mini
faq.html
IalsohaveafewprocmailandSmartListlinksoffofmyownwebpages.