Zenoss Event Management

Version 3
September 2009 Updated January 2010 Jane Curry Skills 1st Ltd www.skills-1st.co.uk

JaneCurry Skills1stLtd 2CedarChase Taplow Maidenhead SL60EU 01628782565 jane.curry@skills1st.co.uk

Skills1stLtd

13December2010

Synopsis
ThispaperisintendedasanintermediateleveldiscussionoftheZenosseventsystem. ItassumesthatthereaderisalreadyfamiliarwiththeZenossEventConsoleandwith basicnavigationaroundtheZenossGraphicalUserInterface(GUI).Itlooksinsome detailatthearchitecturebehindtheZenosseventsystemthedaemonsandhowthey areinterrelatedanditlooksatthestructureofaZenosseventandtheeventlifecycle. ZenosscanreceiveeventsfrommanysourcesinadditiontoZenossitself.Eventsfrom Windows,UnixsyslogsandSimpleNetworksManagementProtocol(SNMP)TRAPsare allexaminedindetail. TheprocessbywhichanincomingeventistransformedintoaparticularZenosseventis knownaseventmappingandhasanumberofdifferentpossibletechniquesfor performingthatconversion.Thesewillallbeexploredalongwiththecreationofnew eventclasses. OnceaneventhasbeenreceivedandclassifiedbyZenoss,automationmayberequired. AlertingbyemailandpagerarediscussedasistheabilitytorunanyscriptasanEvent Command. ThispaperwaswrittenusingZenoss2.4.1. ThepaperisacompaniontexttotheZenossEventManagementWorkshop.

Notations
Throughoutthispaper,texttobytyped,filenamesandmenuoptionstobeselected,are highlightedbyitalics;importantpointstotakenoteofareshowninbold.

Skills1stLtd

13December2010

Table of Contents
1Introduction..........................................................................................................................5 2Zenosseventarchitecture....................................................................................................5 2.1EventConsole...............................................................................................................5 2.2EventdatabasetablesandtheEventManager.........................................................9 2.3Eventlifecycle............................................................................................................13 2.3.1Eventgeneration.................................................................................................15 2.3.2Applicationofdevicecontext..............................................................................16 2.3.3Eventclassmapping...........................................................................................17 2.3.4Applicationofeventcontext...............................................................................18 2.3.5Eventtransforms.................................................................................................18 2.3.6Databaseinsertions............................................................................................19 2.3.7Resolution............................................................................................................20 2.3.8Ageingoutevents................................................................................................21 3EventsgeneratedbyZenoss..............................................................................................21 3.1zenping........................................................................................................................22 3.2zenstatus.....................................................................................................................23 3.3zenwin..........................................................................................................................23 3.4zenprocess...................................................................................................................24 3.5zenperfsnmp................................................................................................................24 3.6Availabilitymonitoringdaemonsanddevicestatuspages.....................................24 4Syslogevents......................................................................................................................25 4.1Configuringsyslog.confandsyslogng.conf..............................................................26 4.2Zenossprocessingofsyslogmessages.......................................................................27 5ZenossprocessingofWindowseventlogs.........................................................................34 6EventMapping...................................................................................................................35 6.1Workingwitheventclassesandeventmappings....................................................36 6.2Rulesineventmappings............................................................................................39 6.3Regexineventmappings...........................................................................................40 6.4Otherelementsofeventmappings...........................................................................41 7Eventtransforms...............................................................................................................42 7.1UsingzendmdtorunPythoncommands..................................................................44 7.1.1ReferencinganexistingZenosseventforuseinzendmd.................................44 7.1.2Usingzendmdtounderstandeventattributes.................................................45 7.1.3Usingzendmdtounderstandeventmethods....................................................47 7.2Transformexamples...................................................................................................48 7.2.1CombininguserdefinedfieldsfromRegexwithtransform.............................48 7.2.2Applyingeventanddevicecontextinrelationtotransforms..........................49 8ZenossandSNMP..............................................................................................................51 8.1SNMPintroduction.....................................................................................................51 8.2ZenossSNMParchitecture........................................................................................52 8.2.1Thezentrapdaemon............................................................................................52 3 Skills1stLtd 13December2010

8.3InterpretingMIBs......................................................................................................55 8.3.1zenmibexample...................................................................................................55 8.3.2AfewcommentsonimportingMIBswithZenoss.............................................58 8.4TheMIBbrowserZenPack.........................................................................................62 8.5MappingSNMPevents..............................................................................................62 8.5.1SNMPeventmappingexample..........................................................................63 9EventCommands...............................................................................................................68 9.1Creatingeventcommands..........................................................................................68 9.2Debuggingeventcommands......................................................................................70 10Events,Alerts&ProductionStatus...............................................................................74 10.1Alertingrulesforemailandpaging........................................................................74 10.2Otheralertingpossibilities......................................................................................76 10.3TheeffectofdeviceProductionStatus....................................................................78 11Conclusions.......................................................................................................................79 12AppendixAzendmdcommandsusefulwithevents......................................................81

Skills1stLtd

13December2010

1 Introduction
ZenossisanOpenSource,multifunctionsystemsandnetworkmanagementtool.There isafree,Coreoffering(whichdoesseemtohavemostthingsyouneed),anda chargeableoffering,Enterprise,whichhasextraaddongoodiessuchashighavailability configurations,distributedmanagementservers,rolebasedaccessandvarioussupport contractswhichincludesomeeducationandconsultancy.Foracomparisonofthefee alternatives,tryhttp://www.zenoss.com/product/pricing. Zenossoffersconfigurationdiscovery,includinglayer3topologymaps,availability monitoring,problemmanagementandperformancemanagement.Itisdesignedaround theITILconceptofaConfigurationManagementDatabase(CMDB),theZenoss StandardModel.ZenossisbuiltusingthePythonbasedZopewebapplicationserver andusestheobjectorientedZopeObjectDatabase(ZODB)astheCMDB,usedtostore Pythonobjectsandtheirstates.ZenossusesZEO,asalayerbetweenZopeandthe ZODB. TherelationalMySQLdatabaseisusedtoholdcurrentandhistoricalevents. PerformancedataisheldinRoundRobinDatabase(RRD)files. ThedefaultprotocolsformonitoringaretypicallyagentlesstheSimpleNetwork Managementprotocol(SNMP),WindowsManagementInstrumentation(WMI)and collectingeventsfromsyslogs.Itisalsopossibletomonitordevicesusingtelnet,sshand touseNagiosplugins. ZenossprovidesagoodGettingStartedwithZenossdocumentalongwithaZenoss AdministrationGuideandaZenossDeveloper'sGuide;youcangetthesefromhttp:// www.zenoss.com/community/docs.ThereisalsoawealthofinformationontheZenoss websitebutitisratherdiffusedbetweenFAQs,HowTos,aWikiandcontributionstothe variousforums.AusefulbookisavailablefromPACKTPublishing,ZenossCore NetworkandSystemMonitoringbyMichaelBadger,whichprovidesmuchofthesame informationastheZenossAdministrationGuidebutinamuchclearerformatwith plentyofscreenshots. ThispaperisanattempttoexpandontheeventinformationintheAdministration Guidebydrawingonmyownexperienceandthecollectedwisdomofthecommunity contributions.

2 Zenoss event architecture

2.1 Event Console
WhenaneventarrivesatZenoss,itisparsed,associatedwithaneventclassification andthentypically(butnotalways),itisinsertedintothestatustableoftheevents

Skills1stLtd

13December2010

database.EventscanthenbeviewedbyusersusingtheEventConsoleoftheZenoss GraphicalUserInterface(GUI). TherearethreewaystoaccesstheEventConsole.ThemainEventConsoleisreached fromtheEventConsolemenuontheleft.Thedefaultistoshowallstatuseventswitha severityofInfoorhigher,sortedfirstbyseverityandthenbytime(mostrecentfirst). Eventsareassigneddifferentseverities:

Critical Error Warning Info Debug Clear

Red Orange Yellow Blue Grey Green

Theeventssystemhastheconceptofactivestatuseventsandhistoricalevents(two differentdatabasetablesintheMySQLeventsdatabase). EventsintheconsolecanbefilteredbySeverity(Infoandabovebydefault)andbyState New,AcknowledgedandSuppressedwhereNewandAcknowledgedareshownby default.AnyeventwhichhasbeenAcknowledgedchangestoawishywashyversionof thesamecolour.ASuppressedeventalsohasthewishywashyversionofthecolour. ThereisaSearchboxatthetoprightforfilteringeventsbasedonthepresenceofany stringwithinanyfieldofanevent.

Skills1stLtd

13December2010

Figure1:ZenossEventConsole

FromtheEventConsole,oneormoreeventscanbeselectedbycheckingthebox alongsidetheeventandthetablemenudropdowncanbeusedforvariousfunctions includingAcknowledgeEvents,MovetoHistoryandMapEventstoClass. ThecolumnheadersoftheEventConsolecanbeusedtochangethesortingcriteriaand theiconatthefarrightoftheeventcanbeusedtodisplaythedetaileddataofthe event. ThedetaileddatashowsthedefaulteventfieldsundertheFieldstab,anyuserdefined fieldsundertheDetailstabandtheLogtabrecordsactionssuchasacknowledgeand clearing,alongwithdate,timeandtheuserthatperformedtheseactions.Itisalso possibletoaddyourownusermessagestoenevent'sLogbutonlywhileitisinthe statustableoftheeventsdatabase,notonceithasbeenmovedtothehistorytable(Ack /UnAckstatusmakesnodifference).

Skills1stLtd

13December2010

Figure2:DetaileddataforaneventdefaultFieldstab

ThefieldsundertheDetailstabmayincludetextmessagesfortheExplanationand Resolutioneventfields.

Skills1stLtd

13December2010

NotethatifyouwishtoUndeleteaneventfromthehistorytablesofthedatabase,this ispossiblewiththedropdowntablemenu;howeveritdoesnotchangeanyprevious Acknowledgedstatus. Bydefault,theEventConsoleisrefreshedevery60seconds.Ifyouwishtofreezethe console,clicktheStoplinkatthetopoftheconsole(besidetheboxwith60init);the linkchangestosayStart.Toreturntoautomaticrefresh,clicktheStartlink. AnEventConsolecanalsobeaccessedwhichautomaticallyfilterseventsfora particulardevice.NavigatetothemainpageforadeviceandusetheEventstabtoshow alleventsforthatdevice. ThethirdmethodofdisplayinganEventConsoleappliesanautomaticfilterofevent classorsubclass.StartfromtheEventsmenuontheleft.TheEventstabwillshowan EventConsolefilteredbythechosenclassorsubclass.Notethatthetopleveldropdown menuhasanoptiontoAddEvent.Thisisusefulforgeneratingtestevents. EachofthesethreemethodsofaccessinganEventConsoleshowsactiveevents.Tosee eventsthathavebeensenttothehistorytableoftheeventsdatabase,clicktheblue ViewEventHistorylinkatthetoprightoftheEventConsole.

2.2 Event database tables and the Event Manager

ZenosseventsareheldinaMySQLdatabasecalledeventswhichiscreatedwhen Zenossisinstalled.Bydefault,thezenossusercanaccessthisdatabasewithapassword ofzenoss.

Skills1stLtd

13December2010

Figure3:UsingMySQLtoexaminetheeventsdatabase

Themaintableswithintheeventsdatabasearestatusandhistory.Theactiveevents arekeptinthestatustableandthehistoricalevents(typicallyresolved,clearedevents) areheldinthehistorytable.Theformatofeachofthesetablesandthevalidfieldsfora Zenosseventcanbeseenbeexaminingthedatabasesetupfilein /usr/local/zenoss/zenoss/Products/ZenEvents/db/zenevents.sql.

10

Skills1stLtd

13December2010

Figure4:Definitionofstatuseventfieldsinzenevents.sql

zenevents.sqlalsodefinesthehistorytableinasimilarfashion. Afurtherfourtablesaredefinedforheartbeat,alert_state,loganddetail.Thedetail tablecanbeusedtoextendthedefaulteventfieldstoincludeanyinformationthatthe Zenossadministratorrequiresforanevent,includingExplanationandResolutiontext.

11

Skills1stLtd

13December2010

Figure5:zenevents.sqlshowingheartbeat,alert_state,loganddetailtables

Someoftheseeventfieldsareparticularlypertinentdependingonhowtheeventwas generated:

Syslogeventspopulatethefacilityandpriorityfields Windowseventspopulatethentevidfield SNMPTRAPspopulateatleastacommunityfieldinthedetailtable.Theyalso usethedetailtabletoprovideanyvariablespassedbyanSNMPTRAP. TheagentfielddenoteswhichZenossdaemongeneratedorprocessedthe incomingevent;forexample,zentrap,zeneventlog,zenping.

Connectioninformationfortheeventsdatabasealongwithcachingandmaintenance parameters,canbeaccessedfromEventManagerinthemenuontheleftoftheZenoss console.

12

Skills1stLtd

13December2010

Figure6:EventManageroptionsforMySQLeventsdatabase

Bydefault,statuseventsofseveritybelowError,areagedouttotheeventshistory tableafter4hours.Historicaleventsareneverdeleted.

2.3 Event life cycle

Thelifecycleofaneventhaseightphases:

Eventgeneration Devicecontextadditionalinformationaboutthedevicethatgeneratedtheevent Eventclassmappingtodistinguishonetype(class)ofeventfromanother Eventcontextadditionalinformationpertinenttoaclassofevent Eventtransformmanipulationofeventfields Databaseinsertion Resolution Ageout

Processingofaneventdependsontheeventclassthataneventisassignedtothe valueofitseventClassfield.Adescriptionofeachofthesephaseswillbegivenhere: subsequentsectionsofthepaperprovidemoredetailsofsomeareas.

13

Skills1stLtd

13December2010

Figure7:Eventlifecycle,generationtodatabaseinsertion

InFigure7,thefirstsixphasesoftheeventlifecycleareshown.Theblue,dashedpath showstheprogressofaninternallygeneratedZenossevent,whichdoesnotpassthrough aneventmappingphase.AneventClassfieldisproducedbythedaemonthat generatedtheevent.Itsonlywaytoapplyatransformisasaclasstransform. ThepurplepathshowstheprogressofaneventthatisgeneratedexternallytoZenoss. TheinitialparsingdaemonmustprovideaneventClassKeyfieldwhichisthenused, alongwithotherfields,inaneventclassmappingRuleand/orRegex,whichinturn providesaneventClassfield.Aftermapping,theeventmaypassthroughbothan eventclasstransformandaneventmappingtransform.

14

Skills1stLtd

13December2010

Figure8:Eventlifecycle,resolutiontoageout

Figure8showsthelatterphasesoftheeventlifecycle.Bluepathsshowtheeffectofthe zPropertyzEventActionontheinsertionoftheeventintothevarioustablesoftheevent database.GreenpathsshowtheeffectwhenthezPropertyzEventSeverityisequalto Cleared.PurplepathsshowtheadditionalclearingeffectofthezProperty zEventClearClasses. Themovementofeventsbetweentablesoftheeventsdatabasearecolourcodedwith greenforclearingbasedonseverity=Cleared,purpleforclearingbasedon zEventClearClasses,redformanualactions(includingAcknowledgewhichdoesnot movetheeventbetweentables),andblackformovementsbasedontimeouts.

2.3.1 Event generation

Fundamentally,eventswilleitherbegeneratedbyZenossitselfintheprocessof discovery,availabilityandperformancechecking,oreventswillbegeneratedoutside ZenossandcapturedbyspecialisedZenossdaemons.

15

Skills1stLtd

13December2010

Zenossdaemon zenping zendisc zenstatus zenprocess zenwin zenperfsnmp

Exampleofwheneventgenerated pingfailureoninterface newdevicediscovered TCP/UDPserviceunavailable processunavailable Windowsservicefailed SNMPperformancedatacollectionfailure

Table2.1.:EventsgeneratedbyZenossitself

Zenossdaemon zensyslog zeneventlog zentrap

Exampleofwheneventgenerated processessyslogeventsreceivedonUDP/514(default) processesWindowseventsreceivedusingWMI processesSNMPTRAPsreceivedonUDP/162

Table2.2.:ExternaleventscapturedbyspecialisedZenossdaemons

EventsgeneratedinternallybyZenossneednofurtherprocessingtointerprettheevent. Thedaemonthatgeneratestheeventparsesthenativeinformationandassignsavalue totheeventClassfieldandanyotherrelevantfieldssuchascomponent,summary, messageandagent.TypicallytheeventClassKeyfieldwillbeblank.SomeZenoss daemonspopulatetheeventKeyfield(forexampleanInterfacediscoveryeventwill populatetheeventKeyfieldwiththeIPaddressofthediscoveredinterface). EventsthatareinitiallygeneratedoutsideZenossarecapturedbyzensyslog, zeneventlogorzentrap.Thesedaemonseachhaveaparsingmechanismtointerpret thenativeeventintotheZenosseventformat.ThePythoncodeforthisparsingisin $ZENHOME/Products/ZenEvents.(Bydefault,$ZENHOMEwillbe /usr/local/zenoss/zenoss).SyslogProcessing.pydecodessyslogevents;zentrap.py decodesSNMPTRAPs. Typically,theseparsingmechanismsdonotdeliveravalueforeventClass;ratherthey deliveravaluefortheeventClassKeyfield,alongwithvaluesforsomeotherfields suchascomponent,summary,messageandagent.Itisthenthejoboftheevent mappingphasetodistinguishtheeventclass.

2.3.2 Application of device context

Earlyintheeventprocessinglifecycle,devicecontextisappliedtotheevent.This meansthatsixfieldsoftheeventarepopulatedbydeterminingthedevicethat 16 Skills1stLtd 13December2010

generatedtheeventandthenlookingupthefollowingvaluesforthedeviceintheZODB database:

prodState Location DeviceClass DeviceGroups Systems ipAddress(mayhavealreadybeenassigned)

2.3.3 Event class mapping

Eventclassmappingtendsonlytobeapplicabletoeventsthatoriginateoutsidethe Zenosssystem.Itistheprocessbywhichaneventisassignedavalueforits eventClassfieldand,potentially,otherfields. Typically,theeventgenerationphasewilldeliveraneventwithafewfieldspopulated; generallythisdoesnotincludetheeventClassfieldbutdoesincludetheeventClassKey field.OftentheZenossparsingdaemon(suchaszensyslog),willusethesame eventClassKeyforseveraldifferentnativeevents.Forexample,aneventClassKeyof dropbearisusedforseveralloginsecurityevents.Thecomponent,summary,message andagentfieldsmayalsobepopulated. Theeventclassmappingphaseexaminestheevent(suchasitis,sofar)andthenusesa numberofteststodeterminetheeventClasstoassigntothisevent: 1. AneventClassKeyfieldmustexistformappingtobesuccessful. 2. APythonRulecanbewrittentotestanyavailablefieldoftheeventorany availableattributeofthedevicefromwhichtheeventcame.Suchrulescanbe complexPythonexpressions,includinglogicalANDsandORs.Iftheruleis satisfied,theincomingevent'seventClassfieldisgiventheclassassociatedwith thatmapping.Iftheruleisnotsatisfied,thismappingisdiscarded,theclassis notassociated,andthenextmappingwillbetestedforamatch.ARuledoesnot havetoexistinamappinginstance. 3. IftheRuleissatisfied(ordoesnotexist),themappingcanthenuseaRegex Pythonregularexpressiontoparsetheevent'ssummaryfield,checkingfor particularstrings.TheRegexcanalsoassignpartsofthesummaryfieldtonew, userdefineddetailfieldsoftheevent.IfaRuleexistsandissatisfied,theclass mappingwillapply,eveniftheRegexisnotsatisfied;anyuserdefinedfieldsin theRegexwillnotbecreatediftheRegexdoesnotmatch.IfaRuledoesnot existthentheRegexmustbesatisfiedforthemapping(andanytransform)to apply.

17

Skills1stLtd

13December2010

4. TheGUIdialoguethatdefinesthemappingspecifiestheeventClassKey,theRule, theRegexandanyTransform.Asequencenumberisalsoavailablesothatif multipleincomingeventshavethesameeventClassKeythenthesequence numberdefinestheorderinwhichthevariousmappingswillbeapplied,lowest numberfirst.ThefirstRule/Regexmappingcombinationthatmatcheswillbe applied.

2.3.4 Application of event context

EventcontextisdefinedbythezPropertiesofanevent.Eventcontextcanbedefinedat theeventclasslevel,foraneventsubclass,orattheeventmappinglevel.Aswithall objectorientedattributes,thevaluesareinheritedbychildobjectssoapplyingevent contexttoaclassautomaticallysetsitforanysubclassesandsubclassmappings.The threeeventcontextattributesare:

zEventAction zEventClearClasses zEventSeverity

status|history|dropdefaultisstatus bydefaultthisisanemptyPythonlistofstrings Originalbydefault

Eventcontextisappliedintheeventlifecycle,afterRuleandRegexprocessingbut beforeanyeventtransforms.Thus,thezEventActionzPropertycanspecifythehistory databasebutaneventtransformcouldoverridethatactionbysettingtheevt._action valuetostatus.

2.3.5 Event transforms

Eventtransformscanbespecifiedforaneventclassmappingorforaneventclass(or subclass).AtransformiswritteninPythonandcanbeusedtomodifyanyavailable fieldsofeithertheeventorthedevicethatgeneratedtheevent.Itcanalsocreateuser definedfields. FromZenoss2.4,cascadingeventtransformsmeanthatclasstransformsareapplied fromeverylevelintheappropriateclasshierarchy,followedbyanytransformforan appliedeventmapping.PriortoZenoss2.4,eitheramappingtransformwasapplied, oraclasstransform,butnotboth.Classtransformswereonlyappliedtotheexact class,notfromtheeventclasshierarchy. AtransforminaneventmappingwillonlybeexecutedoncetheeventClassKeyhasbeen matched,andtheRulehasbeensatisfied(ifitexists).IfaRuledoesnotexist,any Regexhastobesatisfiedforthetransformtobeexecuted.

18

Skills1stLtd

13December2010

2.3.6 Database insertions

ZenosseventsarestoredinaMySQLdatabasecalledevents(bydefault).Theevents databaseshasanumberoftablesdefinedsee /usr/local/zenoss/zenoss/Products/ZenEvents/db/zenevents.sqlfortheconfiguration oftablesandtriggersfortheeventsdatabase. Themaintablesfortheeventlifecyclearethestatustableforactiveevents,the historytableforresolvedeventsandthedetailtableforuserdefinedfieldsofevents. Somefieldsoftheeventareonlyassignedatdatabaseinsertiontimetheyarenot availableateventmappingoreventtransformtime.Theseinclude:

count evid stateChange dedupid suppid eventClassMapping firstTimeisthesameaslastTimeuntildatabaseinsertion

ThePythoncodethatdrivesdatabaseinsertioncanbefoundin$ZENHOME/Products/ ZenEvents;EventClass.py,EventClassInst.pyandMySqlSendEvent.pyarethemain files.Someoftheeventfieldscanonlybedeterminedbyreferencetootherevents alreadyinthedatabasesuchascount,dedupid,stateChangeandfirstTime. Zenossautomaticallyappliesaduplicationdetectionrulesothatifaduplicateevent arrives,thentherepeatcountofanexistingeventwillsimplybeincremented. duplicateisdefinedashavingthefollowingfieldsthesame:

device component eventClass eventKey severity

IftheeventdoesnotpopulatetheeventKeyfield,thenthesummaryfieldmustalso match.Atdatabaseinsertiontime,thededupidfieldiscreatedbyconcatenatingthe abovefieldstogether,separatedbythepipe(verticalbar)symbol.Thusanexample dedupidmightbe:

zenoss.skills-1st.co.uk|su|/Security/Su||5|FAILED SU (to root)jane on /dev/pts/1

wherethedeviceiszenoss.skills1st.co.uk,componentisSecurity,eventClassis /Security/Su,theeventKeyisunset,severityis5(Critical),andthesummaryis FAILEDSU(toroot)janeon/dev/pts/1. 19 Skills1stLtd 13December2010

2.3.7 Resolution
Resolutionisgenerallytheprocessbywhichaneventismovedfromthestatustableof theeventsdatabasetothehistorytable. Asecondpossibledefinitionofresolutionisthattheeventisdroppedentirely,never reachingthestatustablethiscanbeachievedeitherfromtheeventcontextbysetting zEventActiontodrop,orinatransformwithevt._action=drop.Thesesamemechanisms canalsobeusedduringinitialeventprocessing,tosettheeventactiontohistory,thus preventingtheeventfromeverappearinginthestatustable. Athirdpossibilityistosettheevent'seventStatefieldtoSuppressedwhichmeansthe eventdoesnotdisplay,bydefault,intheEventConsolestatusdisplay.Thesuppressed eventStatemechanismonlyappearstobeusedbythezenpingdaemon. Aneventcanberesolvedbyhumanintervention.TheEventConsoledropdowntable menuprovidesanoptiontoMovetohistory;theZenossAdministrationGuidedescribes thisashistorifying.ItalsoprovidestheoptiontoAcknowledgeanevent. AcknowledgingchangestheeventStatefieldtoAcknowledged(asopposedtoNew);it changesthecolouroftheeventtoawishywashyversionofthesamecolour.Itdoesnot movetheeventintothehistorytable. Themoreinterestingformsofeventresolutioninvolvecorrelationofevents;thereare twodifferentmechanisms.Thebasicprincipleisthatgoodnewsclearsbadnews. ThefirstclearingmechanismisthatanyeventwithaseverityofClearedwillsearch thestatustableoftheeventsdatabaseandmoveanysimilareventstothehistorytable. SimilarisdefinedashavingthesameeventClass,deviceandcomponentfields.All similareventsarecleared,notjustthemostrecent. Whencorrelationtakesplaceanumberoftheexistingbadnewseventfieldsare updated.stateChange,deletedTimeandclearidareallmodifiedwithclearidbecoming thevalueoftheevidfieldoftheclearing,goodnewsevent.Thegoodnewsevent, withitsClearedseverity,isautomaticallymovedtothehistorytable. Thesecondcorrelationmechanismistospecifyintheeventcontext,oneormoreevent classesinthezEventClearClasseszProperty.Thisattributewillonlybeusedon clearingevents.Theeffectisthatanysimilareventsofthelistedclasseswillalsobe cleared,inadditiontoeventsofthesameclassastheclearingevent.Similarinthis casemeansthesamedeviceandcomponentfields,plustheclassspecifiedin zEventClearClasses.Notethatthesameeffectcanbeachievedinatransformby assigningalistofclassnamestoevt._clearClasses.

20

Skills1stLtd

13December2010

2.3.8 Ageing out events

Maintenanceisrequiredonthetablesoftheeventsdatabaseorthediskwillsimplyfill upeventually.TwomechanismsareprovidedbytheEventManager:

Bydefault,eventswithseveritylessthanErrorwillbeagedfromthestatustable tothehistorytableafter4hours.Theseparameterscanbemodified. Historicaleventscanbedeletedoncetheyareolderthanagivennumberofdays. Thedefaultis0thatis,noeventsaredeletedfromthehistorytable automatically.

ManualmaintenanceontheMySQLdatabasemayalsoberequired.Zenossprovidesa utilityin$ZENHOME/Products/ZenUtils:
ZenDeleteHistory.py --numDays=10

toclearhistoryeventsolderthan10days

Thescriptshouldberunasthezenossuser. Alternatively,theMySQLdatabasecanbemanipulateddirectlybythosewithSQL knowledge.Hereareafewexamples;again,workasthezenossuser(noteyouneedthe trailingsemicolononSQLstatements):

mysql -uzenoss -pzenoss -Devents select * from heartbeat; Accessmysqlaszenossuser,toeventsdb Showallheartbeatevents Deletelocalheartbeatevents

delete from heartbeat where device='localhost'; select * from status where device='mybox';

Showallstatuseventsformybox

select * from detail where evid NOT IN (select evid from status UNION select evid from history); Showdetailrecordsforeventsdeletedfromstatusandhistory delete from detail where evid NOT IN (select evid from status UNION select evid from history); Deletedetailrecordsforeventsdeletedfromstatusandhistory

3 Events generated by Zenoss

Inthecourseofdiscovery,availabilitymonitoringandperformancemonitoring,Zenoss maygenerateeventstorepresentachangeinthecurrentstatus.Althoughmanyevents arebadnewsitshouldberecognisedthateventscanalsobegoodnewsInterface Up,Thresholdnolongerbreached,etc. EventsgeneratedbyZenossaredependentonthevariouspollingintervalsconfigured. Toexaminethedefaultparameters,usetheCollectorsoptionfromtheZenosslefthand menu.Clickonlocalhost(thecollectorontheZenosssystem).Notethatearlierversions ofZenossusedthetermandmenuoptionMonitorsratherthanCollectors.

21

Skills1stLtd

13December2010

Figure9:DefaultparametersforlocalhostCollectors

Parameterstonoteparticularlyare:

SNMPpollingcycle Pollingforprocesses StatuspollingforTCP/UDPservices PollingforWindowsservices WindowsWMIpoll Pingpolling

300secs(5mins) 180secs(3mins) 60secs(1min) 60secs(1min) 60secs(1min) 60secs(1min)

3.1 zenping
Themostbasiclevelofavailabilitycheckingistopingpoll.Thezenpingdaemonwill, bydefault,pingpolleachinterface,everyminute.Aninterfacedowneventisgenerated whenthepingfailstogetaresponse.Thiseventisautomaticallyclearedwhena similarpingissuccessful;meantime,whileaninterfaceremainsdown,thecountfieldof theeventisincreased. Thezenpingdaemoncandetectwhenthenetworkpathtoadeviceisbroken,for exampleifasinglepointoffailurerouterisdown.Inthiscase,aneventisgenerated

22

Skills1stLtd

13December2010

withaneventStatefieldofSuppressedandthesummaryfieldreportsnotonlythe interfaceforwhichthepingfailed,butalsothecausaldevice;forexample: ip10.191.101.1isdown,failedatbino.skills1st.co.uk Allotherdeviceavailabilitymonitoringisdependentonpingaccess.Onceapinghas failed,SNMP,process,TCP/UDPserviceandwindowsservicemonitoringwillallbe suspendeduntilpingaccessisrestored.Thecountfieldofthehigherlevelmonitoring eventswillnotincreaseuntilpingaccessisresumed. Alsonotethatifthereisnopingaccess,noperformanceinformationwillbecollected.If adevicereallydoesnotsupportping,perhapsbecauseoffirewallrestrictions,then ensurethatthezPropertyzPingMonitorIgnoreissettoTrue;thiswillpermitSNMPand sshavailabilitymonitoringandperformancedatacollection. Thelogfileforzenpingiszenping.login$ZENHOME/log.

3.2 zenstatus
ThezenstatusdaemoncanbeconfiguredtocheckforaccesstovariousTCPand/orUDP portsonbothWindowsandUnixarchitectures.Bydefault,itcheckseveryminute. Zenosscomeswithahugenumberofservicespreconfigured;thesecanbeexamined fromtheServices>IpServicelefthandmenu.Bydefault,noneoftheseservice monitorsareactive.Servicemonitoringforadevicecanbeconfiguredfromthedevice's mainpageusetheOStabandthetabledropdownmenubesideIPServicestoadda servicetobemonitoredforthisdevice. Aswithpingpolling,agoodnewsserviceeventforadeviceautomaticallyclearsa similarbadnewseventandthecountfieldoftheeventincreaseswhilsttheservice remainsdown. Thelogfileforzenstatusiszenstatus.login$ZENHOME/log.

3.3 zenwin
ThezenwindaemonmonitorsWindowsservices(notTCP/UDPservices).Thesecan beexaminedfromtheServices>WinServicelefthandmenu.Bydefault,noneofthese monitorsareactive.Windowsservicemonitoringforadevicecanbeconfiguredfromthe device'smainpageusetheOStabandthetabledropdownmenubesideWinServices toaddaservicetobemonitoredforthisdevice. zenwinusestheWindowsManagementInstrumentation(WMI)interfacetoaccess servicesontheremotesystemeveryminute,bydefault.ThezPropertiesforadevice(or deviceclass)mustbeconfiguredtoallowaccesstoWMIbeforewindowsservicepolling canbesuccessful. Aswithpingpolling,agoodnewswindowsserviceeventforadeviceautomatically clearsasimilarbadnewseventandthecountfieldincreasesonsubsequentfailed polls. Thelogfileforzenwiniszenwin.login$ZENHOME/log. 23 Skills1stLtd 13December2010

3.4 zenprocess
zenprocessmonitorsWindowsandUnixsystemsforthepresenceofprocesses.Ina Unixcontext,thiswouldbewhethertheprocessappearsinapseflisting;inaWindows context,theprocessmustappearintheWindowsTaskManager(andnotethatthis checkiscasesensitiveonbotharchitectures).Monitoringisevery3minutes,bydefault. Configurationofprocessmonitoringforadeviceissimilarasforservicesusethe device'smainpage>OStabandthetabledropdownmenubesideOSProcessestoadda processtobemonitored. ProcessmonitoringisactuallyachievedusingtheHostResourcesManagement InformationBase(MIB)ofSNMP,byretrievingthehrSWRuntable.Thismeansthat ifSNMPaccesstoadeviceisbroken,therewillbenoprocessinformation. Aswiththeotheravailabilitydaemons,goodnewseventsclearbadnewseventsand thecountfieldincreasesonsubsequentfailedpolls. Thelogfileforzenprocessiszenprocess.login$ZENHOME/log.

3.5 zenperfsnmp
zenperfsnmppollseachdeviceevery5minutes,bydefault.ItcancollectbothSNMP performanceinformationandstatusinformationforprocesses.Processmonitoringis achievedusingtheSNMPHostResourcesMIBsothatifanSNMPagentfails,then processmonitoringisalsoaffected.ThisisnotthecaseforWindowsservicemonitoring. TCP/UDPservicemonitoringalsodoesnotrelyonSNMPonanyplatform. Within5minutesofanSNMPpollfailure,ansnmpagentdowneventshouldbe generated.Withinafurther3minutesthereshouldbeanUnabletoreadprocesseson device..event,ifprocessmonitoringisconfigured.Notealsothatthecountfieldfor individualmissingprocesseventsshouldstopincreasing.Countsformissingservice eventswillbeunaffectedbythelossofSNMP.WhileSNMPaccesstothedevice remainsbroken,thecountfieldfortheUnabletoreadprocessesondevice..eventwill increaseevery3minutes.

3.6 Availability monitoring daemons and device status pages

EachdevicehasitsownDeviceStatuspage.Thereisanoverallstatusiconbuttonat thetopofthepagethatfundamentallyreportspingaccess(ifpingaccessisenabled) greenforgoodstatus;redforbad.Notethatifadevice'szPropertiesarecustomisednot topingmonitor,thenthisoverallstatusbuttonwillalwaysremaingreen! ThetoprighthandpaneloftheDeviceStatuspagereportscomponentstatusforother monitoredelementssuchasinterfaces,servicesandprocesses.TheOthercategory includesstatusforeventsreportedviasyslogorWindowseventlogs,wherethecolourof 24 Skills1stLtd 13December2010

thebuttonrepresentstheseverityoftheevent.NotethatAcknowledgingsuchevents hasnoeffectontheDeviceStatuspage.

4 Syslog events
TheUnixsyslogmechanismispervasivethroughoutallversionsofUnix/Linux althoughslightlydifferentversionsandformatsexist.Therearealsoopensource implementationsofsyslogforWindowssystemsandmanynetworkingdevicesalso supportthesyslogconcept. Typicallysystemmessagesareoutputtooneormorelogfilessuchas /var/log/messages.Thesyslogsubsystemcanalsobeconfiguredtosendsyslog messagestoacentralsyslogratherthanholdingfilesoneachsystem.Thewellknown defaultportforforwardingsyslogmessagesisUDP/514. Astandardsyslogsystemisconfiguredbythesyslog.conffile,typicallyin/etc.Anewer versionofsyslogisimplementedonsomesystems,syslogng,whichhasgreater filteringcapabilities.Thesyslogngconfigurationfileistypically/etc/syslogng/syslog ng.conf. Asyslogmessageincludesapriorityandafacility.Theprioritiesare: 0 1 2 3 4 5 6 7 emerg alert crit err warning notice info debug authpriv(10) daemon(3) kern(0) mail(2) syslog(5) uucp(8)

Facilitiesinclude: auth (4) cron (9) ftp(11) lpr(6) news (7) user (1)

Thesedefinitionscanbefoundinsyslog.h(typicallyin/usr/include/sys).Bothpriority andfacilityareencodedinasingle32bitintegerwherethebottom3bitsrepresent priorityandtheremaining28bitsareusedtorepresentfacilities. 25 Skills1stLtd 13December2010

Forexample,ifthefacility/prioritytagis<22>,thiswouldbe00010110inbinary,where thebottom110representsapriorityof6(info)andthetop00010representsafacilityof 2=mail.

4.1 Configuring syslog.conf and syslog-ng.conf

AnydevicethatisgoingtoreportsyslogeventstoZenossmusthaveitssyslog.conffile configuredwiththedestinationaddressoftheZenosssystem.Theoriginalsyslog.conf permitsfilteringbasedonpriorityandfacilityso,acatchallstatementtosendall eventstotheZenosssystem,wouldbe:
*.debug @<IP address of your Zenoss system>

syslogng.confrequiresatleastasource,adestinationandalogstatement.syslogng offerssuperiorfilteringovertheoriginalsyslogsooneormorefilterstatementsmay alsobepresent.

Figure10:syslogng.conftosendalleventstoZenosssystemat10.0.0.131(nofilteringactive)

26

Skills1stLtd

13December2010

4.2 Zenoss processing of syslog messages

TocollectsyslogmessageswithZenoss,thezensyslogprocessautomaticallystartson portUDP/514andcollectsanysyslogmessagesdirectedfromothersystems.zensyslog thenparsesthesemessagesintoZenossevents.Youmustensurethatthesyslog.conf fileontheZenosssystemdoesnotenablecollectingremotesyslogsorthesyslogdand zensyslogprocesseswillclashoverwhogetsUDP/514(itispossibletoreconfigureeither daemon,ifrequired). Toexaminetheincomingsyslogmessagesandtheparsingthatzensyslogperforms,the levelofzensyslogloggingcanbeincreased. 1. UsetheSettingsmenuontheZenosslefthandmenuandchoosetheDaemons tab. 2. Clicktheeditconfiglinkforthezensyslogdaemon. 3. ChangethefollowingparametersandclickSave: logorig logseverity selectthis Debug

4. Inspecttheunderlyingconfigurationfilein$ZENHOME/etc/zensyslog.conf. 5. Thelogoriglinesaystologtheoriginalincomingsyslogmessage;itwillbein $ZENHOME/log/origsyslog.log.Notethatthisparameterisuniquetozensyslog andisusefulfordebugging. 6. ThelogseveritylineisagenericZenossdaemonparameter;avalueof10isthe maximumDebuglevel. 7. Don'tforgettoSavethischange 8. UsetheRestartlinktorecyclezensyslog.Alternatively,asthezenossuser,issue thecommand:

zensyslog restart

9. Examinethezensysloglogfilein$ZENHOME/log/zensyslog.log 10. Anewincomingeventstartswithalineshowinghostnameandipaddress,eg.

host=zen241.class.example.org, ip=172.16.222.241

11. Thenext2linesshowtherawmessageandthedecodingforfacilityandpriority. 12. Linesstartingwithtagshowthezensyslogparsingprocessasitteststhe incominglineagainstvariousPythonregularexpressions,hopefullyendingwith atagmatchline. 13. Ifamatchissuccessful,aneventClassKeymaybedetermined 14. ThelastlineforaparsedeventshouldbeaQueueingevent.

27

Skills1stLtd

13December2010

Wheneverdifferentnativeeventlogsystemsareintegratedthereisalmostinevitablya mismatchofseverities.Thefollowingtabledemonstratesthis. Zenoss Critical(red)(5) Error(orange)(4) Warning(yellow)(3) Info(blue)(2) Debug(grey)(1) Clear(green)(0) syslogpriority emerg(0) alert(1) crit(2) err(3) warning(4) notice(5) info(6) debug(7)
Table4.1.:EventseveritiesforZenoss,syslogandWindows

Windows Error(1) Warning(2) Informational(4) Securityauditsuccess(8) Securityauditfailure(16)

NotethatthenumericvalueofZenosseventseveritydecreasesaseventsgetless criticalbutthatthepriorityofsyslogeventsincreasesaseventsgetlesscritical. DefaultmappingfromsyslogprioritytoZenosseventseverity,isperformedby /usr/local/zenoss/zenoss/Products/ZenEvents/SyslogProcessing.pysearchfor defaultSeverityMaparoundline163.Theresultisthat:

syslogpriority<3(emerg,alert,crit)maptoZenossseverity5(Critical) syslogpriority3(err)mapstoZenossseverity4(Error) syslogpriority4(warning)mapstoZenossseverity3(Warning) syslogpriority5or6(notice,info)maptoZenossseverity2(Info)

Outofthebox,allsyslogeventsmaptotheZenosseventclassof/Unknown. SyslogProcessing.pyin$ZENHOME/Products/ZenEventsisthecodethatparsesany incomingsyslogmessageandgeneratesaZenossevent. ThefirstsectionhasaseriesofPythonregularexpressionstomatchagainstthe incomingsyslogline.Eachexpressionischeckedinturnuntilamatchisfound.Ifno matchisfoundthenanentrygoesto$ZENHOME/log/zensyslog.logwithparseTag failed.

28

Skills1stLtd

13December2010

Figure11:SyslogProcessing.pyregularexpressionstomatchsyslogtags

ThemainbodyofSyslogProcessing.pystartsbyassigningvaluesfromtheincoming eventtoZenosseventclassfields,asfollows:
def process(self, msg, ipaddr, host, rtime): evt = dict(device=host, ipAddress=ipaddr, firstTime=rtime, lastTime=rtime, eventGroup='syslog')

Atthisstage,noaccountofduplicatesistakensothefirstTimeandlastTimefieldsare bothsettothetimestampontheincomingevent.NotethattheZenosseventGroupfield ishardcodedatthisstagetosyslog.

29

Skills1stLtd

13December2010

Figure12:SyslogProcessing.pyprocessmainroutine

parsePRIisthePythonfunctioncalledtoparseoutthesyslogpriorityandfacility. ThedefaultSeverityMapfunctioniscalledfromwithintheparsePRIfunctiontosetthe severityfieldoftheZenossevent.

30

Skills1stLtd

13December2010

Figure13:SyslogProcessing.pyparsingofpriority,facilityandseverity

Next,theparseHEADERfunctioniscalledtoextractthetimestampandhostnamefrom theincomingevent.Ifthehostnamedoesnotexistthenanattemptismadetolookup thenamefromtheIPaddressusingagethostbynamecall.ThedevicefieldoftheZenoss eventissetattheendofthisfunction.

31

Skills1stLtd

13December2010

Figure14:SyslogProcessing.pyprocessingtheheaderinformation

TheparseTagfunctioniscalledtoparseoutthesyslogtag,usingtheregexexpressions atthebeginningofthefile.IfnomatchexiststhenaparseTagfailedmessageislogged. TheendofthefunctionreturnstheremainderoftheincomingmessageintheZenoss eventsummaryfield.

32

Skills1stLtd

13December2010

Figure15:SyslogProcessing.pyparsingthesyslogtag

ThecruxofeventprocessinginZenossistoderiveaneventClassKeythisisdone withthebuildEventClassKeyfunction.

Figure16:SyslogProcessing.pydeterminingtheEventClassKey

33

Skills1stLtd

13December2010

Notethatiftheeventhasthecomponentfieldpopulatedthenthatisusedasthe eventClassKeyaftercheckingforapreexistingeventClassKeyandforanntevidfield.

5 Zenoss processing of Windows event logs

ThezeneventlogdaemonisresponsibleforprocessingeventsfromWindowsevent logs.ItusestheWMImechanismso,likethezenwindaemonformonitoringWindows services,theWindowszPropertiesforadevice(ordeviceclass)mustbeconfigured correctly.Fromthedropdowntablemenuofadevice'sstatuspage,chooseMore> zProperties.ScrolldowntothezWin...propertiesandcheckyouhavethefollowing settings.Don'tforgettoSaveifnecessary.

zWinEventlog zWinEventlogMinSeverity zWinPassword zWinUser zWmiMonitorIgnore

True 16(thiswillgatherALLevents) Administrator False <thecorrectpasswordforAdministrator>

NoteespeciallythezWinEventLogpropertytoturnon/offeventlogcollectionand zWinEventLogMinSeveritythatprovidesacrudefilteringmechanism(seetheearlier Table4.1onpage28forthedifferentseveritiesforWindowsEventlogs). ItisthezeneventlogdaemoninZenossthatreceivesincomingWindowseventsand parsesthemintoZenossevents.Typically,theSourcefieldontheWindowseventmaps tothecomponentfieldintheZenossevent;theZenosseventClassKeyiscomposedofthe Windows<Source>_<EventID>(eg.Perflib_2003);theZenosseventGroupbecomesthe Windowslogfilename(Application,Security,etc)andtheWindowsEventIDismapped totheZenossntevidfield. ManyWindowseventlogeventsareautomaticallymappedtoeventclassesbutthey mayhavealowseverity(suchasDebug)andtheymayhavetheirzEventActionevent zPropertysettohistorysothattheydonotappearinthestatustableoftheevents database. Watchoutforeventsofclass/Status/Wmi/Conn,typicallyafteraWindowssystem reboot.Untilthiseventismovedtohistory,noothereventswillbereceivedfromthe WMIinterfaceontheWindowssystemasthezenwindaemonwillnotreconnectand zeneventlogwillreceivenoevents. ThereisalsoasyslogutilityavailableforWindowssystemsfromDatagramConsulting athttp://syslogserver.com.TheclientutilityisSyslogAgentandismadeavailable undertheGNUlicense.SyslogserverutilitiesforWindowsarealsoavailableas

34

Skills1stLtd

13December2010

chargeableproducts.ThismeansthatWindowseventlogscanalsobecollectedwith thezensyslogdaemon. NotethattheSyslogagentiscapableofbeingconfiguredtomonitorWindows applicationlogfiles,inadditiontothestandardWindowseventlogs.Whenmonitoring thestandardeventlogs,therearebetterfilteringcapabilitiesthanwhenusing zeneventlog.

6 Event Mapping
ZenosseventsarecategorisedintoahierarchyofeventClasses,manyofwhichare definedoutoftheboxbutwhichcaneasilybemodifiedoraugmented.Theprocessof EventClassMappingisaboutassociatinganincomingeventwithaparticularZenoss EventClass(settingitseventClassfield)and,potentially,modifyingotherfieldsofthat eventbyusinganeventtransform. Eventclassesandsubclassesaretreatedidenticallyfromthepointofviewofeventclass mapping.Theclasshierarchycanbeusefulinthateventcontext,asimplementedby eventzProperties(suchaszEventSeverity,zEventAction),followsthenormalrulesfor objectinheritanceifzEventActionissettodropontheeventclass/Ignore,thenany subclassesof/Ignorewillalsoinheritthatproperty. NotableoutoftheboxeventzPropertiesarethat/Ignoreclassesandsubclassesdrop incomingevents(ie.theydonotappearineitherthestatusorthehistorydatabases); /Archiveclassesandsubclassesautomaticallymoveeventstothehistorydatabase. Mosteventclasseshaveoneormoremappingsassociatedwiththemtheseareknown asinstances.Notethataneventdoesnothavetohaveanymappingsassociated,in whichcaseaneventofthatclasswillonlyappearinanEventConsoleifthedaemon thatgeneratestheevent,assignstheeventclassatthattime(/Perfeventsmaywell comeintothiscategory,forexample).Outoftheboxeventclassmappingsaredefinedin $ZENHOME/Products/ZenModel/data/events.xml.Theycanbeinspectedfromthe ZenossGUIbyselectingEventsinthelefthandmenuanddrillingdownthesubclass hierarchyundertheClassestab.Alternatively,theMappingstabshowsamore mappingcentriclist,ratherthanaclasscentriclist. MostoutoftheboxeventclassmappingssimplymatchontheeventClassKeyfield whichispopulatedbythenativeeventparsingmechanism(suchaszensyslog, zeneventlog,zentrap).Thesemechanismsmaygenerateseveraldifferenteventswith thesameeventClassKeyfield;thusothertechniquesareneededtodistinguishbetween sucheventsandpotentiallytoseparatethemintodifferenteventclasses. Thesequencenumberinaneventmappinggivestheorderinwhichmappingsaretested againsttheincomingevent.Dependingonwhichmappingactuallymatches(ifany)will determinetheresultingeventClassoftheevent. 35 Skills1stLtd 13December2010

6.1 Working with event classes and event mappings

Eventsareorganisedinanobjectorientedhierarchy;thusattributesassignedtoa parenteventclassareinheritedbyachildeventsubclass. NeweventclassescanbedefinedbyselectingthelefthandEventsmenu,drillingdown toarelevantsubclass,ifrequired,andthenusingthedropdowntablemenualongside SubClassestoAddNewOrganizer.Thenamesuppliedisthenameofthenewevent class.Forexample,drilldowntothe/Securityeventclassandcreateanewsubclass calledSu. Thesimplestwaytomapaneventtoanewordifferentclassistostartfromanexisting eventintheEventConsole.Thefollowingscenarioexplainsthis,creatinganewevent classmappingcalledsuwhichmapsanincomingeventtotheeventclass/Security/Su. 1. GenerateasyslogFAILEDSUeventattheZenosssystem. 2. OpenanEventConsolethatshowstheeventandinspectitsdetails,includingthe Detailstab. 3. Selecttheeventandusingthetablemenu,selectMapEventstoClass.Select yournew/Security/Suclassfromthedropdownlist.Youshouldbeshownthe eventclassmappingpanel.ClicktheEdittab. 4. Youshouldfindthatthenameoftheneweventclassmappingissettosuand theEventClassKeyissettosu(notelowercasesinbothcases).The eventClassKeyfieldisactuallyderivedfromthecomponentfieldoftheincoming eventinSyslogProcessing.py(aroundline256).Thesummaryfieldoftheevent shouldhavebeencopiedintothemappingExamplebox. 5. AddatextstringtotheExplanationboxsuchasAutoaddedbyeventmapping. 6. AddatextstringtotheResolutionboxsuchasThisisadummyresolution. 7. OpenaZenossGUIwindowthatshowsallSuevents(youmayfinditusefulto haveseveralbrowsertabsopentofocusondifferentaspectsoftheZenossGUI). SelectalltheSueventsandMovetoHistory. 8. GenerateanewSuevent. 9. CheckthedetailsoftheneweventintheEventConsole.Theeventshouldhave mappedtoeventClass/Security/Su.TheseverityshouldbeInfo(blue).The detailsoftheeventshouldshowtheeventClassMappingfieldsetto /Security/Su/su. AnyexistingeventmappingcanbemodifiedstartingfromtheEventslefthandmenu andusetheMappingstab.TheShowAllbuttonatthebottomofthefirstpage,will displayallmappingsinalphabeticalorder.Oncethemappingisselected,changescan bemadefromtheEdittab.

36

Skills1stLtd

13December2010

Figure17:Editdialogueforeventclassmapping

Wheneveryouchangeaneventmapping,itisadvisabletoclear(MovetoHistory)any existingeventsofthatcategorybeforetestingthenewconfiguration.Thisisachieved byselectingoneormoreeventsandusingthedropdowntablemenu. Whenyouareworkingwitheventmappings,don'tforgettheEventtabwhichfiltersan EventConsolebyEventClass;also,anyEventConsolehasafilteravailableatthetop rightofthescreen. Itisusefultorefertoeventclassesusingthebreadcrumbpathseenatthetopofa page,suchas/Events/Security/Su. TesteventscanbecreatedfromtheEventConsoledropdowntablemenu,AddEvent NotethatthisisonlyavailablefromanEventConsolereachedbystartingfromtheleft handEventsmenu,notfromagenericconsoleortheeventsforaspecificdevice(this changedbetweenZenoss2.3and2.4).

37

Skills1stLtd

13December2010

Figure18:Dialoguetocreateatestevent

Alternatively,thecommandlinezensendeventcanbeused(youshouldensureyouare thezenossuser).Thistakesparameters: d device p component k eventClassKey s severity c eventClass port=PORT defaultis8081 server=SERVER defaultislocalhost auth=AUTH defaultisadmin:zenoss Theremainderofthelineaftertheseoptionsisusedforthesummaryfield (strictlytheMessagefieldintheGUIdialoguepopulatestheeventsummaryfield)

38

Skills1stLtd

13December2010

6.2 Rules in event mappings

TheRuleelementofaneventclassmappingusesPythonexpressionstotestany instantiatedfieldoftheincomingeventagainstavalue.Expressionscanbecomplex includingPythonmethodcallsandlogicalANDsandORs.Thedefaulteventfieldsthat aredefined,aregiveninAppendixD3oftheZenossAdministrationGuide.Notethat someofthesefieldsarenotactuallyavailableateventmappingtimenotablyevid, stateChange,count,dedupid,suppidandeventClassMapping.

Figure19:Eventmappinglinetest,showingcomplexRuletestingeventanddeviceattributes

TheRuleelementcanalsousePythonexpressionstotestforvaluesofattributesofthe devicethatgeneratedtheevent.Someofthemethodsandattributesthatare availablefordevicesaredocumentedinAppendixD2oftheZenossAdministration Guide,underthesectiononTALESexpressions(TemplateAttributeLanguage ExpressionSyntaxispartofZope.ZopeistheapplicationserverthatZenossisbuilt on).

39

Skills1stLtd

13December2010

TheRuleelementwillonlybeusediftheeventClassKeyfieldinthemappinghas achievedamatchwiththeincomingevent.Afterthat,ifaRuleexists,itmustbe satisfiedbeforethismapping(andhenceclass)isapplied.

6.3 Regex in event mappings

TheRegexelementofaneventclassmappingcanbeusedtoparsethesummaryfieldof theincomingevent,whichispresentedbytheparsingdaemon(zensyslog,zeneventlog, zentrap).TheRegexelementusesthePythonformatforregularexpressionsandcan usethePythonnamedgroupsyntaxtonotonlycheckforliteralstringsbutalsoto defineregularexpressionsforvariablepartsofastring,andassociatethatvariablepart withaname.VariablepartsofthestringarecapturedintoPythonnamedgroups thismeansthat:

Youcanhaveoneexpressionmatchlotsofsimilarbutdifferentincomingevents Thevariablepart(typicallybetweenthe(?Pand\S+))canbepassedtotherest oftheeventprocessingmechanismasanamedfieldoftheevent. Thus,intheRegex

exitbeforeauth\(user'(?P<eventKey>\S+)',(?P<failures>\S+)fails\):Maxauthtriesreached

(?P<eventKey>\S+)willparsethecharactersafteruser'uptothenext singlequoteandplacethatstringintotheeventKeyfieldoftheevent. Similarly(?P<failures>\S+)willholdthestringthatfollowsacommaand spaceandisendedbyspaceandfails. Matchingtheliteralstringrepresentingabracketrequiresthebackslash escapeorthebracketwillbeinterpretedasametacharacter. TherestoftheeventsummarymustmatchtheliteraltextintheRegex; however,othertextcanappearbeyondtheendaftertriesreached. TheExampleboxusuallyshowsasampleeventsummarythatismatched bytheregularexpressionintheRegexbox.

40

Skills1stLtd

13December2010

Figure20:EventmappingdialoguewithRegexforsufailure

Hence,theeventsummaryfieldcanbeparsedtogeneratenew,userdefinedfieldsfor theeventwhichwillbeshownintheDetailstabofanevent'sdetailsandcanbeusedin anysubsequenteventtransforms. TheRegexelementisonlyusedifboththeeventClassKeyandtheRule(ifany)are satisfied.IftheRulefails,theRegexwillnotbetested,norwillanynamedgroup,user definedfieldsbegenerated.IfaRuledoesnotexistandtheRegexdoesnotmatch,the userdefinedfieldswillnotbegeneratedandtheeventclassmappingtothiseventclass willfail.Noeventtransformswilltakeplace.IfaRuledoesexistandissatisfiedbut theRegexfailsthenanyuserdefinedfieldswillnotbegeneratedbuttheeventclass mappingwillbesuccessfulandanymappingtransformwilltakeplace.

6.4 Other elements of event mappings

TheExampleelementofaneventclassmappingisasamplestringthatisusefulwhen constructingaRegex.TheRegexwillturnrediftheRegexdoesnotmatchtheExample stringwhentheSavebuttonisused. TheExplanationandResolutionelementsofaneventclassmappingarestringsthat canbeconfiguredtoprovidefurtherinformationtoZenossusers.Theyappearinthe Detailstabofanevent'sdetail.Notethattheseelementscanonlybeliteralstrings; theycannotuseeitherstandardoruserdefinedfieldsfromtheevent. ThecombinationofeventClassKey,RuleandRegexdeterminetheeventclassthatwill beassociatedwiththeincomingeventandwhattransforms(ifany)willtakeplace. Theremaystillbemultiplecombinationsofthesethatsatisfyanygivenincomingevent. 41 Skills1stLtd 13December2010

Ifso,theSequencetabisusedtodecidetheprecedenceofevaluationofmatchingevent mappings.Themappingswillbetestedfromthelowesttothehighestsequence number.Onceamatchisfound,anysubsequentmappings(withhighersequence numbers)willbeignored.Generally,amappingwithmorespecificmatchingcriteria willhavealowersequencenumber. Aparticularexampleofeventmappingsthatusesequencenumbers,istheeventclass mappingcalleddefaultmappingwhichmusthaveaneventClassKeyof defaultmapping.Thereareatleast6mappings,allcalleddefaultmapping,outofthe box.Eachmapstoadifferentclass.Adefaultmappingisaspecialcasethatisusedby theeventmappingprocessifnomatchcanbefoundfortheeventClassKeyfield(note thatiftheeventClassKeyfielddoesnotexistthennomappingatallwillbeapplied).In thecasewhereaneventClassKeymatchisnotfound,themappingprocessreevaluates lookingforamatchwiththespecialeventClassKeyofdefaultmapping.Itispossibleto createnewmappings,eitherwiththenameofdefaultmappingor,indeed,witha differentname,providedtheeventClassKeyisdefaultmapping.Thesequencenumbers ofallsuchdefaultmappingsshouldbeadjustedtoprioritisethesedefaultmappings.

7 Event transforms
Transformscanbeusedtodoanumberofcleverthings!Somedefaulteventfieldscan bemodified;new,userdefinedfieldscanbecreated;fieldscanberetrievedfromevents alreadyintheMySQLdatabase.Youcanhavesimpleassignmentsoffieldvaluesorset thembasedoncomplexPythonprograms.Thetransformmechanismcanbeappliedin twoways:

eventclasstransforms eventclassmappingtransforms

PriortoZenoss2.4,aneventclasstransformwasonlyusedforeventsinserteddirectly tothatexacteventclassbytheparsingmechanism(zensyslog,zentrap,zensendevent, AddEventwithEventClassspecified,etc).Ifatransformexistedinaneventclass mappingthatwasused,theeventclasstransformwasnotused. Zenoss2.4introducedcascadingeventtransforms.Thischangesthingsintwoways. Givenaneventclass/Toptestwithasubclassof/T1,ifaneventarrivesthatalready hasclass/Toptest/T1,thentheToptesttransformwillbeapplied,followedbytheT1 transform.Ifaneventarrivesthatdoesnothaveapreallocatedclassbutwhoseevent classisdeterminedtobe/Toptest/T1,bytheRule/Regexoftheeventclassmapping, t1,thentransformswillbeappliedintheorder:
Toptestclass>T1class>t1eventclassmapping

Itisperfectlypossibleforatransformtouseuserdefinedeventfieldsinstantiatedby earliertransforms;however,beveryawarethatifanystatementinatransformfails (perhapsbecauseafielddoesn'texist),thentheprocessingofthattransformwillstopat 42 Skills1stLtd 13December2010

thatpointandnofurtherstatementswillbeexecuted.Anyfurthertransformswillbe executed(atleastuntilanerrorisreached). AlltransformsareexecutedoncetheRuleandRegexelementsofamappinghavebeen successfullytestedandafterdeviceandeventcontexthavebeenapplied.Thus,at transformtime,mostofthestandardeventfieldsareavailable,exceptthosepopulated atdatabaseinsertionstime(evid,stateChange,dedupid,count,eventClassMapping andthefirstTimeandlastTimefieldswillbethesame).Anyuserdefinedfieldscreated bytheRegexarealsoavailable. Eventclasstransformscanbeusefulonthe/Unknownclasstoselectivelychangethe classforeventsthatwouldotherwisebe/Unknown. Otherthanthat,thetwoapplicationsoftransformsareverysimilaritisbasically Pythoncodetomodifyeventfields. Notethatifatransformtriestoreferenceafieldofaneventthatdoesnotyetexist (likecount)thenthatlineofthetransformandanysubsequentlineswillbeignored. Suchanerrorwillnottriggeranyerrormessagesinthetransformdialogue.Inspectthe endof$ZENHOME/log/zenhub.logand$ZENHOME/log/event.logtoseetheerror messagereportingtheabsenceoftheattribute. AclasstransformisconfiguredstartingfromtheEventslefthandmenu,bydrilling downtotheclassinquestion(nottheclassmapping)andthenusethetopdropdown tablemenutobringuptheMore>Transformmenu. Amappingtransformisspecifiedaspartofthesameeventmappingdialoguethat definestheRuleandRegexfields.Ineachcase,ifthePythonsyntaxisincorrect,when youusetheSavebutton,thenthetransformisalldisplayedinredtext,indicatingan error. Figure19onpage39showedaneventmappingcalledlinetestwhichincludesa transformtocreateseveraluserdefinedeventfields,somebasedonvaluesfromthe eventandsomewithvaluesfromthedevicethatgeneratedtheevent.Theevent summaryfieldissettoastringconstructedfromliteraltext,standardeventfieldsand userdefinedfields.

Figure21:Transformforthelinetesteventmapping

43

Skills1stLtd

13December2010

Thelinetestmappingtransformalsodemonstratesthat,attransformtime,the firstTimeandlastTimefieldsarethesame;thevalueofevt.mytimeSpanisalways0. Mostoftheuserdefinedfieldsareassignedtosimpleattributesofeithertheeventor thedevice;forexample,evt.firstTime,device.snmpContact.Theevt.mytimeSpanline demonstratesusingsimplearithmeticontwoeventattributes.Thetwolinesbeforethe enddemonstrateusingaPythonmethodtogetvalues;forexample device.getSnmpStatusString()(notethe()attheendthisisthecluethatitisamethod ratherthananattribute).

7.1 Using zendmd to run Python commands

Sohowdoesoneworkoutwhatattributesandmethodsareavailable?TheZenoss AdministrationGuidedocumentstheEventDatabaseDictionaryinAppendixD3.A dictionaryinPythonisabuiltinobjecttypeformappings;inotherwords,itisaway ofholdingacollectionof<key>,<value>pairs.Thewayyouaccessdatatypically,isto specifythekeyforwhichyourequirethecurrentvalueevt.evid,forexample,asksfor thevalueassociatedwiththeevidkeyfortheevteventobject.AppendixD3provides someofthedictionarykeysavailablefortheeventobjectbutnotallofthem! Similarly,AppendixD2oftheZenossAdministrationGuidedocumentsattributesand methodsavailableinTALESexpressionsbutthisinformationisincompleteandrather misleading(inthatitsaysthoseitemswithparenthesesafterthemaremethodsbut noneoftheitemsactuallyshowparentheses,eventhoughsomearemethods!) Fortunately,ZenossprovidesaPythoncommandlineinterface,zendmd,wherecodefor transformscanbetestedoutandtheattributesandmethodsavailablecanbeexplored. Youshouldrunzendmdasthezenossuser.ThissectionisnotsupposedtobeaPython tutorial,sosomeusefulzendmdcommandsareprovidedinAppendixAofthispaper. Thatsaid,hereareacoupleoftrickswithzendmd.

7.1.1 Referencing an existing Zenoss event for use in zendmd

Ifyouwanttoexploretheattributesandmethodsavailableforaneventorthedevice thatgeneratedtheevent,usingzendmd,youneedawaytoreferenceanevent.When executingatransform,theseobjectsaremadeavailabletoyouautomaticallyastheevt variableandthedevicevariablebutinazendmdtestenvironmentyouneedtosupply these.Thefollowingfigureshowsawaytoaccessanexistingeventineitherthestatus orhistorytablesoftheeventsdatabase.Youneedtosupplytheevidvaluebyfindingan appropriateeventintheEventConsole,bringingupthedetaileddata,andcuttingand pastingtheevidvalueintothestatementinzendmd.

44

Skills1stLtd

13December2010

Figure22:UsingzendmdtosettheevtvariabletoanexistingZenossevent

7.1.2 Using zendmd to understand event attributes

AZenosseventisaPythonobjectitisadictionarydatatypeadatastructureof <key>,<value>pairs.Toseewhatkeys(attributes)areavailable,usethemethod showninthefollowingfigure:

45

Skills1stLtd

13December2010

Figure23:Usingzendmdtoprinteventattribute<key><value>pairs

TherearemoreattributesherethanthosedocumentedintheAppendicesoftheZenoss AdministrationGuidebuttheyallseemtobeavailabletoeventtransforms.In particular,theeventzPropertiesareavailableas_actionand_clearClasses.Youcan alsoseetheuserdefinedfieldsinthe_detailsvalue. Notecarefullytheindentationofthesecondzendmdstatement.Pythonisvery particularaboutindentationtointerpretstructuresuchasforloops.Itdoesn'tmatter howmanyspacesyouindentthebodyoftheforloopbutitmustbeindentedfromthe forlineandeachlineinthemainbodyofthatforloopmusthavethesameindentation. Thebodyofaforloop,insideaforloop,wouldindentfurtherandsoon. 46 13December2010

Skills1stLtd

7.1.3 Using zendmd to understand event methods

Intheprevioussection,evt.__dict__.items()wasusedtounderstandthesimpleattributes availablefortheeventevt.Ifyoualsowanttounderstandthemethodsthatare available,thedirfunctionisuseful:

Figure24:Usingzendmdtoshowattributesandmethodsforanevent

Notethatthisisonlyapartiallisting!

47

Skills1stLtd

13December2010

7.2 Transform examples

7.2.1 Combining user defined fields from Regex with transform
Inthisexample,wewillreturntothe/Security/Susubclassofeventsandcombine regularexpressionsandtransforms.Theobjectiveis,forimportantdevices,toescalate theeventseverityifausertriestosutorootbuttodecreasetheseverityifthesuevent comeseitherfromanunimportantdeviceorifthesuistoaparticularuserid(mysql inthiscase).ImportantdevicesaredeterminedbytheeventfieldDevicePriority(note twocapitallettersinthisfieldname).Thedevicepriorityforadevicecanbechanged fromtheEdittabofadevice'sdetailspage. ThisexampleisthesameasshowninFigure20butherewefocusonthetransform ratherthantheRegex.

Figure25:su_rooteventmappingwithtransform

Theuserdefinedfieldto_user,createdbytheRegex,istestedagainsttheliteralstring 'root'.TheresultislogicallyANDedwithatestofthestandardeventfield DevicePriorityfor>2.IftheresultisTruethenthestandardeventfieldseverityisset to5(Critical). AgainnotethemandatoryPythonindentationfortheclausefollowingtheifstatement. Thesecondtestintheelifstatementisasimilartestbutthebodyoftheelif demonstratestheabilitytooverridetheeventzPropertyzEventActionbyassigninga valuetoevt._action.Inthiscase,ratherthantheeventbeinginsertedintothestatus tableoftheeventsdatabase,itwillgodirectlytothehistorytable.

48

Skills1stLtd

13December2010

7.2.2 Applying event and device context in relation to transforms

EventcontextisappliedthroughthezPropertiestabofaneventclassoreventclass mapping.DevicecontextcomprisestheeventfieldsprodState,Location, DeviceClass,DeviceGroupsandSystems.Valuesforthesefieldsarelookedupin theZenossdatabaseforthedevicethatgeneratedtheevent.Thiseventmapping exampledemonstratestheorderinwhichdevicecontext,eventcontextandthemapping transformareapplied. UsingthezPropertiestab,setthezEventSeverityeventcontextvaluetoError(4), zEventActiontohistoryandzEventClearClassesto/Skills. Testthemappingwithatestevent(usingtheAddEventdropdowntablemenu) ensuringthattheseverityisCritical(5)andthesummaryfieldisThisisadevice contextevent(inordertosatisfytheRegexshown).Thetesteventsetthedevicefieldto server.class.example.orgwhichisincludedintheLocationcalled/Raddle100.The eventClassKeyshouldbesettodevice_contextandtheeventClassshouldbeblank.

Figure26:CombiningaRule,contextandatransformforthedevice_contexteventmapping

TheRuledemonstratesthePythongetattrfunctiontotest:

Theevt.Locationfieldsetbydevicecontext,whichshouldevaluateTRUEatRule timeie.devicecontexthasbeenapplied Theevt._actionfieldthatissetbyeventcontexttohistory.Thetestshownabove actuallyevaluatesTRUEshowingthateventcontexthasnotbeenappliedat Ruletime.

49

Skills1stLtd

13December2010

Similarly,theevt._clearClassesfieldtestevaluatesTRUEshowingthatevent contexthasnotbeenapplied.ThePythonsyntaxforcheckingevt._clearClassesis alittledifferentasthisattributeisdefinedasaPythonlistratherthanastring. Theevt.securitystartsat5inthegeneratedeventandeventcontextsetsitto4. ThistestevaluatesTRUEconfirmingthateventcontexthasnotbeenapplied. Notethatthesyntaxforthelastfieldofthegetattris2singlequotes

Insummary,theRuleandRegexshouldevaluatesuccessfullythetransformwillbe applied. Thetransformdemonstrates:

Changingastandardeventfield,evt.Location,toconcatenatetheoriginalfield valuewithliteraltext,usingthe+operator. Changingtheevt.severityfieldagainitwouldhavebeenmodifiedfromthe originalvalue(5)downto(4)whentheeventcontextwasappliedafterRuleand Regexprocessing.Thetransformchangesitto3. Severaluserdefinedvariablesarecreated.Theevt.myClearClassesline demonstratesthatalluserdefinedfieldsappeartobeoftypestringbut evt._clearClassesisdefinedasaPythonlist(checkbackwiththezendmdoutput shownfortheeventdirectoryinFigure23._clearClassesisfollowedbysquare brackets[]thisdenotesalist.Strictly,referringtothesamefigure,the_details fieldisofPythontypeTuplewhichisanimmutablesequenceratherlikea string.Theroundbrackets()denotethetuple).Thebottomlineisthatyou cannotassignevt.myClearClassestosomethingoftypelistunlessyouusethe joinfunctiontosticktogetherthelistelementsbackintoastringtype. Theuserdefinedfieldsdemonstratethatbothdevicecontextandeventcontext havebeenappliedbytransformtime

50

Skills1stLtd

13December2010

8 Zenoss and SNMP

8.1 SNMP introduction
TheSimpleNetworkManagementProtocol(SNMP)definesManagementInformation Base(MIB)variablesthatcanbepolledtoprovideperformanceandconfiguration information.TheSNMPstandardalsoprovidesforagentstosendeventstoamanager. Version1ofSNMPdefinestheseasTRAPs;version2ofthestandardcallsthem NOTIFICATIONs(Zenosssupportsboth).BothMIBvariablesandTRAPs/ NOTIFICATIONsuseObjectIdentifiers(OIDs)todenotedifferentvariablesandevents. SNMPTRAPsaredistinguishedbytheirEnterpriseObjectId(OID),thegenericTRAP numberandthespecificTRAPnumber. Natively,OIDsaredefinedasstringsofdotteddecimalsthatrepresentapaththrougha treebasedhierarchy,wheretherootofthetreeis1andrepresentstheisoorganisation; ithasasubbranch,3,whichrepresentsorganisations(org);ithasasubbranch,6, whichrepresentstheUSDepartmentofDefense(dod);ithasasubbranch,1,which representsinternet,andsoon.Thus,allOIDsstartwith1.3.6.1. Thereisastandard,MIB2,whichdefinesanumberofvariablesthateverySNMP capabledevicemustsupport;thesearelargelysimple,networkrelatedvariables,such asinterfaceInOctets.InadditiontoMIB2,therearealargenumberofstandardised MIBsdefinedinRequestForComment(RFC)documents;anexamplewouldbeRFC 1493definingthebridgeMIB.ThethirdcategoryofMIBsareknownasEnterprise Specific,whicharespecifictoaparticularvendor'sparticularagentforexample,the CiscoFirewallMIB.EnterprisespecificMIBsoftenincludedefinitionsofEnterprise SpecificTRAPs,inadditiontoMIBvariables. MIBsourcefilestranslatedotteddecimalOIDsintomoremeaningfultext.MIBfiles areavailableformanystandards(liketheHOSTRESOURCESMIB)and,typically,any supplierwhogeneratestheirownenterprisespecificMIBvariablesandTRAPs,should makeavailableasourceMIBfiletoaidthistranslation. SNMPagentstypicallycomeaspartofthebaseOperatingSystem(Windows,Unix, Linux,CiscoIOS);howevertheymaynotbeactivatedautomaticallyandwillrequire someconfiguration.SomeagentssupportlittlemorethanMIB2;otherssupportawide rangeofstandardMIBsandenterprisespecificMIBs. TheSNMPcommunicationprotocolvariesdependingontheversionofSNMP.Version1 usesacommunitynamestringasanauthenticationmechanismbetweenSNMP managerandagent.Managersmustbeconfiguredwiththecorrectcommunitynamesto useforanagent;SNMPagentsmustbeconfiguredforwhichmanager(s)areallowed accesstothem,andwhichSNMPmanager(s)tosendTRAPsto.

51

Skills1stLtd

13December2010

InadditiontorequestingMIB2variables,ZenosswilltrytoaccessthestandardHost ResourcesMIBtogetprocessinformationforservermachines.Itwillalsoattemptto accesstheWindowsInformantMIBforallWindowsserversystems,inordertogetCPU andfilesysteminformation.TheInformantMIBisafreeextensionsubagentandMIB availablefromInformantathttp://www.wtcs.org/informant/index.htm.Notethatthe baseWindowsSNMPagentshouldbeinstalledandconfiguredbeforeinstallingthe Informantextension. OnceSNMPagentsareconfiguredwithcommunitynameandTRAPdestination,a simplewaytotestthemissimplytorecycletheSNMPagent(indeedtheywillneed recyclingafteranyconfigurationchanges).OnaWindowssystem,usetheServices utilitytostopandstartSNMP;onaLinuxsystem,/etc/init.d/snmpdrestartwill usuallysuffice.IneithercaseyoushouldeitherseeacoldstartTRAP(genericTRAP 0)orawarmstartTRAP(genericTRAP1)intheZenossEventConsole.TheDetails tabshouldshowthecommunitynamefromtheTRAPpacket. AnothergoodwayofgeneratingTRAPsistoforceanauthenticationTRAP(generic TRAP4).Aneasywaytodothisistousethesnmpwalkcommandwithabad communityname.Ifthecommunityispublic,forahostsystemcalledzenoss,try:
snmpwalk -v 1 -c public zenoss system snmpwalk -v 1 -c fred zenoss system test with good community to generate several TRAP 4's

8.2 Zenoss SNMP architecture

8.2.1 The zentrap daemon
zentrapistheZenossdaemonthatprocessesincomingSNMPTRAPs.Bydefault, zentrapwillsitonthewellknowSNMPTRAPportofUDP/162thiscanbe reconfigured,ifrequired.BothSNMPversion1TRAPsandSNMPversion2 NOTIFICATIONsaresupported. zentrapprocessingisimplementedbythePythonprogram $ZENHOME/Products/ZenEvents/zentrap.py.

52

Skills1stLtd

13December2010

Figure27:zentrap.pypart1checkingforextra0andprocessingofgenericTRAPs

zentrap.pyparsestheincomingSNMPProtocolDataUnit(PDU)toretrievethe enterpriseOID,thegenericTRAPnumberandthespecificTRAPnumber. ThealgorithmforinterpretingincomingTRAPEnterprisefieldshaschangedseveral timesbetweenZenoss2.2.xand2.4.xbecausesomeagentshaveanextra0definedin theirMIBwhichtheydonotsendonanactualTRAP(seethecommentsinthecodein Figure27).InZenoss2.4.1zentrap.py,thealgorithmfirsttriestofindaMIBinthe ZODBdatabasethatcorrespondswiththeincomingTRAP,withtheextra0;ifthis fails,thenamatchissearchedforwithouttheextra0. ThegenericTRAPs(0through5)aretranslatedtostringssuchassnmp_coldStart. usingtheeventTypedictionary.ForspecificTRAPs(genericTRAP6),eventTypedelivers theconcatenationoftheenterpriseOIDandthespecificTRAPnumber;forexample, 1.3.6.1.4.1.123istheenterprise,thespecifictrapnumberis1234,soeventTypedelivers 1.3.6.1.4.1.123.1234.AnyvariablesoftheTRAP(varbinds)arealsoparsedoutintoOID /valuepairs. Theoid2namefunctionlooksupintheZODBdatabasetoseeiftranslationsare availablefortheenterpriseOID,thespecificTRAPnumberandthevarbindidentifiers, totranslatefromdotteddecimalnotationtotextualstrings. 53 Skills1stLtd 13December2010

Figure28:zentrap.pypart2eventfieldsettings

Thefollowingeventfieldsarethenset:

summary device component eventClassKey eventGroup severity firstTime lastTime community

snmptrapfollowedbyeventType settothedevicename leftblank settoeventType trap 3 settotimestamp settotimestamp settocommunitynamestring(thisisauserdefinedfield)

54

Skills1stLtd

13December2010

8.3 Interpreting MIBs

TohelpdecodeSNMPTRAPenterpriseOIDsfromdotteddecimal(suchas. 1.3.6.1.4.1.8072.4.0.2)intoslightlymoremeaningfultext(likensNotifyShutdown)the zenmibcommandcanbeusedtoimportbothstandardMIBsourcefiles(suchas SNMPv2SMIwhichdefinesstandardOIDs)andvendorspecificMIBs.Thebase directoryforMIBsinlaterversionsofZenossis$ZENHOME/share/mibs. SincemanyMIBsreferencedefinitionsinotherMIBfilesviatheIMPORTsectionatthe topoftheMIBfile,thezenmibcommandneedstoknowwheretolookforprerequisite/ corequisiteMIBs.TheenvironmentvariableSMIPATHshouldbedefinedtoincludeall thestandardsubdirectoriesunder$ZENHOME/share/mibs. ThezenmibcommandwithoutparameterswilltrytoimportallMIBfilesthatarein $ZENHOME/share/mibs/site.AspecificMIBfilecanbeprovidedasaparameter;the commandshouldeitherberunfromthe$ZENHOME/share/mibsdirectory(inwhich caseafullpathnameisnotrequiredandthefileisexpectedtobeinthatdirectory)ora fullyqualifiedpathnamecanbespecified.

8.3.1 zenmib example

Tohelpunderstandthezenmibcommand,hereisaworkedexample.Itusestheagent fornetsnmpwhichistheagenttypicallyshippedwithaLinuxsystem.Theenterprise OIDfornetsnmpis.1.3.6.1.4.1.8072. 1. Recycleanetsnmpagentwith/etc/init.d/snmpdrestart.Inadditiontothe genericcoldstartTRAP,youshouldalsoseeTRAP.1.3.6.1.4.1.8072.4.2.This comesfromthenetsnmpenterprise(.1.3.6.1.4.1.8072). 2. TheactualTRAPisdefinedinthefileNETSNMPAGENTMIB.txtwhichshould beshippedaspartoftheOperatingSystemnetsnmppackage.Typicallythis MIBfilecanbefoundunder/usr/share/snmp/mibs.FindandexamineNET SNMPAGENTMIB.txt.Strictly,theMIBfileisdefiningSNMPV2 NOTIFICATIONs,ratherthanSNMPV1TRAPssearchinthefileforthe stringNOTIFItofindtherelevantlines.AlsonotetheIMPORTSsectionatthe topoftheMIBfile,especiallytheimportfromNETSNMPMIB.Thisindicates thatNETSNMPAGENTMIBisdependentonalsoloadingNETSNMPMIB.

55

Skills1stLtd

13December2010

Figure30:MIBfileforNETSNMPAGENTMIBshowingnotifications Figure29:MIBfileforNET_SNMP_AGENTMIBshowingIMPORTSsection

3. InspecttheNETSNMPMIB.txtfileandsearchforthestringNotifications.You shouldseethatthenetSnmpNotificationPrefixisdefinedasbranch4beneath netSnmpandthatnetSnmpNotificationsisbranch0under netSnmpNotificationPrefix.

Figure31:MIBfileforNETSNMPMIBshowingOIDsfornotificationhierarchy

4. AtthetopofthefileyoushouldfindthelinesthatdefinetheenterpriseOIDfor netSnmp.

56

Skills1stLtd

13December2010

Figure32:MIBfileforNETSNMPMIBshowingOIDfornetSnmp

5. Betweenthem,thesefilesgiveus(almost)theOIDfortheunknownTRAPwe received1.3.6.1.4.1.8072.4.0.2.

1.3.6.1.4.1isthestandardiso.org.dod.internet.private.enterprisesOID whichisdefinedintheIMPORTfromSNMPv2SMI netSnmpis{enterprises8072} netSnmpNotificationPrefixisbranch4undernetSnmp netSnmpNotificationsisbranch0undernetSnmpNotificationPrefix nsNotifyShutdownisNOTIFICATION2undernetSnmpNotifications

6. NotethatsomeSNMPagents(includingthenetsnmpagent)areknowntoomit the0fromtheTRAPthattheyactuallygenerate,whichiswhytheoidfieldinthe DetailstaboftheeventdoesnotquitematchtheOIDspecifiedintheMIBfile. 7. ToimportMIBsintoZenoss,theMIBsourcefileneedscopyingto $ZENHOME/share/mibs/site.CopyNETSNMPAGENTMIB.txttothis directory.AtthispointdonotcopyNETSNMPMIB.txt;wewilldemonstrate theerrormessagewhencorequisiteMIBsarenotavailable. 8. ToimportintoZenossuse:

zenmib run -v10

9. YoushouldseethattheNETSNMPAGENTMIB.txtfileisimportedbutthere thereshouldbeanINFOtaggedmessagesayingtheNETSNMPMIBcouldnot befound.

57

Skills1stLtd

13December2010

10. FromtheZenossGUI,usethelefthandmenutoviewMibs.Youwillprobably findthattheNETSNMPAGENTMIBisnotlisted;alternatively,itmaybethere butifyouclickonit,itshowsnoOIDMappingsandnoTRAPs. 11. CopyNETSNMPMIB.txtto$ZENHOME/share/mibs/siteandrerunthezenmib command.ReturntotheZenossGUIandrefreshtheMibsmenu.Clickingonthe NETSNMPAGENTMIBshouldnowdisplayalonglistofOIDMappingsand threeTRAPs,includingnsNotifyShutdown. 12. RestartthesnmpagentontheZenosssystemwith/etc/init.d/snmpdrestart.You shouldseeaneventintheEventConsolethatnowcontainssnmptrap nsNotifyShutdowninthesummaryfield,ratherthansnmptrap 1.3.6.1.4.1.8072.4.2.Ifthisdoesnotwork,youmayneedtorecyclethezentrap daemon.YoucandothiswiththeGUIfromtheSettingsmenuandchoosethe Daemonstab;or,asthezenossuserfromacommandline,usezentraprestart. 13. ZenosshasimplementedanumberofchangesinthewayMIBsareinterpreted betweenversions2.2.xand2.4.x.RememberfromFigure31that netSnmpNotificationsisbranch0undernetSnmpNotificationPrefix;however, someagentsomitthis0whentheyactuallygenerateTRAPs.Zenoss2.4nowhas processingin$ZENHOME/Products/ZenEvent/zentrap.pytotryandinterpret actualTRAPsbothwithandwithouttheextra0.Theeventconsoleshowedan eventwithOID1.3.6.1.4.1.8072.4.2fortheoriginalevent;comparetheDetails taboftheoriginaleventwiththenewonethatcontainsnsNotifyShutdowninthe summaryfield.Youshouldfindthattheneweventhasanoidfieldof 1.3.6.1.4.1.8072.4.0.2. 14. Examine$ZENHOME/Products/ZenEvent/zentrap.py(aroundline457for Zenoss2.4.1)toseethecodethathandlesthisextra0digitprocessing.

8.3.2 A few comments on importing MIBs with Zenoss

ThereareafewquirkstodowithimportingMIBsintoZenossandthequirkshave changedsubtlyoverseveralversionsbetweenZenoss2.2.xand2.4.x. NotethatMIBsimportedintoZenossareonlyusedforinterpretingSNMPV1TRAPs andSNMPV2NOTIFICATIONsforuseintheEventsubsystem.AlthoughtheOIDs areimportedfromMIBs,theycannotbeusedforMIBbrowsingorwhenworkingwith OIDsforperformancesampling,thresholdingandgraphing.

AlwaysensureyoudoMIBworkasthezenossuser. ThedirectoriescontainingMIBschangedin2007.Theyusedtobeunder $ZENHOME/../common/share;withmorerecentversionsofZenosstheyare assumedtobeunder$ZENHOME/share.ToensureMIBimportingworksrun thefollowingcommandstoestablishsymboliclinks:

58

Skills1stLtd

13December2010

cd $ZENHOME ln -s /usr/local/zenoss/common/share ln -s /usr/local/zenoss/common/libexec

MIBimportingoftenrequiresprerequisiteMIBstoalsobeimported.Zenoss providesmanyofthestandardMIBsintheietf,ianaandirtfsubdirectoriesof $ZENHOME/share/mibs.ToautomaticallyfindtheseprerequisiteMIBs,an environmentvariable,SMIPATHisrequiredwhichincludeseachofthe directoriesunder$ZENHOME/share/mibs.Thebestwaytoensurethisisto modifythe.bashrcfileforthezenossuserandincludethevariablethere(allon oneline):

export SMIPATH=$ZENHOME/share/mibs/iana:$ZENHOME/share/mibs/ietf:$ZENHOME/ share/mibs/irtf:$ZENHOME/share/mibs/site:$ZENHOME/share/mibs/tubs/

Bydefault.zenmibrunv10willtryandimporteverythingunder $ZENHOME/share/mibs/site.Thev10simplyaddsmoreverboseoutput. zenmibshouldcheckintheotherdirectories(in$SMIPATH)forprerequisites. Sometimesthisjustdoesn'tseemtowork! WheneveryouhaveimportedaMIB,checkattheGUIontheMibspage.You shouldseethenameoftheMIBandyoushouldusuallyseenonzerocounts undertheNodesand/orNotificationscolumns.

Figure33:/MibspageshowingsuccessfullyimportedMIBs

59

Skills1stLtd

13December2010

TherearesomeMIBsthatwillresultinzerocounts,forexampleiftheMIB sourcefileonlydefinesSNMPstructureanddoesnotincludethedefinitionfor anyOIDs(thatgettranslatedundertheNodescolumn)oranyTRAPsor NOTIFICATIONs(thatgettranslatedundertheNotificationscolumn).Ifyou importaMIBandgetzerosinbothcolumns,checkthesourcefileoftheMIBto seewhetherthereshouldbeentities. Checktheoutputofthezenmibcommandcarefullyforerrormessages. IfyougetanerrormessageoryougetzerocountsintheGUI,youshouldseein theverboseoutputofthezenmibcommandthatthesmidumpcommandisbeing runwitherroroutputsentto/dev/null.Asadebuggingtool,copyandpastethe smidumplinebutomittheending2>/dev/null.Thisshouldshowmore informationonwhatisgoingwrongwiththemibdefinitionsandprerequisite chainsofIMPORTs.NotethatthesmidumpcommandisonlycheckingMIB viability;itisnotdoinganythingtoactuallyimporttheMIBintoZenoss. smidumpmustspecifyallitsprerequisitesinthecorrectorder.

Figure34:zenmibruncommandwithmissingIMPORTchain

60

Skills1stLtd

13December2010

NotethemessagesaboutmissingCISCOTCandCISCOSMIeventhoughthey havebeenimportedpreviously Alsonotethesmidumpcommandthatcanbecutandpaste,omitting 2>/dev/nullforextradebugging.IfthesmidumpendsinlistingtheMIB,rather thanerrormessages,thentheproblemisnotreallywiththeMIBitselforit's IMPORTchain. AlsonotethesmidumpcheckstheMIBtobecompiledforIMPORTsandhas automaticallyfoundtheprerequisites(pparameters).

Figure35:zenmibruncommandspecificallyincludingprerequisitefiles

EvenifyouhavealreadyimportedaprerequisiteMIBsuccessfully,theZenoss databasedoesnotknowabouttheIMPORTchains;itonlyhasOIDsandTRAPs/ NOTIFICATIONs.Forthisreasonyouwillsometimesneedtorunthezenmib commandwithanymissingprerequisitesonthezenmibline,aheadofthemib thatyouactuallywanttoimport. NotethattheprerequisitefilesthathavealreadybeenloadedgetaDEBUG commentthatsaysitalreadyexitsIsuspectthisshouldsayalreadyexists! ItisalwayspossibletospecifyafullpathnametoaMIBfile.However,ifyou makeanerrorintypingfilenameorpaththenyouwillgetaFailedtolocate mib...messagefromthesmidumpcommand. BeforeattemptingtoreimportaMIB,selecttheMIBandusethetablemenufrom theMibspanelof/MibsandselectDeleteMibs. Skills1stLtd 13December2010

61

BewarethatsomeMIBshavearatherdifferentMIBname(onthefirstlineofthe MIBsource)thanthefilenamethatcontainsthem.Thisprovidesample opportunityforconfusion! BewarethatsomeMIBsarewrong!Forexample,theCISCOCONFIGMANMIB usesatypeofUnsigned64whichisnotlegalinitscontext.Youcanmakethis MIBimportbyeditingthesourcefiletochangebothoccurrencesofUnsigned64to Unsigned32. GoodsitestolookforCiscoMIBsandtheirprerequisitesare:

http://tools.cisco.com/Support/SNMP/ ftp://ftpsj.cisco.com/pub/mibs/ includingV1versionsofV2SNMPMIBs IfyoucannotseeimportedMIBsorthoseMIBscanbeseen,havenonzero countsbutdon'ttranslateincomingevents,trythefollowingcommandsfrom withinzendmd:

dmd.Mibs.reIndex() commit()

Othertrickstotry,foundfromtheZenossfora:

TocheckfromwithinzendmdwhatMIBsareloaded(notethatthesecondline mustbeindented):
for mib in dmd.Mibs.mibs(): print mib commit()

TryrecyclingthezentrapcommandfromtheDaemonstaboftheSettings menuor,asthezenossuser,runzentraprestart.

8.4 The MIB browser ZenPack

ThereisanexcellentcommunityZenPackavailabletoperformMIBBrowsing.Thisis notdirectlyrelevanttoTRAP/NOTIFICATIONprocessing,butitisusefulfor investigatingMIBswithaviewtobuildingSNMPperformancetemplates.Itcanbe downloadedfromhttp://www.zenoss.com/community/projects/zenpacks/mibbrowser. ItprovidesaMIBbrowsertoexploreanyOIDthathasbeenloadedintoZenoss,along withatestfacilitytosnmpwalkaconfigurabledevicetoretrievevaluesforanyselected partoftheMIBtree.

8.5 Mapping SNMP events

ZenossprovidessomeeventmappingsforSNMPTRAPsoutofthebox.Asdiscussedin anearliersection,thefile$ZENHOME/Products/ZenModel/data/events.xml configuresallthestandardmappingssosearchingthisfileforSNMPprovidesinsight fordefaultcustomisation. 62 Skills1stLtd 13December2010

MostSNMPTRAPsmaptotheZenoss/Unknowneventclass.Thereareoneortwo exceptionsforsomegenericTRAPssuchasLinkUp(3),LinkDown(2)andthe AuthenticationTRAP(4).Eventfieldsthatareautomaticallypopulatedbythezentrap processingincludesummary,eventClassKeyandagent.TheDetailstaboftheevent detailshowsthecommunityandoidField/Valuepairs(theoidfieldisarecent additiontotheDetailstab). Thismeansthat,typically,theeventonlymapsontheEventClassKey,whichis interpretedbyzentrap.pyasenterprises.<enterprisenumber>.<specifictrap>ifthe SNMPv2SMIhasbeenimportedor1.3.6.1.4.1.<enterprisenumber>.<specifictrap> otherwise.Thesummaryfieldwillbesnmptrap<enterpriseOID><specifictrap>and theagentfieldwillbesettozentrap. TRAPsandNOTIFICATIONsmayhaveoneormoreTRAPvariables(varbinds).These varbindsappearintheDetailstabofaneventdetailwherethefieldnameisthevarbind OIDandthecorrespondingfieldvalueisthevalueofthatvarbind.Eventclass mappingscanbedevisedwithvariousRule,RegexandTransformelements,toparseout theintelligencefromSNMPTRAPsandeithercreatenewuserdefinedeventfieldsor modifyexistingfields(suchasevt.summary). NotethateventmappingsthatparseoutSNMPOIDsandvarbindsmustbeawareof whethertherelevantMIBshavebeenimported,ornot.IfaMIBisimported,OID mappingbasedonmatchingdotteddecimalnotationwillfailastheMIBOID translationshappenbeforeeventmapping.

8.5.1 SNMP event mapping example

InordertointerpretenterprisespecificTRAPs,mappingsareusuallyrequired.Often anactionormodificationisrequired,effectivelybasedonwhatenterprisetheTRAP camefrom(Cisco,netsnmp,...),soasubclassofeventsarerequiredthatinheritsome commoncharacteristicsbutsomeeventdetailsvarydependingontheexactenterprise specificTRAPnumber. ManyenterpriseTRAPsalsoincludeseveralvarbindsthatneedtobeinterpretedand processed. Inthemappingexampleshownhere,threesmallscriptsareusedtogenerateTRAPs fromthe1.3.6.1.4.1.123enterpriseoneforeachofspecificTRAPs1234,1235and1236. ThefirsttwohaveasinglevarbindwhosestringtypevalueisHelloworld4,wherethe endnumberis4or5;thethirdscriptgeneratesaTRAPwith2varbinds.Notethateach ofthevarbindsexhibittheextra0behaviour,ie.thevarbindfieldwillbe 1.3.6.1.4.1.123.0.1234. 1. Withoutanymapping,whengen_mytrap_1234.shisrun,itwillmaptothe /Unknowneventclass. 2. CreateaneweventsubclassSnmpundertheclass/Skills. 63 Skills1stLtd 13December2010

3. Mapthe1234eventbyselectingitstickboxandusingthetablemenutoMap EventstoClass.Choose/Skills/Snmpfromthedropdownselectionbox.Leave therestoftheEventClassMappingparametersasdefaultsfornow.Thismeans thattheeventonlymapsontheeventClassKey,whichtranslatesto<enterprise OID>.<specifictrap>.Themappingnameisautomaticallyassignedthenameof theeventClassKey(1.3.6.1.4.1.123.1234ifSNMPv2SMIisnotimported; enterprises.123.1234ifitis).Referbacktothesnippetofthezentrapcodein Figure33formoreinformationontheparsingoftheTRAPintoeventfields. Checkthatyoureventclassmappingworks. Thenextstepistointerpretthevarbind.EachoftheTRAPsgeneratedbythetest scriptscomefromtheEnterprise1.3.6.1.4.1.123andhenceeachofthevarbindfieldsin theDetailstabwillstartwith1.3.6.1.4.1.123.Atransformwillextractthatpartofthe OIDafter1.3.6.1.4.1.123.Itwillalsosubstitutethevalueofthevarbindintotheevent summary. 1. ReturntoyourEventClassMappingcalled1.3.6.1.4.1.123under/Events/Skills/ Snmp. 2. EdittheTransformbox:
for attr in dir(evt): if attr.startswith('1.3.6.1.4.1.123.'): evt.myRestOfOID=attr.replace('1.3.6.1.4.1.123.','') evt.myFieldValue=getattr(evt,attr) evt.summary=(evt.summary + + evt.myFieldValue)

3. Thedir(evt)syntaxprovidesalistofthemethodsandattributesavailableforthe currentevent,evt. 4. Thestartswithlineensuresthattransformsonlytakeplacewithattributesthat startwith1.3.6.1.4.1.123ie.varbindattributefields. 5. NotethatthereplacelineisreplacingtheOIDspecified,withthenullstring thesyntaxafterthecommaissinglequotesinglequote.Therestoftheattribute (ie.the0.1234bit)iskeptandbecomesthevalueoftheuserfieldmyRestOfOID. 6. Thegetattrlinegetsthevalueofthegivenattributeoftheevent. 7. Runningthescripttogeneratea1234TRAPshouldnowgenerateanevent with:

Theeventmappedtothe/Skills/Snmpclass Thesummaryfieldshouldsaysnmptrap1.3.6.1.4.1.123.1234Helloworld 4. TheDetailstabofthedetaileddatashouldshowvaluesforcommunity,oid, myFieldValueandmyRestOfOID,inadditiontothedefaultvarbind field/valuepairof1.3.6.1.4.1.123.0.1234/Helloworld4

64

Skills1stLtd

13December2010

8. Runningthescripttogeneratea1235TRAPwillstillgenerateaneventwith the/UnknownclassastheeventclassmappingisbasedontheeventClassKeyof 1.3.6.1.4.1.123.1234. Sofar,weareonlymatchingasingleSNMPTRAPwiththeeventClassKeyfield.The objectiveistomapalleventsfromtheenterprise1.3.6.1.4.1.123.WithSNMP,you oftenwanttoapplyatransformtoseveralsimilareventswhichareoftendistinguished bythelaterpartsoftheOIDfield.Thetestscriptsallgenerateeventswhose eventClassKeystartwith1.3.6.1.4.1.123.buttheydifferinthelastnumber. ARulewillbeusedtomatchallappropriateevents.However,aRuleisonlyinspected iftheeventClassKeyhasalreadymatchedsuccessfullyandwehavenocontroloverthe eventClassKeythatissetbyzentrap.py.Thus,thedefaultmappingconceptwillbe used. 1. ClearallSNMPeventsforyourZenosssystem. 2. Editthe1.3.6.1.4.1.123.1234mapping.

IntheRuleboxputevt.eventClassKey.startswith('1.3.6.1.4.1.123.') ChangetheNameofthemappingto1.3.6.1.4.1.123 Savethemappingaway

3. Runthegen_mytrap_1234.shscriptandthegen_mytrap_1235.shscript. 4. ChecktheeventsintheEventConsole 5. Youshouldfindthatthe1234TRAPmapssuccessfullybutthe1235TRAP doesn't.Thisisbecausetheinitialtestforeventclassmappingchecksthe eventClassKeythatisstillsetto1.3.6.1.4.1.123.1234sotheprocessingnever evengetsasfarascheckingourRule!Notethatwehavenocontroloverhowthe eventClassKeyfieldispopulatedbytheeventprocessingmechanismitisparsed outforusbyzentrap.py(seeFigure33again). 6. ThisiswherethemagicstringofdefaultmappingcanbeusedintheEventClass Keyfield.SettheEventClassKeytodefaultmapping(Noteitmustbealllower case).IftheprocessofmappinganeventcannotfindamatchfortheEventClass KeythenitwillrerunthemappingprocesswithanEventClassKeyof defaultmapping. 7. Savethemapping. 8. OpentheSequencetab.ThereareseveralmappingsthatallmaponanEvent ClassKeyofdefaultmapping.Chooseasuitablesequencenumberforthenew defaultmapping.Savethemapping. 9. Clearexistingevents.Rerunbothscripts.Checkthatbotheventsnowmap correctly.

65

Skills1stLtd

13December2010

Figure36:MappingforSNMPTRAPwithrule,transformandeventClassKeyofdefaultmapping

Thetesteventsusedsofar,onlyhaveonevarbind.WhatifyourTRAPhasseveral varbindsandyouwanttouseinformationfromeachofthem?Thescript gen_mytrap_1236.shgeneratesaspecificTRAP1236,withtwovarbinds:

varbind1 varbind2

1.3.6.1.4.1.123.0.12361 1.3.6.1.4.1.123.0.12362

Helloworldvarbind161 Helloworldvarbind162

Runningthescriptgen_mytrap_1236.shshouldresultinaneventthatmapstothe /Skills/Snmpclass,withthemyFieldValueandmyRestOfOIDfieldsmatchingthedata inthelastvarbindandthesummaryalsoreflectingonlythedatainthelastvarbind. Tomakethemappingtakeaccountofbothvarbinds: 1. Editthe1.3.6.1.4.1.123mappingandexaminetheTransformbox.Notethatit startswithforattrindir(evt):.Thisconstructwilllooparoundalleventfields. Thesecondlinewillselectjustthefieldnamesthatstartwith.1.3.6.1.4.1.123.. 2. Toconstructaneventsummarythatcontainsinformationfromeachvarbind eventfield,changetheevt.summaryfieldto:

evt.summary=(evt.summary+ +evt.myFieldValue+ )

3. ClearoldSNMPeventsandrungen_mytrap_1236.sh.Yoursummaryfield shouldnowreflectthedatainalloftheTRAPvarbinds.

66

Skills1stLtd

13December2010

Supposeyounowwantedtomodifythesummaryfieldsothat,ifitstartedwiththe standardsnmptrapfollowedbytheOID,andthesummaryfieldnowhashad appendedanumberofvarbinds,atleastoneofwhichcontainsthestringHello,then chopthesnmptrapOIDbitoffthefrontofthesummary. HereisatransformthatusesthePythonsplitandjoinmethodstodothat.

Figure37:EventtransformforSNMPTRAPwithsplitandjoin

ThesecondifclausedemonstratesthePythonfindfunctiontosearchforthestring Hello(itreturnstheoffsetinthestringofthesubstringyouspecified(thefirst characterhasposition0)ifthesubstringdoesn'texistthenfindreturns1). evt.summary.split()[3:]splitstheevt.summaryfieldintoalistofstringsandthe[3:]on theendselectseverythingafterthethirdelementinthelist.Theresultisthatthe substringssnmp,trapandtheOIDarechoppedoffthebeginningofevt.summary. Theproblemisthattheresultingdatastructureisstillalistofstringsand evt.summaryisdefinedasastring,notalist.Torectifythis,thePythonjoinfunctionis usedtoconvertoursummarybackintoastring. Checktheendof$ZENHOME/log/zenhub.logand$ZENHOME/log/event.logfor debugginghelp.

67

Skills1stLtd

13December2010

9 Event Commands
Whenaneventoccurs,itispossibletorunacommandontheZenossserver.Thefields oftheeventandtheattributesofthedevicethatgeneratedtheevent,aremade availabletouseintheeventcommand.Eventcommandsareshellscripts(thoughthey cancallotherprogramssuchasPythonscripts).Itispossibletofurthercustomisesuch commandstobuildinadelaybeforeexecutionandtorunthescriptatrepeated intervals. EventCommandshaveaccesstotheeventfieldsanddeviceattributes,usingTALES expressions(TemplateAttributeLanguageExpressionSyntax,fromZope)toreference fieldsoftheevent,evtandattributesofthedevice,dev.SeeAppendixDoftheZenoss AdministrationGuideformoredetails.NotethatyoumustuseTALESthe evt.<eventfield>syntaxusedinmappingrulesandtransformsdoesnotworkinevent commands.TALESsyntaxtakestheform:
${evt/<event field>} ${dev/<device attribute>}

9.1 Creating event commands

EventcommandsarecreatedusingthelefthandEventManagermenu,andthenselect theCommandstab.

Figure38:Creatingneweventcommands

TypethenewcommandfilenameintheboxatthebottomofthepanelandclickAdd. Clickonthenewnamethatyouhavejustcreatedtodefinethecommand.

68

Skills1stLtd

13December2010

Figure39:DefiningasimpleEventCommandcalledwriteFile

Thedialoguetoconfigureaneventcommandincludes:

Anenabledflag Atimeoutfieldforthecommand Adelayfieldsuchthatadelaycanbebuiltinbetweentheeventoccurringand thecommandrunning.Forglitchscenarios,aneventmaybeclearedrapidlyso anautomationscriptshouldbedelayedpendingthatpossibility. Arepeatfieldthatdenoteshowfrequentlytorepeatthecommand Differentcommandscanberundependingonwhethertheeventisreportinga problemorwhethertheeventisbeingcleared(movedtothehistorytableofthe eventsdatabase). Throughtheuseofoneormorefilters,itispossibletobeveryspecificaboutwhat eventsshouldresultincommandsbeingrun.Filtersshouldbeusedjudiciouslyor performancedegradationislikelytotakeplaceifeventcommandsrunonmost events.

69

Skills1stLtd

13December2010

Figure40:Eventcommanddialoguewithfilteroptionsdropdown

FiltersforanEventCommandarelogicallyANDed.ToimplementalogicalORof criteria,createoneormoreenabledeventcommands.Eachandeverycommandthat satisfiesitsfilteringcriteria,willbeexecuted.Ifmultiplecommandsarevalid,theywill allberun. Eventcommandsareexecutedasynchronouslybythezenactionsdaemonwhich,by default,runseveryminute.Commandswillberunonceforallopeneventsinthe statustableoftheeventsdatabase,providedthecommandhasnotyetbeenrun.The alertstatetableoftheeventsdatabaserecordswhetheracommandhasbeenrunor notforaparticularcommand,foraparticularevent. zenactionsalsorunsanyalertingrules(discussedinthenextsection).It'slogfileis $ZENHOME/log/zenactions.log.

9.2 Debugging event commands

Youwillfindthatduplicateeventsdonotruneventcommands.Ifloggingfor zenactionsisincreased,itispossibletounderstandwhatisgoingon.Toincreasethe debugginglevelofzenactionstoDebug(10),usethelefthandSettingsmenuandthe Daemonstab.ReferbacktoPage27wherethisprocedurewasdiscussedforincreasing loggingforthezensyslogdaemon.Youwillneedtorecyclethezenactionsdaemonfor thistotakeeffect. Intheexamplebelow,theeventcommandiscalledwriteFile.Itlogstheeventfields firstTime,lastTimeandcounttoanoutputfile,forbothcommandsandclearcommands,

70

Skills1stLtd

13December2010

forbothgoodnewsandbadnewsevents.Initially,abadnewseventisgenerated, followedbyaduplicatebadnewsevent,followedbyadifferentbadnewsevent.

Figure41:zenactions.logshowingactionsforwriteFile

Onexaminationofzenactions.log:

Youshouldseezenactionswakingupeveryminutetoprocesscommands YoushouldseeaSELECTonthestatusdatabaserelevanttowriteFile Youshouldseetheechocommandbeingexecuted Youshouldalsoseealinesimilarto:

INSERTINTOalert_stateVALUES('0a0000833781ac5cffbfcc4','','writeFile', NULL)ONDUPLICATEKEYUPDATElastSent=now()

71

Skills1stLtd

13December2010

ReferbacktoFigure5onpage12forthedefinitionofthealert_statetablein theeventsdatabase Thislineiseffectivelypreventinganyfurthercommandactionsbythis commandonthisevent

ThereisalsoaSELECTonthehistorydatabase

Ifagoodnewsisthengenerated,itshouldclearallofthebadnewseventsandrunthe scriptfortheClearCommand.

Figure42:zenactions.logshowingclearingeventcommands

zenactions.logshouldshow:

SELECTstatementsretrievingdatafromstatusandhistorydatabases Skills1stLtd 13December2010

72

Anechocommandforeachclearedevent ADELETEFROMalertstatestatementforeachclearedevent,whichis effectivelyclearingtheduplicatesflagforthewriteFilecommandforeach event

Interestingly,inspectingtheoutputfilegeneratedbytheeventcommand,showsthat theeventfirstTime,lastTimeandcountfieldsareavailabletoeventcommands(unlike duringtheeventmappingprocess),sincethegoodnewslinesthatareoutputinclude thecorrectvaluesfortheduplicatedevent.Thisisperfectlyreasonableas zenactions.logisshowingthattheinformationisactuallyqueriedfromtheMySQL statusandhistorydatabasetables.

73

Skills1stLtd

13December2010

10 Events, Alerts & Production Status

10.1 Alerting rules for email and paging
Zenossprovidestwostandardmechanismsforreportingeventstousersemailand paging.Thesearesetuponaperuserbasis.Themailserver/pageserverhost parametersneedtobedefinedonceforaZenosssystem.UsethelefthandSettings menutoconfigureparametersforamailserver.

Figure43:SettingupamailserverdestinationfortheZenosssystem

NotethatthisconfigurationisgenericforthewholeZenosssystem,forsendingemails; individualusersalsoneedconfigurationtoreceiveemailsorpagealerts.User parameterscaneitherbeconfiguredfromthelefthandSettingsmenu,andselectthe Userstab;or,usethePreferenceslinkatthetoprightoftheZenossGUI.

Figure44:Configuringemaildestinationaddressforeachuser

NotethatthereisahandyTestbuttonalongsidetheemailaddress.

74

Skills1stLtd

13December2010

Alertingrulesarealsoconfiguredonaperuserbasistodeterminewhateventsto alerton,howtoalertandwhentoalert.Thealertingrulesusethesamefiltering mechanismasEventCommandstodecidewhicheventsshouldgeneratealerts. Tosetupalertingrules,navigatetoauser'sPreferencespageandselecttheAlerting Rulestab.UsethedropdowntablemenutoAddAlertingRulesandprovideaname.

Figure45:DefiningmethodandfiltersforAlertingRules

Notethatiftheemailaddressfieldisleftblank,theaddressconfiguredintheuser's Preferencespagewillbeused. Emailscanbedelayedforaperiodafteraneventhasoccurredandthealertcanbe repeated,ifrequired. ThebottomhalfoftheAlertingRulesEditpaneldefineswhateventsgeneratealerts;it isverysimilartothefiltersseeninEventCommands.Acombinationoffilterscanbe appliedtodeterminewhetheranAlertissent,ornot.AswithEventCommands,filters arelogicallyANDed. TheMessagetabdefinesthecontentoftheemailtobesent.Themessageformatisa Pythonformatstringwithusefuladviceatthebottomofthepanelastowhatfieldsare availableforsubstitution.Thescreenshotbelowisthedefault.

75

Skills1stLtd

13December2010

Figure46:Themessageconfigurationofanemailalert

Notethatintheabovescreenshot,theBodymessagehasfurtherlinesabovethatdetail theDevice,ComponentandSeverity,similartothoselinesseenintheClearBodypanel. TheScheduletabcanbeusedtorestrictwhenalertsshouldbeactive.Bydefault,they arealwaysactive.

Figure47:Alertingrulescheduleexample

AlertsaregeneratedbythesamezenactionsdaemonthatrunsEventCommands.

10.2 Other alerting possibilities

AlthoughZenossonlyprovidesforemailandpagingoutofthebox,itisperfectly possibletocraftotheralertingmechanismsusingEventCommands.Forexample,an EventCommandcouldbeusedtodriveanSNMPTRAPscript,passingparametersfrom 76 Skills1stLtd 13December2010

theZenosseventinTRAPvarbinds,toahigherlevelSNMPmanager.Hereisasimple TRAPgenerationexample:

Figure48:gen_alert_trapscripttogenerateSNMPTRAPwitheventparameters

ThescriptisthendrivenbyaZenossEventCommand:

77

Skills1stLtd

13December2010

Figure49:ZenossEventCommandtodriveSNMPTRAPscript

10.3 The effect of device Production Status

TheProductionStatusofadevicecanbeusedtocontroldifferentmanagement aspectsofasystem.ProductionStatusforadeviceisconfiguredintheEdittabofa device'shomepage. Chapter8oftheVersion2.4ZenossAdministrationGuidedescribesthedifferent ProductionStatesandtheeffectthatthesehave.Threedifferenttypesofmanagement aredefined:

Monitoring Alerting Dashboard

pingpollingandeventgeneration generatingalerts(emails,pagers,eventcommands) whethertoincludeintheDeviceIssuesportlet

Inpractise,anythingtodowithalerting,includingeventcommands,iscontrolledbythe filtersinthealertingruleortheeventcommand.IfnoProductionStatefilteris configuredinthealertingruleoreventcommand,thenthealert/commandwillrun,by default. AdeviceProductionStatusofProductionwillresultineventscontributingtotheDevice IssuesportletoftheZenossDashboardandinIPmonitoringtakingplace.Alertswillbe generatedandeventcommandswillbeexecutediftheirfilterspermit. AproductionStatusofDecommissionedshouldresultinIPmonitoringceasing;hence, alleventswillceaseandnoalertswillbegenerated.Thedevicewillnotberecordedin theDashboardDeviceIssuesportlet.NotethattheoverallStatusicononadevice's Statuspagewillturngreen!Ifthedeviceisalreadydownwhenthestatusissetto DecommissionedthenyoushouldseeexistingInterfacedownmessagesclearedto HistorybyaclearingeventwhosemessageisNolongertestingdevice.... 78 Skills1stLtd 13December2010

AnyProductionStatusotherthanProductionwillresultinthedevicenotbeing includedontheDashboardDeviceIssuesportlet. TheProductionStateofadevicecanbechangedautomaticallytoallowformaintenance windows.Thisisachievedfromadevice'sstatuspageselectMorefromthetable dropdownmenuandAdministration.Maintenancewindowsaredisplayed;theycanbe modified,addedanddeleted. Theexamplebelowshowsthemaintenancewindowforabackupsystemthatonlyruns forabout3hoursatnight,between23:45and3:00.From3:00,for20hoursand50 minutes,itismovedtoaProductionStatusofOriginal,whichforthismachineis definedasDecommissioned.Thusmonitoring,eventsandalertsareonlygenerated duringthefewhoursthatthisbackupsystemisup.

Figure50:Maintenancescheduleformachinethatonlyrunsfrom23:45until3:00

Notethatmaintenanceschedulescanalsobeappliedtodeviceclasshierarchies;theydo nothavetobeappliedtospecificdevices.

11 Conclusions
ZenosshasanextensiveeventsystemcapableofreceivingeventsfromWindows,syslogs andSNMPTRAPs,inadditiontoreceivingtheeventsgeneratedinternallybyZenoss's owndiscovery,availabilityandperformancemonitoring. AlargenumberofeventclassesaredefinedandconfiguredwhenZenossisinstalled. Thesecanbemodified,removedoraddedto. Aneventfollowsafairlycomplexeventlifecycleprocesswherebyitismappedtoan eventclassandthen,optionally,itistransformedsuchthatdefaultfieldsoftheevent canbechangedanduserdefinedfieldscanbecreated. EventmappingforeventsfromWindows,syslogsorSNMP,dependsontheinitial ZenossparsingdaemondeliveringaneventClassKeyfieldwhichmustcorrespondtoa definedmapping.Subsequently,aPythonRuleand/oraPythonRegexcanbeusedto furtherdistinguishbetweenincomingeventsandmaptodifferenteventclasses. 79 Skills1stLtd 13December2010

AneventclassincludeseventcontextzEventAction,zEventSeverityand zEventClearClasseswhichcanbeappliedtoindividualsubclassesofeventsortoclass hierarchies.Thismeanstransformscanbeaffectedbyeventtype. Devicecontextisalsoappliedtoanincomingevent;devicecontextincludesthe prodState,Location,DeviceClass,DeviceGroupsandSystemsfieldvalues.Device contextprovidestheabilityfortransformstotakeaccountofthedeviceordeviceclass hierarchy. Eventtransformscanbesimpleassignmentofeventfieldsorcanincludecomplex Pythonprograms.AgoodenvironmentfortestingPythonisthezendmdcommandline utility.Transformsand/ortheeventcontextcanbeusedtohelpcleareventsthathave beenresolved.AnyeventwithaseverityofClearedwillautomaticallyclearother similarevents;zEventClearClassescanbeusedtolistextraclassesthatareclearedin addition. EventsaresavedintheMySQLeventsdatabase.Bydefault,eventsgotothestatus table;whentheyarecleared,theyaremovedtothehistorytable. Wheneventsoccur,alertscanbegeneratedusingemailorapagingsystem; alternatively,anyscriptcanberunontheZenosssystem,asaneventcommand. Aswithanyenterprisemanagementsystem,Zenosshasthetoolstoconfigurealmost anyresponsetoanyevent.

80

Skills1stLtd

13December2010

12 Appendix A zendmd commands useful with events

zendmdisautilitysuppliedbyZenosswhichprovidesaPythonshellenvironmentwith accesstotheZenossobjectdatabase(ZEO).Thiscanbeuseful,especiallyfortesting eventmappingtransforms. zendmdshouldberunastheZenossuser.Commandrecall(uparrowkey)isavailable andextremelyuseful. NotethatPythonisveryparticularaboutlineindentation;somesyntaxrequiresextra indentation(forexamplethebodyofaforloop).Itdoesnotmatterhowmanyspacesare usedbutthenumbermustbeconsistentforthewholeofthatbody.

Printalleventclassmappinginstancesforaneventclass,Skills:
print dmd.Events.Skills.instances.objectIds()

Printalleventswiththeireventid:
print dmd.ZenEventManager.getEventList()

Printalleventclasses:
for ec in dmd.Events.getSubOrganizers(): print ec.getOrganizerName()

Printalleventclasses(noteventclassmappings)thathavetransforms:
for ec in dmd.Events.getSubOrganizers(): if ec.transform: print ec.getOrganizerName()

Setupthevariableevttopointtoanexistingevent.Thisisanextremelyuseful testingtechnique!

FromanEventConsole,bringupdetailsforanevent.Copytheeventid.
evt=dmd.ZenEventManager.getEventDetailFromStatusOrHistory(<paste id>) print evt.summary print evt._details

Printallattribute/valuepairsforanevent.Thisincludesuserdefinedevent fieldsinthe_detailsfieldandalsothefieldsforeventzProperties(againwith attributenamesprefacedwithanunderscore(eg._action).Notethatthis commanddoesnotshowmethodsforevt,onlyattributes.Notethesyntaxaround dictbelow,is2underscoresbeforeandafter.Alsonotethelinehas2dotsinit andacolonontheend!

for key,value in evt.__dict__.items(): print key,value

Printattributesandmethodsforanevent.Notethatthex=linecouldbe omittedandtheprintlinehavexsubstitutedbygetattr(evt,attr) Skills1stLtd 13December2010

81

for attr in dir(evt): x=getattr(evt,attr) print attr,x

Printalistofthedataforanevent,thedetailsforaneventandthefieldsofan event:
print evt.getEventData() print evt.getEventDetails() print evt.getEventFields()

Twodifferentwaystogetattributesfromanevent.Thelatterwillreturnanull stringiftheattributeismissing:
evt.<attribute> ) getattr(evt, <attribute>, '') Note 2 single quotes before

Ifthingsgethorriblymessedup,try:
dmd.Events.reIndex() commit()

82

Skills1stLtd

13December2010

References
1. Zenossnetwork,systemsandapplicationmonitoringhttp://www.zenoss.com/ 2. ZenossAdministrationGuidehttp://www.zenoss.com/community/docs 3. ZenossDeveloper'sGuidehttp://www.zenoss.com/community/docs 4. ZenossFrequentlyAskedquestions(FAQs) http://www.zenoss.com/community/docs/faqs/faqenglish/ 5. ZenossHowToshttp://www.zenoss.com/community/docs/howtos 6. Zenosswikihttp://www.zenoss.com/community/wiki 7. FordocumentationonZenossfunctionality,theZEOobjectdatabaseandZope.try http://www.zenoss.com/community/docs/zenossapidocs/2.1/ 8. ZenossCoreNetworkandSystemMonitoringbyMichaelBadger,publishedby PACKTPublishing,June2008,ISBN9781847194282. 9. SNMPRequestsForComment(RFCs)http://www.ietf.org/rfc.html 10. 11. 12. V1RFCs1155,1157,1212,1213,1215 V2RFCs2578,2579,2580,3416,3417,3418 V3RFCs25782580,341618,3411,3412,3413,3414,3415

13. AsageneralPythonreference,tryLearningPythonbyMarkLutz,publishedby O'Reilly 14. SNMPHostResourcesMIB,RFCs1514and2790http://www.ietf.org/rfc.html 15. ForinformationonTALESexpressions,see http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/AppendixC.stx 16. ForinformationonPythonregularexpressions,see http://www.python.org/doc/2.5.2/lib/resyntax.html and http://docs.python.org/dev/howto/regex.html 17. FortheextensionSNMPMIBfromInformant,goto http://www.wtcs.org/informant/index.htm 18. GoodsitestolookforCiscoMIBsandtheirprerequisitesare:

http://tools.cisco.com/Support/SNMP/ ftp://ftpsj.cisco.com/pub/mibs/ includingV1versionsofV2SNMPMIBs

19. ZenossZenPacksitefortheMIBBrowserZenPack http://www.zenoss.com/community/projects/zenpacks/mibbrowser 20. DatagramSyslogClienthttp://syslogserver.comforsyslogWindowssystems. 21. Raddlenetworkemulationopensourcepackagehttp://raddle.sourceforge.net/ 22. ZenossEventManagementWorkshopavailablefromSkills1stLtd, http://www.skills1st.co.uk/products/courses/

83

Skills1stLtd

13December2010

Abouttheauthor
JaneCurryhasbeenanetworkandsystemsmanagementtechnicalconsultantand trainerfor25years.Duringher11yearsworkingforIBMshefulfilledbothpresales andconsultancyrolesspanningthefullrangeofIBM'sSystemViewproductspriorto 1996andthen,whenIBMboughtTivoli,shespecialisedinthesystemsmanagement productsofDistributedMonitoring&IBMTivoliMonitoring(ITM),thenetwork managementproduct,TivoliNetViewandtheproblemmanagementproductTivoli EnterpriseConsole(TEC).AllarebasedaroundtheTivoliFrameworkarchitecture. Since1997Janehasbeenanindependentbusinesswomanworkingwithmany companies,bothlargeandsmall,commercialandpublicsector,deliveringTivoli consultancyandtraining.Overthelast5yearsherworkhasbeenmoreinvolvedwith OpenSourceofferings.

84

Skills1stLtd

13December2010