Professional Documents
Culture Documents
Version 3
September 2009 Updated January 2010 Jane Curry Skills 1st Ltd www.skills-1st.co.uk
Skills1stLtd
13December2010
Synopsis
ThispaperisintendedasanintermediateleveldiscussionoftheZenosseventsystem. ItassumesthatthereaderisalreadyfamiliarwiththeZenossEventConsoleandwith basicnavigationaroundtheZenossGraphicalUserInterface(GUI).Itlooksinsome detailatthearchitecturebehindtheZenosseventsystemthedaemonsandhowthey areinterrelatedanditlooksatthestructureofaZenosseventandtheeventlifecycle. ZenosscanreceiveeventsfrommanysourcesinadditiontoZenossitself.Eventsfrom Windows,UnixsyslogsandSimpleNetworksManagementProtocol(SNMP)TRAPsare allexaminedindetail. TheprocessbywhichanincomingeventistransformedintoaparticularZenosseventis knownaseventmappingandhasanumberofdifferentpossibletechniquesfor performingthatconversion.Thesewillallbeexploredalongwiththecreationofnew eventclasses. OnceaneventhasbeenreceivedandclassifiedbyZenoss,automationmayberequired. AlertingbyemailandpagerarediscussedasistheabilitytorunanyscriptasanEvent Command. ThispaperwaswrittenusingZenoss2.4.1. ThepaperisacompaniontexttotheZenossEventManagementWorkshop.
Notations
Throughoutthispaper,texttobytyped,filenamesandmenuoptionstobeselected,are highlightedbyitalics;importantpointstotakenoteofareshowninbold.
Skills1stLtd
13December2010
Table of Contents
1Introduction..........................................................................................................................5 2Zenosseventarchitecture....................................................................................................5 2.1EventConsole...............................................................................................................5 2.2EventdatabasetablesandtheEventManager.........................................................9 2.3Eventlifecycle............................................................................................................13 2.3.1Eventgeneration.................................................................................................15 2.3.2Applicationofdevicecontext..............................................................................16 2.3.3Eventclassmapping...........................................................................................17 2.3.4Applicationofeventcontext...............................................................................18 2.3.5Eventtransforms.................................................................................................18 2.3.6Databaseinsertions............................................................................................19 2.3.7Resolution............................................................................................................20 2.3.8Ageingoutevents................................................................................................21 3EventsgeneratedbyZenoss..............................................................................................21 3.1zenping........................................................................................................................22 3.2zenstatus.....................................................................................................................23 3.3zenwin..........................................................................................................................23 3.4zenprocess...................................................................................................................24 3.5zenperfsnmp................................................................................................................24 3.6Availabilitymonitoringdaemonsanddevicestatuspages.....................................24 4Syslogevents......................................................................................................................25 4.1Configuringsyslog.confandsyslogng.conf..............................................................26 4.2Zenossprocessingofsyslogmessages.......................................................................27 5ZenossprocessingofWindowseventlogs.........................................................................34 6EventMapping...................................................................................................................35 6.1Workingwitheventclassesandeventmappings....................................................36 6.2Rulesineventmappings............................................................................................39 6.3Regexineventmappings...........................................................................................40 6.4Otherelementsofeventmappings...........................................................................41 7Eventtransforms...............................................................................................................42 7.1UsingzendmdtorunPythoncommands..................................................................44 7.1.1ReferencinganexistingZenosseventforuseinzendmd.................................44 7.1.2Usingzendmdtounderstandeventattributes.................................................45 7.1.3Usingzendmdtounderstandeventmethods....................................................47 7.2Transformexamples...................................................................................................48 7.2.1CombininguserdefinedfieldsfromRegexwithtransform.............................48 7.2.2Applyingeventanddevicecontextinrelationtotransforms..........................49 8ZenossandSNMP..............................................................................................................51 8.1SNMPintroduction.....................................................................................................51 8.2ZenossSNMParchitecture........................................................................................52 8.2.1Thezentrapdaemon............................................................................................52 3 Skills1stLtd 13December2010
8.3InterpretingMIBs......................................................................................................55 8.3.1zenmibexample...................................................................................................55 8.3.2AfewcommentsonimportingMIBswithZenoss.............................................58 8.4TheMIBbrowserZenPack.........................................................................................62 8.5MappingSNMPevents..............................................................................................62 8.5.1SNMPeventmappingexample..........................................................................63 9EventCommands...............................................................................................................68 9.1Creatingeventcommands..........................................................................................68 9.2Debuggingeventcommands......................................................................................70 10Events,Alerts&ProductionStatus...............................................................................74 10.1Alertingrulesforemailandpaging........................................................................74 10.2Otheralertingpossibilities......................................................................................76 10.3TheeffectofdeviceProductionStatus....................................................................78 11Conclusions.......................................................................................................................79 12AppendixAzendmdcommandsusefulwithevents......................................................81
Skills1stLtd
13December2010
1 Introduction
ZenossisanOpenSource,multifunctionsystemsandnetworkmanagementtool.There isafree,Coreoffering(whichdoesseemtohavemostthingsyouneed),anda chargeableoffering,Enterprise,whichhasextraaddongoodiessuchashighavailability configurations,distributedmanagementservers,rolebasedaccessandvarioussupport contractswhichincludesomeeducationandconsultancy.Foracomparisonofthefee alternatives,tryhttp://www.zenoss.com/product/pricing. Zenossoffersconfigurationdiscovery,includinglayer3topologymaps,availability monitoring,problemmanagementandperformancemanagement.Itisdesignedaround theITILconceptofaConfigurationManagementDatabase(CMDB),theZenoss StandardModel.ZenossisbuiltusingthePythonbasedZopewebapplicationserver andusestheobjectorientedZopeObjectDatabase(ZODB)astheCMDB,usedtostore Pythonobjectsandtheirstates.ZenossusesZEO,asalayerbetweenZopeandthe ZODB. TherelationalMySQLdatabaseisusedtoholdcurrentandhistoricalevents. PerformancedataisheldinRoundRobinDatabase(RRD)files. ThedefaultprotocolsformonitoringaretypicallyagentlesstheSimpleNetwork Managementprotocol(SNMP),WindowsManagementInstrumentation(WMI)and collectingeventsfromsyslogs.Itisalsopossibletomonitordevicesusingtelnet,sshand touseNagiosplugins. ZenossprovidesagoodGettingStartedwithZenossdocumentalongwithaZenoss AdministrationGuideandaZenossDeveloper'sGuide;youcangetthesefromhttp:// www.zenoss.com/community/docs.ThereisalsoawealthofinformationontheZenoss websitebutitisratherdiffusedbetweenFAQs,HowTos,aWikiandcontributionstothe variousforums.AusefulbookisavailablefromPACKTPublishing,ZenossCore NetworkandSystemMonitoringbyMichaelBadger,whichprovidesmuchofthesame informationastheZenossAdministrationGuidebutinamuchclearerformatwith plentyofscreenshots. ThispaperisanattempttoexpandontheeventinformationintheAdministration Guidebydrawingonmyownexperienceandthecollectedwisdomofthecommunity contributions.
Skills1stLtd
13December2010
Skills1stLtd
13December2010
Figure1:ZenossEventConsole
FromtheEventConsole,oneormoreeventscanbeselectedbycheckingthebox alongsidetheeventandthetablemenudropdowncanbeusedforvariousfunctions includingAcknowledgeEvents,MovetoHistoryandMapEventstoClass. ThecolumnheadersoftheEventConsolecanbeusedtochangethesortingcriteriaand theiconatthefarrightoftheeventcanbeusedtodisplaythedetaileddataofthe event. ThedetaileddatashowsthedefaulteventfieldsundertheFieldstab,anyuserdefined fieldsundertheDetailstabandtheLogtabrecordsactionssuchasacknowledgeand clearing,alongwithdate,timeandtheuserthatperformedtheseactions.Itisalso possibletoaddyourownusermessagestoenevent'sLogbutonlywhileitisinthe statustableoftheeventsdatabase,notonceithasbeenmovedtothehistorytable(Ack /UnAckstatusmakesnodifference).
Skills1stLtd
13December2010
Figure2:DetaileddataforaneventdefaultFieldstab
ThefieldsundertheDetailstabmayincludetextmessagesfortheExplanationand Resolutioneventfields.
Skills1stLtd
13December2010
NotethatifyouwishtoUndeleteaneventfromthehistorytablesofthedatabase,this ispossiblewiththedropdowntablemenu;howeveritdoesnotchangeanyprevious Acknowledgedstatus. Bydefault,theEventConsoleisrefreshedevery60seconds.Ifyouwishtofreezethe console,clicktheStoplinkatthetopoftheconsole(besidetheboxwith60init);the linkchangestosayStart.Toreturntoautomaticrefresh,clicktheStartlink. AnEventConsolecanalsobeaccessedwhichautomaticallyfilterseventsfora particulardevice.NavigatetothemainpageforadeviceandusetheEventstabtoshow alleventsforthatdevice. ThethirdmethodofdisplayinganEventConsoleappliesanautomaticfilterofevent classorsubclass.StartfromtheEventsmenuontheleft.TheEventstabwillshowan EventConsolefilteredbythechosenclassorsubclass.Notethatthetopleveldropdown menuhasanoptiontoAddEvent.Thisisusefulforgeneratingtestevents. EachofthesethreemethodsofaccessinganEventConsoleshowsactiveevents.Tosee eventsthathavebeensenttothehistorytableoftheeventsdatabase,clicktheblue ViewEventHistorylinkatthetoprightoftheEventConsole.
Skills1stLtd
13December2010
Figure3:UsingMySQLtoexaminetheeventsdatabase
10
Skills1stLtd
13December2010
Figure4:Definitionofstatuseventfieldsinzenevents.sql
11
Skills1stLtd
13December2010
Figure5:zenevents.sqlshowingheartbeat,alert_state,loganddetailtables
Someoftheseeventfieldsareparticularlypertinentdependingonhowtheeventwas generated:
12
Skills1stLtd
13December2010
Figure6:EventManageroptionsforMySQLeventsdatabase
Bydefault,statuseventsofseveritybelowError,areagedouttotheeventshistory tableafter4hours.Historicaleventsareneverdeleted.
13
Skills1stLtd
13December2010
Figure7:Eventlifecycle,generationtodatabaseinsertion
14
Skills1stLtd
13December2010
Figure8:Eventlifecycle,resolutiontoageout
15
Skills1stLtd
13December2010
Table2.1.:EventsgeneratedbyZenossitself
Table2.2.:ExternaleventscapturedbyspecialisedZenossdaemons
EventsgeneratedinternallybyZenossneednofurtherprocessingtointerprettheevent. Thedaemonthatgeneratestheeventparsesthenativeinformationandassignsavalue totheeventClassfieldandanyotherrelevantfieldssuchascomponent,summary, messageandagent.TypicallytheeventClassKeyfieldwillbeblank.SomeZenoss daemonspopulatetheeventKeyfield(forexampleanInterfacediscoveryeventwill populatetheeventKeyfieldwiththeIPaddressofthediscoveredinterface). EventsthatareinitiallygeneratedoutsideZenossarecapturedbyzensyslog, zeneventlogorzentrap.Thesedaemonseachhaveaparsingmechanismtointerpret thenativeeventintotheZenosseventformat.ThePythoncodeforthisparsingisin $ZENHOME/Products/ZenEvents.(Bydefault,$ZENHOMEwillbe /usr/local/zenoss/zenoss).SyslogProcessing.pydecodessyslogevents;zentrap.py decodesSNMPTRAPs. Typically,theseparsingmechanismsdonotdeliveravalueforeventClass;ratherthey deliveravaluefortheeventClassKeyfield,alongwithvaluesforsomeotherfields suchascomponent,summary,messageandagent.Itisthenthejoboftheevent mappingphasetodistinguishtheeventclass.
generatedtheeventandthenlookingupthefollowingvaluesforthedeviceintheZODB database:
17
Skills1stLtd
13December2010
18
Skills1stLtd
13December2010
2.3.7 Resolution
Resolutionisgenerallytheprocessbywhichaneventismovedfromthestatustableof theeventsdatabasetothehistorytable. Asecondpossibledefinitionofresolutionisthattheeventisdroppedentirely,never reachingthestatustablethiscanbeachievedeitherfromtheeventcontextbysetting zEventActiontodrop,orinatransformwithevt._action=drop.Thesesamemechanisms canalsobeusedduringinitialeventprocessing,tosettheeventactiontohistory,thus preventingtheeventfromeverappearinginthestatustable. Athirdpossibilityistosettheevent'seventStatefieldtoSuppressedwhichmeansthe eventdoesnotdisplay,bydefault,intheEventConsolestatusdisplay.Thesuppressed eventStatemechanismonlyappearstobeusedbythezenpingdaemon. Aneventcanberesolvedbyhumanintervention.TheEventConsoledropdowntable menuprovidesanoptiontoMovetohistory;theZenossAdministrationGuidedescribes thisashistorifying.ItalsoprovidestheoptiontoAcknowledgeanevent. AcknowledgingchangestheeventStatefieldtoAcknowledged(asopposedtoNew);it changesthecolouroftheeventtoawishywashyversionofthesamecolour.Itdoesnot movetheeventintothehistorytable. Themoreinterestingformsofeventresolutioninvolvecorrelationofevents;thereare twodifferentmechanisms.Thebasicprincipleisthatgoodnewsclearsbadnews. ThefirstclearingmechanismisthatanyeventwithaseverityofClearedwillsearch thestatustableoftheeventsdatabaseandmoveanysimilareventstothehistorytable. SimilarisdefinedashavingthesameeventClass,deviceandcomponentfields.All similareventsarecleared,notjustthemostrecent. Whencorrelationtakesplaceanumberoftheexistingbadnewseventfieldsare updated.stateChange,deletedTimeandclearidareallmodifiedwithclearidbecoming thevalueoftheevidfieldoftheclearing,goodnewsevent.Thegoodnewsevent, withitsClearedseverity,isautomaticallymovedtothehistorytable. Thesecondcorrelationmechanismistospecifyintheeventcontext,oneormoreevent classesinthezEventClearClasseszProperty.Thisattributewillonlybeusedon clearingevents.Theeffectisthatanysimilareventsofthelistedclasseswillalsobe cleared,inadditiontoeventsofthesameclassastheclearingevent.Similarinthis casemeansthesamedeviceandcomponentfields,plustheclassspecifiedin zEventClearClasses.Notethatthesameeffectcanbeachievedinatransformby assigningalistofclassnamestoevt._clearClasses.
20
Skills1stLtd
13December2010
ManualmaintenanceontheMySQLdatabasemayalsoberequired.Zenossprovidesa utilityin$ZENHOME/Products/ZenUtils:
ZenDeleteHistory.py --numDays=10
toclearhistoryeventsolderthan10days
delete from heartbeat where device='localhost'; select * from status where device='mybox';
Showallstatuseventsformybox
select * from detail where evid NOT IN (select evid from status UNION select evid from history); Showdetailrecordsforeventsdeletedfromstatusandhistory delete from detail where evid NOT IN (select evid from status UNION select evid from history); Deletedetailrecordsforeventsdeletedfromstatusandhistory
21
Skills1stLtd
13December2010
Figure9:DefaultparametersforlocalhostCollectors
Parameterstonoteparticularlyare:
3.1 zenping
Themostbasiclevelofavailabilitycheckingistopingpoll.Thezenpingdaemonwill, bydefault,pingpolleachinterface,everyminute.Aninterfacedowneventisgenerated whenthepingfailstogetaresponse.Thiseventisautomaticallyclearedwhena similarpingissuccessful;meantime,whileaninterfaceremainsdown,thecountfieldof theeventisincreased. Thezenpingdaemoncandetectwhenthenetworkpathtoadeviceisbroken,for exampleifasinglepointoffailurerouterisdown.Inthiscase,aneventisgenerated
22
Skills1stLtd
13December2010
withaneventStatefieldofSuppressedandthesummaryfieldreportsnotonlythe interfaceforwhichthepingfailed,butalsothecausaldevice;forexample: ip10.191.101.1isdown,failedatbino.skills1st.co.uk Allotherdeviceavailabilitymonitoringisdependentonpingaccess.Onceapinghas failed,SNMP,process,TCP/UDPserviceandwindowsservicemonitoringwillallbe suspendeduntilpingaccessisrestored.Thecountfieldofthehigherlevelmonitoring eventswillnotincreaseuntilpingaccessisresumed. Alsonotethatifthereisnopingaccess,noperformanceinformationwillbecollected.If adevicereallydoesnotsupportping,perhapsbecauseoffirewallrestrictions,then ensurethatthezPropertyzPingMonitorIgnoreissettoTrue;thiswillpermitSNMPand sshavailabilitymonitoringandperformancedatacollection. Thelogfileforzenpingiszenping.login$ZENHOME/log.
3.2 zenstatus
ThezenstatusdaemoncanbeconfiguredtocheckforaccesstovariousTCPand/orUDP portsonbothWindowsandUnixarchitectures.Bydefault,itcheckseveryminute. Zenosscomeswithahugenumberofservicespreconfigured;thesecanbeexamined fromtheServices>IpServicelefthandmenu.Bydefault,noneoftheseservice monitorsareactive.Servicemonitoringforadevicecanbeconfiguredfromthedevice's mainpageusetheOStabandthetabledropdownmenubesideIPServicestoadda servicetobemonitoredforthisdevice. Aswithpingpolling,agoodnewsserviceeventforadeviceautomaticallyclearsa similarbadnewseventandthecountfieldoftheeventincreaseswhilsttheservice remainsdown. Thelogfileforzenstatusiszenstatus.login$ZENHOME/log.
3.3 zenwin
ThezenwindaemonmonitorsWindowsservices(notTCP/UDPservices).Thesecan beexaminedfromtheServices>WinServicelefthandmenu.Bydefault,noneofthese monitorsareactive.Windowsservicemonitoringforadevicecanbeconfiguredfromthe device'smainpageusetheOStabandthetabledropdownmenubesideWinServices toaddaservicetobemonitoredforthisdevice. zenwinusestheWindowsManagementInstrumentation(WMI)interfacetoaccess servicesontheremotesystemeveryminute,bydefault.ThezPropertiesforadevice(or deviceclass)mustbeconfiguredtoallowaccesstoWMIbeforewindowsservicepolling canbesuccessful. Aswithpingpolling,agoodnewswindowsserviceeventforadeviceautomatically clearsasimilarbadnewseventandthecountfieldincreasesonsubsequentfailed polls. Thelogfileforzenwiniszenwin.login$ZENHOME/log. 23 Skills1stLtd 13December2010
3.4 zenprocess
zenprocessmonitorsWindowsandUnixsystemsforthepresenceofprocesses.Ina Unixcontext,thiswouldbewhethertheprocessappearsinapseflisting;inaWindows context,theprocessmustappearintheWindowsTaskManager(andnotethatthis checkiscasesensitiveonbotharchitectures).Monitoringisevery3minutes,bydefault. Configurationofprocessmonitoringforadeviceissimilarasforservicesusethe device'smainpage>OStabandthetabledropdownmenubesideOSProcessestoadda processtobemonitored. ProcessmonitoringisactuallyachievedusingtheHostResourcesManagement InformationBase(MIB)ofSNMP,byretrievingthehrSWRuntable.Thismeansthat ifSNMPaccesstoadeviceisbroken,therewillbenoprocessinformation. Aswiththeotheravailabilitydaemons,goodnewseventsclearbadnewseventsand thecountfieldincreasesonsubsequentfailedpolls. Thelogfileforzenprocessiszenprocess.login$ZENHOME/log.
3.5 zenperfsnmp
zenperfsnmppollseachdeviceevery5minutes,bydefault.ItcancollectbothSNMP performanceinformationandstatusinformationforprocesses.Processmonitoringis achievedusingtheSNMPHostResourcesMIBsothatifanSNMPagentfails,then processmonitoringisalsoaffected.ThisisnotthecaseforWindowsservicemonitoring. TCP/UDPservicemonitoringalsodoesnotrelyonSNMPonanyplatform. Within5minutesofanSNMPpollfailure,ansnmpagentdowneventshouldbe generated.Withinafurther3minutesthereshouldbeanUnabletoreadprocesseson device..event,ifprocessmonitoringisconfigured.Notealsothatthecountfieldfor individualmissingprocesseventsshouldstopincreasing.Countsformissingservice eventswillbeunaffectedbythelossofSNMP.WhileSNMPaccesstothedevice remainsbroken,thecountfieldfortheUnabletoreadprocessesondevice..eventwill increaseevery3minutes.
thebuttonrepresentstheseverityoftheevent.NotethatAcknowledgingsuchevents hasnoeffectontheDeviceStatuspage.
4 Syslog events
TheUnixsyslogmechanismispervasivethroughoutallversionsofUnix/Linux althoughslightlydifferentversionsandformatsexist.Therearealsoopensource implementationsofsyslogforWindowssystemsandmanynetworkingdevicesalso supportthesyslogconcept. Typicallysystemmessagesareoutputtooneormorelogfilessuchas /var/log/messages.Thesyslogsubsystemcanalsobeconfiguredtosendsyslog messagestoacentralsyslogratherthanholdingfilesoneachsystem.Thewellknown defaultportforforwardingsyslogmessagesisUDP/514. Astandardsyslogsystemisconfiguredbythesyslog.conffile,typicallyin/etc.Anewer versionofsyslogisimplementedonsomesystems,syslogng,whichhasgreater filteringcapabilities.Thesyslogngconfigurationfileistypically/etc/syslogng/syslog ng.conf. Asyslogmessageincludesapriorityandafacility.Theprioritiesare: 0 1 2 3 4 5 6 7 emerg alert crit err warning notice info debug authpriv(10) daemon(3) kern(0) mail(2) syslog(5) uucp(8)
Facilitiesinclude: auth (4) cron (9) ftp(11) lpr(6) news (7) user (1)
Figure10:syslogng.conftosendalleventstoZenosssystemat10.0.0.131(nofilteringactive)
26
Skills1stLtd
13December2010
11. Thenext2linesshowtherawmessageandthedecodingforfacilityandpriority. 12. Linesstartingwithtagshowthezensyslogparsingprocessasitteststhe incominglineagainstvariousPythonregularexpressions,hopefullyendingwith atagmatchline. 13. Ifamatchissuccessful,aneventClassKeymaybedetermined 14. ThelastlineforaparsedeventshouldbeaQueueingevent.
27
Skills1stLtd
13December2010
Wheneverdifferentnativeeventlogsystemsareintegratedthereisalmostinevitablya mismatchofseverities.Thefollowingtabledemonstratesthis. Zenoss Critical(red)(5) Error(orange)(4) Warning(yellow)(3) Info(blue)(2) Debug(grey)(1) Clear(green)(0) syslogpriority emerg(0) alert(1) crit(2) err(3) warning(4) notice(5) info(6) debug(7)
Table4.1.:EventseveritiesforZenoss,syslogandWindows
28
Skills1stLtd
13December2010
Figure11:SyslogProcessing.pyregularexpressionstomatchsyslogtags
ThemainbodyofSyslogProcessing.pystartsbyassigningvaluesfromtheincoming eventtoZenosseventclassfields,asfollows:
def process(self, msg, ipaddr, host, rtime): evt = dict(device=host, ipAddress=ipaddr, firstTime=rtime, lastTime=rtime, eventGroup='syslog')
29
Skills1stLtd
13December2010
Figure12:SyslogProcessing.pyprocessmainroutine
30
Skills1stLtd
13December2010
Figure13:SyslogProcessing.pyparsingofpriority,facilityandseverity
31
Skills1stLtd
13December2010
Figure14:SyslogProcessing.pyprocessingtheheaderinformation
32
Skills1stLtd
13December2010
Figure15:SyslogProcessing.pyparsingthesyslogtag
ThecruxofeventprocessinginZenossistoderiveaneventClassKeythisisdone withthebuildEventClassKeyfunction.
Figure16:SyslogProcessing.pydeterminingtheEventClassKey
33
Skills1stLtd
13December2010
Notethatiftheeventhasthecomponentfieldpopulatedthenthatisusedasthe eventClassKeyaftercheckingforapreexistingeventClassKeyandforanntevidfield.
NoteespeciallythezWinEventLogpropertytoturnon/offeventlogcollectionand zWinEventLogMinSeveritythatprovidesacrudefilteringmechanism(seetheearlier Table4.1onpage28forthedifferentseveritiesforWindowsEventlogs). ItisthezeneventlogdaemoninZenossthatreceivesincomingWindowseventsand parsesthemintoZenossevents.Typically,theSourcefieldontheWindowseventmaps tothecomponentfieldintheZenossevent;theZenosseventClassKeyiscomposedofthe Windows<Source>_<EventID>(eg.Perflib_2003);theZenosseventGroupbecomesthe Windowslogfilename(Application,Security,etc)andtheWindowsEventIDismapped totheZenossntevidfield. ManyWindowseventlogeventsareautomaticallymappedtoeventclassesbutthey mayhavealowseverity(suchasDebug)andtheymayhavetheirzEventActionevent zPropertysettohistorysothattheydonotappearinthestatustableoftheevents database. Watchoutforeventsofclass/Status/Wmi/Conn,typicallyafteraWindowssystem reboot.Untilthiseventismovedtohistory,noothereventswillbereceivedfromthe WMIinterfaceontheWindowssystemasthezenwindaemonwillnotreconnectand zeneventlogwillreceivenoevents. ThereisalsoasyslogutilityavailableforWindowssystemsfromDatagramConsulting athttp://syslogserver.com.TheclientutilityisSyslogAgentandismadeavailable undertheGNUlicense.SyslogserverutilitiesforWindowsarealsoavailableas
34
Skills1stLtd
13December2010
6 Event Mapping
ZenosseventsarecategorisedintoahierarchyofeventClasses,manyofwhichare definedoutoftheboxbutwhichcaneasilybemodifiedoraugmented.Theprocessof EventClassMappingisaboutassociatinganincomingeventwithaparticularZenoss EventClass(settingitseventClassfield)and,potentially,modifyingotherfieldsofthat eventbyusinganeventtransform. Eventclassesandsubclassesaretreatedidenticallyfromthepointofviewofeventclass mapping.Theclasshierarchycanbeusefulinthateventcontext,asimplementedby eventzProperties(suchaszEventSeverity,zEventAction),followsthenormalrulesfor objectinheritanceifzEventActionissettodropontheeventclass/Ignore,thenany subclassesof/Ignorewillalsoinheritthatproperty. NotableoutoftheboxeventzPropertiesarethat/Ignoreclassesandsubclassesdrop incomingevents(ie.theydonotappearineitherthestatusorthehistorydatabases); /Archiveclassesandsubclassesautomaticallymoveeventstothehistorydatabase. Mosteventclasseshaveoneormoremappingsassociatedwiththemtheseareknown asinstances.Notethataneventdoesnothavetohaveanymappingsassociated,in whichcaseaneventofthatclasswillonlyappearinanEventConsoleifthedaemon thatgeneratestheevent,assignstheeventclassatthattime(/Perfeventsmaywell comeintothiscategory,forexample).Outoftheboxeventclassmappingsaredefinedin $ZENHOME/Products/ZenModel/data/events.xml.Theycanbeinspectedfromthe ZenossGUIbyselectingEventsinthelefthandmenuanddrillingdownthesubclass hierarchyundertheClassestab.Alternatively,theMappingstabshowsamore mappingcentriclist,ratherthanaclasscentriclist. MostoutoftheboxeventclassmappingssimplymatchontheeventClassKeyfield whichispopulatedbythenativeeventparsingmechanism(suchaszensyslog, zeneventlog,zentrap).Thesemechanismsmaygenerateseveraldifferenteventswith thesameeventClassKeyfield;thusothertechniquesareneededtodistinguishbetween sucheventsandpotentiallytoseparatethemintodifferenteventclasses. Thesequencenumberinaneventmappinggivestheorderinwhichmappingsaretested againsttheincomingevent.Dependingonwhichmappingactuallymatches(ifany)will determinetheresultingeventClassoftheevent. 35 Skills1stLtd 13December2010
36
Skills1stLtd
13December2010
Figure17:Editdialogueforeventclassmapping
Wheneveryouchangeaneventmapping,itisadvisabletoclear(MovetoHistory)any existingeventsofthatcategorybeforetestingthenewconfiguration.Thisisachieved byselectingoneormoreeventsandusingthedropdowntablemenu. Whenyouareworkingwitheventmappings,don'tforgettheEventtabwhichfiltersan EventConsolebyEventClass;also,anyEventConsolehasafilteravailableatthetop rightofthescreen. Itisusefultorefertoeventclassesusingthebreadcrumbpathseenatthetopofa page,suchas/Events/Security/Su. TesteventscanbecreatedfromtheEventConsoledropdowntablemenu,AddEvent NotethatthisisonlyavailablefromanEventConsolereachedbystartingfromtheleft handEventsmenu,notfromagenericconsoleortheeventsforaspecificdevice(this changedbetweenZenoss2.3and2.4).
37
Skills1stLtd
13December2010
Figure18:Dialoguetocreateatestevent
Alternatively,thecommandlinezensendeventcanbeused(youshouldensureyouare thezenossuser).Thistakesparameters: d device p component k eventClassKey s severity c eventClass port=PORT defaultis8081 server=SERVER defaultislocalhost auth=AUTH defaultisadmin:zenoss Theremainderofthelineaftertheseoptionsisusedforthesummaryfield (strictlytheMessagefieldintheGUIdialoguepopulatestheeventsummaryfield)
38
Skills1stLtd
13December2010
Figure19:Eventmappinglinetest,showingcomplexRuletestingeventanddeviceattributes
39
Skills1stLtd
13December2010
(?P<eventKey>\S+)willparsethecharactersafteruser'uptothenext singlequoteandplacethatstringintotheeventKeyfieldoftheevent. Similarly(?P<failures>\S+)willholdthestringthatfollowsacommaand spaceandisendedbyspaceandfails. Matchingtheliteralstringrepresentingabracketrequiresthebackslash escapeorthebracketwillbeinterpretedasametacharacter. TherestoftheeventsummarymustmatchtheliteraltextintheRegex; however,othertextcanappearbeyondtheendaftertriesreached. TheExampleboxusuallyshowsasampleeventsummarythatismatched bytheregularexpressionintheRegexbox.
40
Skills1stLtd
13December2010
Figure20:EventmappingdialoguewithRegexforsufailure
Hence,theeventsummaryfieldcanbeparsedtogeneratenew,userdefinedfieldsfor theeventwhichwillbeshownintheDetailstabofanevent'sdetailsandcanbeusedin anysubsequenteventtransforms. TheRegexelementisonlyusedifboththeeventClassKeyandtheRule(ifany)are satisfied.IftheRulefails,theRegexwillnotbetested,norwillanynamedgroup,user definedfieldsbegenerated.IfaRuledoesnotexistandtheRegexdoesnotmatch,the userdefinedfieldswillnotbegeneratedandtheeventclassmappingtothiseventclass willfail.Noeventtransformswilltakeplace.IfaRuledoesexistandissatisfiedbut theRegexfailsthenanyuserdefinedfieldswillnotbegeneratedbuttheeventclass mappingwillbesuccessfulandanymappingtransformwilltakeplace.
Ifso,theSequencetabisusedtodecidetheprecedenceofevaluationofmatchingevent mappings.Themappingswillbetestedfromthelowesttothehighestsequence number.Onceamatchisfound,anysubsequentmappings(withhighersequence numbers)willbeignored.Generally,amappingwithmorespecificmatchingcriteria willhavealowersequencenumber. Aparticularexampleofeventmappingsthatusesequencenumbers,istheeventclass mappingcalleddefaultmappingwhichmusthaveaneventClassKeyof defaultmapping.Thereareatleast6mappings,allcalleddefaultmapping,outofthe box.Eachmapstoadifferentclass.Adefaultmappingisaspecialcasethatisusedby theeventmappingprocessifnomatchcanbefoundfortheeventClassKeyfield(note thatiftheeventClassKeyfielddoesnotexistthennomappingatallwillbeapplied).In thecasewhereaneventClassKeymatchisnotfound,themappingprocessreevaluates lookingforamatchwiththespecialeventClassKeyofdefaultmapping.Itispossibleto createnewmappings,eitherwiththenameofdefaultmappingor,indeed,witha differentname,providedtheeventClassKeyisdefaultmapping.Thesequencenumbers ofallsuchdefaultmappingsshouldbeadjustedtoprioritisethesedefaultmappings.
7 Event transforms
Transformscanbeusedtodoanumberofcleverthings!Somedefaulteventfieldscan bemodified;new,userdefinedfieldscanbecreated;fieldscanberetrievedfromevents alreadyintheMySQLdatabase.Youcanhavesimpleassignmentsoffieldvaluesorset thembasedoncomplexPythonprograms.Thetransformmechanismcanbeappliedin twoways:
eventclasstransforms eventclassmappingtransforms
PriortoZenoss2.4,aneventclasstransformwasonlyusedforeventsinserteddirectly tothatexacteventclassbytheparsingmechanism(zensyslog,zentrap,zensendevent, AddEventwithEventClassspecified,etc).Ifatransformexistedinaneventclass mappingthatwasused,theeventclasstransformwasnotused. Zenoss2.4introducedcascadingeventtransforms.Thischangesthingsintwoways. Givenaneventclass/Toptestwithasubclassof/T1,ifaneventarrivesthatalready hasclass/Toptest/T1,thentheToptesttransformwillbeapplied,followedbytheT1 transform.Ifaneventarrivesthatdoesnothaveapreallocatedclassbutwhoseevent classisdeterminedtobe/Toptest/T1,bytheRule/Regexoftheeventclassmapping, t1,thentransformswillbeappliedintheorder:
Toptestclass>T1class>t1eventclassmapping
thatpointandnofurtherstatementswillbeexecuted.Anyfurthertransformswillbe executed(atleastuntilanerrorisreached). AlltransformsareexecutedoncetheRuleandRegexelementsofamappinghavebeen successfullytestedandafterdeviceandeventcontexthavebeenapplied.Thus,at transformtime,mostofthestandardeventfieldsareavailable,exceptthosepopulated atdatabaseinsertionstime(evid,stateChange,dedupid,count,eventClassMapping andthefirstTimeandlastTimefieldswillbethesame).Anyuserdefinedfieldscreated bytheRegexarealsoavailable. Eventclasstransformscanbeusefulonthe/Unknownclasstoselectivelychangethe classforeventsthatwouldotherwisebe/Unknown. Otherthanthat,thetwoapplicationsoftransformsareverysimilaritisbasically Pythoncodetomodifyeventfields. Notethatifatransformtriestoreferenceafieldofaneventthatdoesnotyetexist (likecount)thenthatlineofthetransformandanysubsequentlineswillbeignored. Suchanerrorwillnottriggeranyerrormessagesinthetransformdialogue.Inspectthe endof$ZENHOME/log/zenhub.logand$ZENHOME/log/event.logtoseetheerror messagereportingtheabsenceoftheattribute. AclasstransformisconfiguredstartingfromtheEventslefthandmenu,bydrilling downtotheclassinquestion(nottheclassmapping)andthenusethetopdropdown tablemenutobringuptheMore>Transformmenu. Amappingtransformisspecifiedaspartofthesameeventmappingdialoguethat definestheRuleandRegexfields.Ineachcase,ifthePythonsyntaxisincorrect,when youusetheSavebutton,thenthetransformisalldisplayedinredtext,indicatingan error. Figure19onpage39showedaneventmappingcalledlinetestwhichincludesa transformtocreateseveraluserdefinedeventfields,somebasedonvaluesfromthe eventandsomewithvaluesfromthedevicethatgeneratedtheevent.Theevent summaryfieldissettoastringconstructedfromliteraltext,standardeventfieldsand userdefinedfields.
Figure21:Transformforthelinetesteventmapping
43
Skills1stLtd
13December2010
44
Skills1stLtd
13December2010
Figure22:UsingzendmdtosettheevtvariabletoanexistingZenossevent
45
Skills1stLtd
13December2010
Figure23:Usingzendmdtoprinteventattribute<key><value>pairs
TherearemoreattributesherethanthosedocumentedintheAppendicesoftheZenoss AdministrationGuidebuttheyallseemtobeavailabletoeventtransforms.In particular,theeventzPropertiesareavailableas_actionand_clearClasses.Youcan alsoseetheuserdefinedfieldsinthe_detailsvalue. Notecarefullytheindentationofthesecondzendmdstatement.Pythonisvery particularaboutindentationtointerpretstructuresuchasforloops.Itdoesn'tmatter howmanyspacesyouindentthebodyoftheforloopbutitmustbeindentedfromthe forlineandeachlineinthemainbodyofthatforloopmusthavethesameindentation. Thebodyofaforloop,insideaforloop,wouldindentfurtherandsoon. 46 13December2010
Skills1stLtd
Figure24:Usingzendmdtoshowattributesandmethodsforanevent
Notethatthisisonlyapartiallisting!
47
Skills1stLtd
13December2010
Figure25:su_rooteventmappingwithtransform
48
Skills1stLtd
13December2010
Figure26:CombiningaRule,contextandatransformforthedevice_contexteventmapping
TheRuledemonstratesthePythongetattrfunctiontotest:
49
Skills1stLtd
13December2010
Changingastandardeventfield,evt.Location,toconcatenatetheoriginalfield valuewithliteraltext,usingthe+operator. Changingtheevt.severityfieldagainitwouldhavebeenmodifiedfromthe originalvalue(5)downto(4)whentheeventcontextwasappliedafterRuleand Regexprocessing.Thetransformchangesitto3. Severaluserdefinedvariablesarecreated.Theevt.myClearClassesline demonstratesthatalluserdefinedfieldsappeartobeoftypestringbut evt._clearClassesisdefinedasaPythonlist(checkbackwiththezendmdoutput shownfortheeventdirectoryinFigure23._clearClassesisfollowedbysquare brackets[]thisdenotesalist.Strictly,referringtothesamefigure,the_details fieldisofPythontypeTuplewhichisanimmutablesequenceratherlikea string.Theroundbrackets()denotethetuple).Thebottomlineisthatyou cannotassignevt.myClearClassestosomethingoftypelistunlessyouusethe joinfunctiontosticktogetherthelistelementsbackintoastringtype. Theuserdefinedfieldsdemonstratethatbothdevicecontextandeventcontext havebeenappliedbytransformtime
50
Skills1stLtd
13December2010
51
Skills1stLtd
13December2010
InadditiontorequestingMIB2variables,ZenosswilltrytoaccessthestandardHost ResourcesMIBtogetprocessinformationforservermachines.Itwillalsoattemptto accesstheWindowsInformantMIBforallWindowsserversystems,inordertogetCPU andfilesysteminformation.TheInformantMIBisafreeextensionsubagentandMIB availablefromInformantathttp://www.wtcs.org/informant/index.htm.Notethatthe baseWindowsSNMPagentshouldbeinstalledandconfiguredbeforeinstallingthe Informantextension. OnceSNMPagentsareconfiguredwithcommunitynameandTRAPdestination,a simplewaytotestthemissimplytorecycletheSNMPagent(indeedtheywillneed recyclingafteranyconfigurationchanges).OnaWindowssystem,usetheServices utilitytostopandstartSNMP;onaLinuxsystem,/etc/init.d/snmpdrestartwill usuallysuffice.IneithercaseyoushouldeitherseeacoldstartTRAP(genericTRAP 0)orawarmstartTRAP(genericTRAP1)intheZenossEventConsole.TheDetails tabshouldshowthecommunitynamefromtheTRAPpacket. AnothergoodwayofgeneratingTRAPsistoforceanauthenticationTRAP(generic TRAP4).Aneasywaytodothisistousethesnmpwalkcommandwithabad communityname.Ifthecommunityispublic,forahostsystemcalledzenoss,try:
snmpwalk -v 1 -c public zenoss system snmpwalk -v 1 -c fred zenoss system test with good community to generate several TRAP 4's
52
Skills1stLtd
13December2010
Figure27:zentrap.pypart1checkingforextra0andprocessingofgenericTRAPs
zentrap.pyparsestheincomingSNMPProtocolDataUnit(PDU)toretrievethe enterpriseOID,thegenericTRAPnumberandthespecificTRAPnumber. ThealgorithmforinterpretingincomingTRAPEnterprisefieldshaschangedseveral timesbetweenZenoss2.2.xand2.4.xbecausesomeagentshaveanextra0definedin theirMIBwhichtheydonotsendonanactualTRAP(seethecommentsinthecodein Figure27).InZenoss2.4.1zentrap.py,thealgorithmfirsttriestofindaMIBinthe ZODBdatabasethatcorrespondswiththeincomingTRAP,withtheextra0;ifthis fails,thenamatchissearchedforwithouttheextra0. ThegenericTRAPs(0through5)aretranslatedtostringssuchassnmp_coldStart. usingtheeventTypedictionary.ForspecificTRAPs(genericTRAP6),eventTypedelivers theconcatenationoftheenterpriseOIDandthespecificTRAPnumber;forexample, 1.3.6.1.4.1.123istheenterprise,thespecifictrapnumberis1234,soeventTypedelivers 1.3.6.1.4.1.123.1234.AnyvariablesoftheTRAP(varbinds)arealsoparsedoutintoOID /valuepairs. Theoid2namefunctionlooksupintheZODBdatabasetoseeiftranslationsare availablefortheenterpriseOID,thespecificTRAPnumberandthevarbindidentifiers, totranslatefromdotteddecimalnotationtotextualstrings. 53 Skills1stLtd 13December2010
Figure28:zentrap.pypart2eventfieldsettings
Thefollowingeventfieldsarethenset:
54
Skills1stLtd
13December2010
55
Skills1stLtd
13December2010
Figure30:MIBfileforNETSNMPAGENTMIBshowingnotifications Figure29:MIBfileforNET_SNMP_AGENTMIBshowingIMPORTSsection
Figure31:MIBfileforNETSNMPMIBshowingOIDsfornotificationhierarchy
4. AtthetopofthefileyoushouldfindthelinesthatdefinetheenterpriseOIDfor netSnmp.
56
Skills1stLtd
13December2010
Figure32:MIBfileforNETSNMPMIBshowingOIDfornetSnmp
5. Betweenthem,thesefilesgiveus(almost)theOIDfortheunknownTRAPwe received1.3.6.1.4.1.8072.4.0.2.
57
Skills1stLtd
13December2010
10. FromtheZenossGUI,usethelefthandmenutoviewMibs.Youwillprobably findthattheNETSNMPAGENTMIBisnotlisted;alternatively,itmaybethere butifyouclickonit,itshowsnoOIDMappingsandnoTRAPs. 11. CopyNETSNMPMIB.txtto$ZENHOME/share/mibs/siteandrerunthezenmib command.ReturntotheZenossGUIandrefreshtheMibsmenu.Clickingonthe NETSNMPAGENTMIBshouldnowdisplayalonglistofOIDMappingsand threeTRAPs,includingnsNotifyShutdown. 12. RestartthesnmpagentontheZenosssystemwith/etc/init.d/snmpdrestart.You shouldseeaneventintheEventConsolethatnowcontainssnmptrap nsNotifyShutdowninthesummaryfield,ratherthansnmptrap 1.3.6.1.4.1.8072.4.2.Ifthisdoesnotwork,youmayneedtorecyclethezentrap daemon.YoucandothiswiththeGUIfromtheSettingsmenuandchoosethe Daemonstab;or,asthezenossuserfromacommandline,usezentraprestart. 13. ZenosshasimplementedanumberofchangesinthewayMIBsareinterpreted betweenversions2.2.xand2.4.x.RememberfromFigure31that netSnmpNotificationsisbranch0undernetSnmpNotificationPrefix;however, someagentsomitthis0whentheyactuallygenerateTRAPs.Zenoss2.4nowhas processingin$ZENHOME/Products/ZenEvent/zentrap.pytotryandinterpret actualTRAPsbothwithandwithouttheextra0.Theeventconsoleshowedan eventwithOID1.3.6.1.4.1.8072.4.2fortheoriginalevent;comparetheDetails taboftheoriginaleventwiththenewonethatcontainsnsNotifyShutdowninthe summaryfield.Youshouldfindthattheneweventhasanoidfieldof 1.3.6.1.4.1.8072.4.0.2. 14. Examine$ZENHOME/Products/ZenEvent/zentrap.py(aroundline457for Zenoss2.4.1)toseethecodethathandlesthisextra0digitprocessing.
58
Skills1stLtd
13December2010
Figure33:/MibspageshowingsuccessfullyimportedMIBs
59
Skills1stLtd
13December2010
TherearesomeMIBsthatwillresultinzerocounts,forexampleiftheMIB sourcefileonlydefinesSNMPstructureanddoesnotincludethedefinitionfor anyOIDs(thatgettranslatedundertheNodescolumn)oranyTRAPsor NOTIFICATIONs(thatgettranslatedundertheNotificationscolumn).Ifyou importaMIBandgetzerosinbothcolumns,checkthesourcefileoftheMIBto seewhetherthereshouldbeentities. Checktheoutputofthezenmibcommandcarefullyforerrormessages. IfyougetanerrormessageoryougetzerocountsintheGUI,youshouldseein theverboseoutputofthezenmibcommandthatthesmidumpcommandisbeing runwitherroroutputsentto/dev/null.Asadebuggingtool,copyandpastethe smidumplinebutomittheending2>/dev/null.Thisshouldshowmore informationonwhatisgoingwrongwiththemibdefinitionsandprerequisite chainsofIMPORTs.NotethatthesmidumpcommandisonlycheckingMIB viability;itisnotdoinganythingtoactuallyimporttheMIBintoZenoss. smidumpmustspecifyallitsprerequisitesinthecorrectorder.
Figure34:zenmibruncommandwithmissingIMPORTchain
60
Skills1stLtd
13December2010
Figure35:zenmibruncommandspecificallyincludingprerequisitefiles
EvenifyouhavealreadyimportedaprerequisiteMIBsuccessfully,theZenoss databasedoesnotknowabouttheIMPORTchains;itonlyhasOIDsandTRAPs/ NOTIFICATIONs.Forthisreasonyouwillsometimesneedtorunthezenmib commandwithanymissingprerequisitesonthezenmibline,aheadofthemib thatyouactuallywanttoimport. NotethattheprerequisitefilesthathavealreadybeenloadedgetaDEBUG commentthatsaysitalreadyexitsIsuspectthisshouldsayalreadyexists! ItisalwayspossibletospecifyafullpathnametoaMIBfile.However,ifyou makeanerrorintypingfilenameorpaththenyouwillgetaFailedtolocate mib...messagefromthesmidumpcommand. BeforeattemptingtoreimportaMIB,selecttheMIBandusethetablemenufrom theMibspanelof/MibsandselectDeleteMibs. Skills1stLtd 13December2010
61
Othertrickstotry,foundfromtheZenossfora:
TocheckfromwithinzendmdwhatMIBsareloaded(notethatthesecondline mustbeindented):
for mib in dmd.Mibs.mibs(): print mib commit()
TryrecyclingthezentrapcommandfromtheDaemonstaboftheSettings menuor,asthezenossuser,runzentraprestart.
MostSNMPTRAPsmaptotheZenoss/Unknowneventclass.Thereareoneortwo exceptionsforsomegenericTRAPssuchasLinkUp(3),LinkDown(2)andthe AuthenticationTRAP(4).Eventfieldsthatareautomaticallypopulatedbythezentrap processingincludesummary,eventClassKeyandagent.TheDetailstaboftheevent detailshowsthecommunityandoidField/Valuepairs(theoidfieldisarecent additiontotheDetailstab). Thismeansthat,typically,theeventonlymapsontheEventClassKey,whichis interpretedbyzentrap.pyasenterprises.<enterprisenumber>.<specifictrap>ifthe SNMPv2SMIhasbeenimportedor1.3.6.1.4.1.<enterprisenumber>.<specifictrap> otherwise.Thesummaryfieldwillbesnmptrap<enterpriseOID><specifictrap>and theagentfieldwillbesettozentrap. TRAPsandNOTIFICATIONsmayhaveoneormoreTRAPvariables(varbinds).These varbindsappearintheDetailstabofaneventdetailwherethefieldnameisthevarbind OIDandthecorrespondingfieldvalueisthevalueofthatvarbind.Eventclass mappingscanbedevisedwithvariousRule,RegexandTransformelements,toparseout theintelligencefromSNMPTRAPsandeithercreatenewuserdefinedeventfieldsor modifyexistingfields(suchasevt.summary). NotethateventmappingsthatparseoutSNMPOIDsandvarbindsmustbeawareof whethertherelevantMIBshavebeenimported,ornot.IfaMIBisimported,OID mappingbasedonmatchingdotteddecimalnotationwillfailastheMIBOID translationshappenbeforeeventmapping.
3. Mapthe1234eventbyselectingitstickboxandusingthetablemenutoMap EventstoClass.Choose/Skills/Snmpfromthedropdownselectionbox.Leave therestoftheEventClassMappingparametersasdefaultsfornow.Thismeans thattheeventonlymapsontheeventClassKey,whichtranslatesto<enterprise OID>.<specifictrap>.Themappingnameisautomaticallyassignedthenameof theeventClassKey(1.3.6.1.4.1.123.1234ifSNMPv2SMIisnotimported; enterprises.123.1234ifitis).Referbacktothesnippetofthezentrapcodein Figure33formoreinformationontheparsingoftheTRAPintoeventfields. Checkthatyoureventclassmappingworks. Thenextstepistointerpretthevarbind.EachoftheTRAPsgeneratedbythetest scriptscomefromtheEnterprise1.3.6.1.4.1.123andhenceeachofthevarbindfieldsin theDetailstabwillstartwith1.3.6.1.4.1.123.Atransformwillextractthatpartofthe OIDafter1.3.6.1.4.1.123.Itwillalsosubstitutethevalueofthevarbindintotheevent summary. 1. ReturntoyourEventClassMappingcalled1.3.6.1.4.1.123under/Events/Skills/ Snmp. 2. EdittheTransformbox:
for attr in dir(evt): if attr.startswith('1.3.6.1.4.1.123.'): evt.myRestOfOID=attr.replace('1.3.6.1.4.1.123.','') evt.myFieldValue=getattr(evt,attr) evt.summary=(evt.summary + + evt.myFieldValue)
3. Thedir(evt)syntaxprovidesalistofthemethodsandattributesavailableforthe currentevent,evt. 4. Thestartswithlineensuresthattransformsonlytakeplacewithattributesthat startwith1.3.6.1.4.1.123ie.varbindattributefields. 5. NotethatthereplacelineisreplacingtheOIDspecified,withthenullstring thesyntaxafterthecommaissinglequotesinglequote.Therestoftheattribute (ie.the0.1234bit)iskeptandbecomesthevalueoftheuserfieldmyRestOfOID. 6. Thegetattrlinegetsthevalueofthegivenattributeoftheevent. 7. Runningthescripttogeneratea1234TRAPshouldnowgenerateanevent with:
64
Skills1stLtd
13December2010
8. Runningthescripttogeneratea1235TRAPwillstillgenerateaneventwith the/UnknownclassastheeventclassmappingisbasedontheeventClassKeyof 1.3.6.1.4.1.123.1234. Sofar,weareonlymatchingasingleSNMPTRAPwiththeeventClassKeyfield.The objectiveistomapalleventsfromtheenterprise1.3.6.1.4.1.123.WithSNMP,you oftenwanttoapplyatransformtoseveralsimilareventswhichareoftendistinguished bythelaterpartsoftheOIDfield.Thetestscriptsallgenerateeventswhose eventClassKeystartwith1.3.6.1.4.1.123.buttheydifferinthelastnumber. ARulewillbeusedtomatchallappropriateevents.However,aRuleisonlyinspected iftheeventClassKeyhasalreadymatchedsuccessfullyandwehavenocontroloverthe eventClassKeythatissetbyzentrap.py.Thus,thedefaultmappingconceptwillbe used. 1. ClearallSNMPeventsforyourZenosssystem. 2. Editthe1.3.6.1.4.1.123.1234mapping.
3. Runthegen_mytrap_1234.shscriptandthegen_mytrap_1235.shscript. 4. ChecktheeventsintheEventConsole 5. Youshouldfindthatthe1234TRAPmapssuccessfullybutthe1235TRAP doesn't.Thisisbecausetheinitialtestforeventclassmappingchecksthe eventClassKeythatisstillsetto1.3.6.1.4.1.123.1234sotheprocessingnever evengetsasfarascheckingourRule!Notethatwehavenocontroloverhowthe eventClassKeyfieldispopulatedbytheeventprocessingmechanismitisparsed outforusbyzentrap.py(seeFigure33again). 6. ThisiswherethemagicstringofdefaultmappingcanbeusedintheEventClass Keyfield.SettheEventClassKeytodefaultmapping(Noteitmustbealllower case).IftheprocessofmappinganeventcannotfindamatchfortheEventClass KeythenitwillrerunthemappingprocesswithanEventClassKeyof defaultmapping. 7. Savethemapping. 8. OpentheSequencetab.ThereareseveralmappingsthatallmaponanEvent ClassKeyofdefaultmapping.Chooseasuitablesequencenumberforthenew defaultmapping.Savethemapping. 9. Clearexistingevents.Rerunbothscripts.Checkthatbotheventsnowmap correctly.
65
Skills1stLtd
13December2010
Figure36:MappingforSNMPTRAPwithrule,transformandeventClassKeyofdefaultmapping
varbind1 varbind2
1.3.6.1.4.1.123.0.12361 1.3.6.1.4.1.123.0.12362
Helloworldvarbind161 Helloworldvarbind162
3. ClearoldSNMPeventsandrungen_mytrap_1236.sh.Yoursummaryfield shouldnowreflectthedatainalloftheTRAPvarbinds.
66
Skills1stLtd
13December2010
Figure37:EventtransformforSNMPTRAPwithsplitandjoin
ThesecondifclausedemonstratesthePythonfindfunctiontosearchforthestring Hello(itreturnstheoffsetinthestringofthesubstringyouspecified(thefirst characterhasposition0)ifthesubstringdoesn'texistthenfindreturns1). evt.summary.split()[3:]splitstheevt.summaryfieldintoalistofstringsandthe[3:]on theendselectseverythingafterthethirdelementinthelist.Theresultisthatthe substringssnmp,trapandtheOIDarechoppedoffthebeginningofevt.summary. Theproblemisthattheresultingdatastructureisstillalistofstringsand evt.summaryisdefinedasastring,notalist.Torectifythis,thePythonjoinfunctionis usedtoconvertoursummarybackintoastring. Checktheendof$ZENHOME/log/zenhub.logand$ZENHOME/log/event.logfor debugginghelp.
67
Skills1stLtd
13December2010
9 Event Commands
Whenaneventoccurs,itispossibletorunacommandontheZenossserver.Thefields oftheeventandtheattributesofthedevicethatgeneratedtheevent,aremade availabletouseintheeventcommand.Eventcommandsareshellscripts(thoughthey cancallotherprogramssuchasPythonscripts).Itispossibletofurthercustomisesuch commandstobuildinadelaybeforeexecutionandtorunthescriptatrepeated intervals. EventCommandshaveaccesstotheeventfieldsanddeviceattributes,usingTALES expressions(TemplateAttributeLanguageExpressionSyntax,fromZope)toreference fieldsoftheevent,evtandattributesofthedevice,dev.SeeAppendixDoftheZenoss AdministrationGuideformoredetails.NotethatyoumustuseTALESthe evt.<eventfield>syntaxusedinmappingrulesandtransformsdoesnotworkinevent commands.TALESsyntaxtakestheform:
${evt/<event field>} ${dev/<device attribute>}
Figure38:Creatingneweventcommands
TypethenewcommandfilenameintheboxatthebottomofthepanelandclickAdd. Clickonthenewnamethatyouhavejustcreatedtodefinethecommand.
68
Skills1stLtd
13December2010
Figure39:DefiningasimpleEventCommandcalledwriteFile
Thedialoguetoconfigureaneventcommandincludes:
Anenabledflag Atimeoutfieldforthecommand Adelayfieldsuchthatadelaycanbebuiltinbetweentheeventoccurringand thecommandrunning.Forglitchscenarios,aneventmaybeclearedrapidlyso anautomationscriptshouldbedelayedpendingthatpossibility. Arepeatfieldthatdenoteshowfrequentlytorepeatthecommand Differentcommandscanberundependingonwhethertheeventisreportinga problemorwhethertheeventisbeingcleared(movedtothehistorytableofthe eventsdatabase). Throughtheuseofoneormorefilters,itispossibletobeveryspecificaboutwhat eventsshouldresultincommandsbeingrun.Filtersshouldbeusedjudiciouslyor performancedegradationislikelytotakeplaceifeventcommandsrunonmost events.
69
Skills1stLtd
13December2010
Figure40:Eventcommanddialoguewithfilteroptionsdropdown
FiltersforanEventCommandarelogicallyANDed.ToimplementalogicalORof criteria,createoneormoreenabledeventcommands.Eachandeverycommandthat satisfiesitsfilteringcriteria,willbeexecuted.Ifmultiplecommandsarevalid,theywill allberun. Eventcommandsareexecutedasynchronouslybythezenactionsdaemonwhich,by default,runseveryminute.Commandswillberunonceforallopeneventsinthe statustableoftheeventsdatabase,providedthecommandhasnotyetbeenrun.The alertstatetableoftheeventsdatabaserecordswhetheracommandhasbeenrunor notforaparticularcommand,foraparticularevent. zenactionsalsorunsanyalertingrules(discussedinthenextsection).It'slogfileis $ZENHOME/log/zenactions.log.
70
Skills1stLtd
13December2010
forbothgoodnewsandbadnewsevents.Initially,abadnewseventisgenerated, followedbyaduplicatebadnewsevent,followedbyadifferentbadnewsevent.
Figure41:zenactions.logshowingactionsforwriteFile
Onexaminationofzenactions.log:
INSERTINTOalert_stateVALUES('0a0000833781ac5cffbfcc4','','writeFile', NULL)ONDUPLICATEKEYUPDATElastSent=now()
71
Skills1stLtd
13December2010
ThereisalsoaSELECTonthehistorydatabase
Ifagoodnewsisthengenerated,itshouldclearallofthebadnewseventsandrunthe scriptfortheClearCommand.
Figure42:zenactions.logshowingclearingeventcommands
zenactions.logshouldshow:
72
73
Skills1stLtd
13December2010
Figure43:SettingupamailserverdestinationfortheZenosssystem
Figure44:Configuringemaildestinationaddressforeachuser
NotethatthereisahandyTestbuttonalongsidetheemailaddress.
74
Skills1stLtd
13December2010
Figure45:DefiningmethodandfiltersforAlertingRules
Notethatiftheemailaddressfieldisleftblank,theaddressconfiguredintheuser's Preferencespagewillbeused. Emailscanbedelayedforaperiodafteraneventhasoccurredandthealertcanbe repeated,ifrequired. ThebottomhalfoftheAlertingRulesEditpaneldefineswhateventsgeneratealerts;it isverysimilartothefiltersseeninEventCommands.Acombinationoffilterscanbe appliedtodeterminewhetheranAlertissent,ornot.AswithEventCommands,filters arelogicallyANDed. TheMessagetabdefinesthecontentoftheemailtobesent.Themessageformatisa Pythonformatstringwithusefuladviceatthebottomofthepanelastowhatfieldsare availableforsubstitution.Thescreenshotbelowisthedefault.
75
Skills1stLtd
13December2010
Figure46:Themessageconfigurationofanemailalert
Figure47:Alertingrulescheduleexample
AlertsaregeneratedbythesamezenactionsdaemonthatrunsEventCommands.
theZenosseventinTRAPvarbinds,toahigherlevelSNMPmanager.Hereisasimple TRAPgenerationexample:
Figure48:gen_alert_trapscripttogenerateSNMPTRAPwitheventparameters
ThescriptisthendrivenbyaZenossEventCommand:
77
Skills1stLtd
13December2010
Figure49:ZenossEventCommandtodriveSNMPTRAPscript
Inpractise,anythingtodowithalerting,includingeventcommands,iscontrolledbythe filtersinthealertingruleortheeventcommand.IfnoProductionStatefilteris configuredinthealertingruleoreventcommand,thenthealert/commandwillrun,by default. AdeviceProductionStatusofProductionwillresultineventscontributingtotheDevice IssuesportletoftheZenossDashboardandinIPmonitoringtakingplace.Alertswillbe generatedandeventcommandswillbeexecutediftheirfilterspermit. AproductionStatusofDecommissionedshouldresultinIPmonitoringceasing;hence, alleventswillceaseandnoalertswillbegenerated.Thedevicewillnotberecordedin theDashboardDeviceIssuesportlet.NotethattheoverallStatusicononadevice's Statuspagewillturngreen!Ifthedeviceisalreadydownwhenthestatusissetto DecommissionedthenyoushouldseeexistingInterfacedownmessagesclearedto HistorybyaclearingeventwhosemessageisNolongertestingdevice.... 78 Skills1stLtd 13December2010
AnyProductionStatusotherthanProductionwillresultinthedevicenotbeing includedontheDashboardDeviceIssuesportlet. TheProductionStateofadevicecanbechangedautomaticallytoallowformaintenance windows.Thisisachievedfromadevice'sstatuspageselectMorefromthetable dropdownmenuandAdministration.Maintenancewindowsaredisplayed;theycanbe modified,addedanddeleted. Theexamplebelowshowsthemaintenancewindowforabackupsystemthatonlyruns forabout3hoursatnight,between23:45and3:00.From3:00,for20hoursand50 minutes,itismovedtoaProductionStatusofOriginal,whichforthismachineis definedasDecommissioned.Thusmonitoring,eventsandalertsareonlygenerated duringthefewhoursthatthisbackupsystemisup.
Figure50:Maintenancescheduleformachinethatonlyrunsfrom23:45until3:00
Notethatmaintenanceschedulescanalsobeappliedtodeviceclasshierarchies;theydo nothavetobeappliedtospecificdevices.
11 Conclusions
ZenosshasanextensiveeventsystemcapableofreceivingeventsfromWindows,syslogs andSNMPTRAPs,inadditiontoreceivingtheeventsgeneratedinternallybyZenoss's owndiscovery,availabilityandperformancemonitoring. AlargenumberofeventclassesaredefinedandconfiguredwhenZenossisinstalled. Thesecanbemodified,removedoraddedto. Aneventfollowsafairlycomplexeventlifecycleprocesswherebyitismappedtoan eventclassandthen,optionally,itistransformedsuchthatdefaultfieldsoftheevent canbechangedanduserdefinedfieldscanbecreated. EventmappingforeventsfromWindows,syslogsorSNMP,dependsontheinitial ZenossparsingdaemondeliveringaneventClassKeyfieldwhichmustcorrespondtoa definedmapping.Subsequently,aPythonRuleand/oraPythonRegexcanbeusedto furtherdistinguishbetweenincomingeventsandmaptodifferenteventclasses. 79 Skills1stLtd 13December2010
AneventclassincludeseventcontextzEventAction,zEventSeverityand zEventClearClasseswhichcanbeappliedtoindividualsubclassesofeventsortoclass hierarchies.Thismeanstransformscanbeaffectedbyeventtype. Devicecontextisalsoappliedtoanincomingevent;devicecontextincludesthe prodState,Location,DeviceClass,DeviceGroupsandSystemsfieldvalues.Device contextprovidestheabilityfortransformstotakeaccountofthedeviceordeviceclass hierarchy. Eventtransformscanbesimpleassignmentofeventfieldsorcanincludecomplex Pythonprograms.AgoodenvironmentfortestingPythonisthezendmdcommandline utility.Transformsand/ortheeventcontextcanbeusedtohelpcleareventsthathave beenresolved.AnyeventwithaseverityofClearedwillautomaticallyclearother similarevents;zEventClearClassescanbeusedtolistextraclassesthatareclearedin addition. EventsaresavedintheMySQLeventsdatabase.Bydefault,eventsgotothestatus table;whentheyarecleared,theyaremovedtothehistorytable. Wheneventsoccur,alertscanbegeneratedusingemailorapagingsystem; alternatively,anyscriptcanberunontheZenosssystem,asaneventcommand. Aswithanyenterprisemanagementsystem,Zenosshasthetoolstoconfigurealmost anyresponsetoanyevent.
80
Skills1stLtd
13December2010
Printalleventclassmappinginstancesforaneventclass,Skills:
print dmd.Events.Skills.instances.objectIds()
Printalleventswiththeireventid:
print dmd.ZenEventManager.getEventList()
Printalleventclasses:
for ec in dmd.Events.getSubOrganizers(): print ec.getOrganizerName()
Printalleventclasses(noteventclassmappings)thathavetransforms:
for ec in dmd.Events.getSubOrganizers(): if ec.transform: print ec.getOrganizerName()
Setupthevariableevttopointtoanexistingevent.Thisisanextremelyuseful testingtechnique!
FromanEventConsole,bringupdetailsforanevent.Copytheeventid.
evt=dmd.ZenEventManager.getEventDetailFromStatusOrHistory(<paste id>) print evt.summary print evt._details
81
Printalistofthedataforanevent,thedetailsforaneventandthefieldsofan event:
print evt.getEventData() print evt.getEventDetails() print evt.getEventFields()
Twodifferentwaystogetattributesfromanevent.Thelatterwillreturnanull stringiftheattributeismissing:
evt.<attribute> ) getattr(evt, <attribute>, '') Note 2 single quotes before
Ifthingsgethorriblymessedup,try:
dmd.Events.reIndex() commit()
82
Skills1stLtd
13December2010
References
1. Zenossnetwork,systemsandapplicationmonitoringhttp://www.zenoss.com/ 2. ZenossAdministrationGuidehttp://www.zenoss.com/community/docs 3. ZenossDeveloper'sGuidehttp://www.zenoss.com/community/docs 4. ZenossFrequentlyAskedquestions(FAQs) http://www.zenoss.com/community/docs/faqs/faqenglish/ 5. ZenossHowToshttp://www.zenoss.com/community/docs/howtos 6. Zenosswikihttp://www.zenoss.com/community/wiki 7. FordocumentationonZenossfunctionality,theZEOobjectdatabaseandZope.try http://www.zenoss.com/community/docs/zenossapidocs/2.1/ 8. ZenossCoreNetworkandSystemMonitoringbyMichaelBadger,publishedby PACKTPublishing,June2008,ISBN9781847194282. 9. SNMPRequestsForComment(RFCs)http://www.ietf.org/rfc.html 10. 11. 12. V1RFCs1155,1157,1212,1213,1215 V2RFCs2578,2579,2580,3416,3417,3418 V3RFCs25782580,341618,3411,3412,3413,3414,3415
13. AsageneralPythonreference,tryLearningPythonbyMarkLutz,publishedby O'Reilly 14. SNMPHostResourcesMIB,RFCs1514and2790http://www.ietf.org/rfc.html 15. ForinformationonTALESexpressions,see http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/AppendixC.stx 16. ForinformationonPythonregularexpressions,see http://www.python.org/doc/2.5.2/lib/resyntax.html and http://docs.python.org/dev/howto/regex.html 17. FortheextensionSNMPMIBfromInformant,goto http://www.wtcs.org/informant/index.htm 18. GoodsitestolookforCiscoMIBsandtheirprerequisitesare:
19. ZenossZenPacksitefortheMIBBrowserZenPack http://www.zenoss.com/community/projects/zenpacks/mibbrowser 20. DatagramSyslogClienthttp://syslogserver.comforsyslogWindowssystems. 21. Raddlenetworkemulationopensourcepackagehttp://raddle.sourceforge.net/ 22. ZenossEventManagementWorkshopavailablefromSkills1stLtd, http://www.skills1st.co.uk/products/courses/
83
Skills1stLtd
13December2010
Abouttheauthor
JaneCurryhasbeenanetworkandsystemsmanagementtechnicalconsultantand trainerfor25years.Duringher11yearsworkingforIBMshefulfilledbothpresales andconsultancyrolesspanningthefullrangeofIBM'sSystemViewproductspriorto 1996andthen,whenIBMboughtTivoli,shespecialisedinthesystemsmanagement productsofDistributedMonitoring&IBMTivoliMonitoring(ITM),thenetwork managementproduct,TivoliNetViewandtheproblemmanagementproductTivoli EnterpriseConsole(TEC).AllarebasedaroundtheTivoliFrameworkarchitecture. Since1997Janehasbeenanindependentbusinesswomanworkingwithmany companies,bothlargeandsmall,commercialandpublicsector,deliveringTivoli consultancyandtraining.Overthelast5yearsherworkhasbeenmoreinvolvedwith OpenSourceofferings.
84
Skills1stLtd
13December2010