Professional Documents
Culture Documents
ABSTRACT
Using interfaces based on open standards such as HTTP/REST, AMQP, XML, and JSON, it’s easy to build
custom integrations with notifications in EMC Smarts Service Assurance Manager in the programming
language of your choice and without proprietary client libraries.
June 2014
Copyright © 2014 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.
All other trademarks used herein are the property of their respective owners.
Part Number H13272
TABLE OF CONTENTS
EXECUTIVE SUMMARY 4
Audience 4
INTRODUCTION 5
Example problem: Trouble Ticket System Integration 5
Environment 5
CONCLUSION 28
Example Code Available 28
EXECUTIVE SUMMARY
This paper examines the steps needed to build a custom integration between SAM and ticketing system, using open standards like AMQP, JSON, and HTTP/REST.
These standards-based interfaces simplify implementation, reduce developer learning time, eliminate the need for proprietary client libraries, and permit integration
using any modern programming language or environment, from almost any modern platform.
AUDIENCE
This white paper is intended for architects, engineers, and project managers planning integration projects with the Smarts® Service Assurance Manager within the
EMC Service Assurance Suite. It is recommended that the reader have a familiarity with RESTful interface programming, and consult the RabbitMQ Getting Started
tutorials provided at http://www.rabbitmq.com/getstarted.html
4
INTRODUCTION
Message-oriented middleware based architectures provide loose coupling between components. Components may publish information without knowing which
component or components consume it, where they are located, which language or platform is used to implement them, etc. Consumers may similarly obtain
information without having to know where or how it is produced.
AMQP, the Advanced Message Queuing Protocol, is an open standard protocol for message-oriented middleware. EMC Smarts Service Assurance Suite (SAM) 9.3
includes support for publishing information about notifications via AMQP, and includes a popular open source AMQP message broker implementation, RabbitMQ, for
routing these messages.
REST, Representational State Transfer, is a way to read and modify data in a system via simple HTTP calls. The EDAA (EMC Data Access API) for Smarts application
provided with SAM offers a full-featured RESTful interface to SAM, and optionally other Smarts domain managers.
In this paper, we’ll examine the use of the AMQP and REST interfaces to integrate with SAM’s notifications system.
ENVIRONMENT
For purposes of this example, we’ll assume you have an existing installation of SAM, with the following options and services configured:
EMC Smarts Service Assurance Manager Server (with notification cache publishing)
5
SUBSCRIBING TO NOTIFICTIONS WITH AMQP
INTRODUCTION
When installing SAM, selecting the “EMC Smarts Service Assurance Manager Server (notification cache publishing)” service option causes SAM to be started with a
bootstrap file than enables the AMQP Notification Publisher component.
Whenever a notification is created, updated, or archived, the notification publisher gathers information about it, renders it into a JSON-formatted message, and
publishes it to the notification exchange in RabbitMQ. Messages are sent with a “routing key” that helps RabbitMQ identify consumers for the message. Based on the
routing key, RabbitMQ routes and files messages into one or more queues (or perhaps no queues, if there are no subscribers for that particular routing key). Finally,
consumers connect to RabbitMQ and listen on one or more queues to obtain messages of interest.
RABBITMQ LOCATION
For security, reliability, and other technical reasons, SAM currently assumes that RabbitMQ will be installed alongside SAM on the same host, and running on the
standard RabbitMQ port (5672). To route messages into a RabbitMQ installation on a different host, please consult the RabbitMQ documentation on options such as
federation or the “shovel” plugin. Many different options exist for creating a distributed message fabric with RabbitMQ, and a full treatment is beyond the scope of
this document. For purposes of this paper, we’ll assume a simple deployment using only the RabbitMQ instance that was installed with SAM.
AMQP support the notion of multiple “virtual hosts” within an AMQP broker. All SAM and Notification Exchange functionality uses a virtual host named
“emc.smarts.notifications”.
As included with SAM, RabbitMQ also contains a second exchange called “emc.smarts.notifications.ingestion”. This exchange is bound to the emc.smarts.notifications
exchange with a pattern of “#”, meaning, “match all”. This second exchange and associated queues are part of the ingestion system that indexes notifications into the
EMC Smarts Notification Cache. To ensure proper operation of the cache and ingestion processes, do not modify this exchange or associated queues. Also, do not
modify the “publisher” or “ingestion” users, other than to change their passwords, as recommended in the product documentation.
In rare cases, a customer may wish to use RabbitMQ for integration, but not use the Notification Cache. In this case only, the binding between the two exchanges
should be removed to prevent a large backlog of messages in the ingestion queues, which will not be serviced by the ingestion process in such a configuration.
Otherwise, use only new bindings to the emc.smarts.notifications exchange for your integrations, and create new, purpose-specific logins in RabbitMQ for such uses,
rather than modifying the existing users or using the admin login for such purposes.
ROUTING KEYS
Each message SAM publishes to the exchange will have a routing key following the pattern
emc.smarts.notification.domain.ClassName.EventName.InstanceName
Where domain is the name of the SAM domain (e.g. INCHARGE-SA-PRES by default), ClassName and EventName are from the event that triggered the notification in
SAM, and InstanceName is the name of the object instance that trigged the notification, or some other value computed from that instance name.
This routing key may be used with pattern matches in a binding to subscribe to a subset of notifications. In AMQP, pattern match strings may contain “*” to match
any single component of a dotted routing key (that is, asterisk matches anything except a dot), or “#” to match any characters including dots (thus, potentially
matching multiple dotted routing key components).
In many cases only a single presentation SAM will be publishing, and thus the domain name will often not be of interest and can simply be specified as “*”. Further,
because routing key strings have a small maximum length in AMQP, and because object instance names in EMC Smarts may be quite lengthy, the InstanceName in
the routing key may be replaced in whole or in part with a hash code. Thus, we do not recommend trying to match on specific subsets of instances based on
InstanceName. This leaves ClassName and EventName as being of the most useful items for pattern matching.
For example, to match only notifications about Routers, one might bind with the pattern:
6
emc.smarts.notifications.*.Router.#
Many applications will be interested in filtering based on other factors, and in this case it will be necessary to do filtering within the consuming application based on
content. For example, Severity is not present in the routing key, so it would be necessary in that case to make sure the binding pattern should be broad enough to
include all messages that are possibly of interest, and then have the consuming code check the severity of each incoming message from its queue and discard it
before further processing if the message is not of interest. In many cases, applications will want to receive all notification updates. In this case, a binding matching all
routing keys can be used:
#
7
CONFIGURATION WALKTHROUGH
We’ll now go through in detail the steps needed to configure RabbitMQ as described above.
To access RabbitMQ’s configuration, use the web-based management interface. This is located on port 15672. The default password for the “admin” login is
“changeme”. EMC strongly recommends changing this default password; see the product documentation for details.
8
After logging in, the Overview tab shows information and statistics about RabbitMQ.
9
Clicking on the Exchanges tab shows the system default and EMC preconfigured exchanges.
10
We want to create a new queue, so click on the Queues tab.
11
Click on the “Add a new queue” section to expand it. Choose a name for your new queue. Here we’ve used “my_queue”. Consult the RabbitMQ documentation for a
full explanation of all the fields. For most integration uses, the defaults shown here, a durable queue without any automatic expiry or time-to-live, is reasonable.
Click the “Add queue” button to create the queue.
12
The new queue is created and is shown on the list of queues.
13
Now we need to bind the queue to the notification exchange. Return to the Exchanges tab and click on emc.smarts.notifications in the list of exchanges to bring up
the detail view for that exchange. Click on the heading labeled Bindings to expand it.
14
You’ll see the existing binding to the emc.smarts.notifications.ingestion queue.
In the “Add binding from this exchange” section, enter the name of your new queue and the binding key. In this example we’ve used “#” to get all notification
messages.
For this simple configuration, no additional arguments for the binding are needed and that section can be left blank.
15
The display is updated to show the new binding.
16
At this point, copies of any notification messages should be flowing into the new queue, but we need a way to retrieve them. EMC strongly recommends creating a
new RabbitMQ user login for this purpose. First, click on the Admin tab. If the Users view isn’t already showing, select it in the right hand column.
17
Click on the “Add a user” heading to expand it.
Enter the login name for your application. We’ve chosen to call it “my_app” here.
Enter the password in both fields. You should choose a strong password for security reasons, but for example purposes we’ve used the password “123456”.
Click the “Add User” button.
18
The display will update to show the new user, but indicating that it has no access. Click on the new user to grant it access.
19
Make sure the Virtual Host “emc.smarts.notifications” is selected.
We don’t want this user to be able to configure or write to the system, so set a regular expression of “^$” for the Configure and Write regexps.
Following the “principle of least privilege”, we’ll grant this new user access only to the one queue it needs to be able to read. Enter “^my_queue$” as the Read
regexp.
20
The display updates showing the new permissions applied.
21
MESSAGE FORMAT
Now that we have a queue of notification update messages, what’s in them?
Messages are sent in a JSON format. JSON (JavaScript Object Notation) is a lightweight data-interchange format that is simpler and more compact than XML. It is
widely supported, with implementations for a large number of languages and platforms. See http://json.org for the specification and links to implementations.
By default SAM sends the messages in the most compact representation, without any extraneous whitespace. This is the most efficient method, and is recommended
for production, but it can be difficult for humans to read. During development, you may wish to enable the option to “pretty-print” messages, which makes them
more readable. This can be accomplished using the following command in dmctl:
put SM_AMQPNotificationPublisher::SM-AMQPNotificationPublisher::pretty true
this setting is temporary for the duration of the SAM server process lifetime or until changed. To make the setting automatic at startup, edit the publisher
configuration file with sm_edit:
sm_edit conf/ics/amqp-notification-pub.conf
This fie is in the standard EMC Smarts “import” format file, so the setting of pretty can be added within the curly braces for the SM-AMQPNotificationPublisher
object. The relevant part of the resulting file might look like:
SM_AMQPNotificationPublisher::SM-AMQPNotificationPublisher {
pretty = TRUE
}
MESSAGE SCHEMA
The mapping of notifications to JSON follows a subset of an EMC standard called the Alert Information Model (AIM), with extensions.
The JSON document for each message contains fields with the AIM-standard name, for notification fields that correspond with those standard fields. All other
Notification fields are placed within a subdocument called Extensibility. We will use the convention of dotted pathnames to indicate data in nested subdocuments.
For example, when we say the field “Extensibility.TroubleTicketID”, we mean the field named “TroubleTicketID” within the ‘Extensibility” subdocument of the main
JSON document.
The fields of the outer document, mapped from fields of the ICS_Notification object in SAM, are as follows:
22
EXTENSIBILITY ICS_NOTIFICATION NOTES
SUBDOCUMENT FIELD
EventName EventDisplayName
EventKind EventType MOMENTARY or DURABLE
LastNotifiedAt LastNotifiedAt ISO 8601, UTC
LastClearedAt LastClearedAt ISO 8601,UTC
InternalEventHandle Name
ClassName ClassName
ClassDisplayName ClassDisplayName
InstanceName InstanceName
InstanceDisplayName InstanceDisplayName
ElementClassName ElementClassName
ElementName ElementName
OccurrenceCount OccurrenceCount
IsProblem IsProblem
IsRoot IsRoot
ClearOnAcknowledge ClearOnAcknowledge
Impact Impact
Certainty Certainty
InMaintenance InMaintenance
TroubleTicketID TroubleTicketID
Owner Owner
EventState EventState
SourceDomainName SourceDomainName
SourceEventType SourceEventType
UserDefined1 UserDefined1
… … …
UserDefined20 UserDefined20
Notification Name SMARTS EDAA URL to this ICS_Notification object in SAM
Note that in the case of a message indicating an ARCHIVED state, the corresponding ICS_Notification
has been deleted, so this URL will no longer be valid.
NotificationLists A list of strings. Contains the “listName” field of each ICS_NotificationList that this notification belongs
to
MESSAGE PROCESSING
Now that we have a queue in RabbitMQ that will receive copies of the messages, a user login with which to retrieve them, and knowledge of the message contents,
we can design our message processing logic.
After connecting to RabbitMQ and listening on the queue, we use a JSON library to parse each message in to data values.
First, we can make sure the message meets our criteria for interest. Messages that are not of interest, perhaps because the Severity is too low, could be discarded.
Also, we should ensure we process messages only of the desired state. In this example, we want to assign trouble ticket IDs to each new notification. So, only
messages where State is “NEW” and where Extensibility.TroubleTicketID is blank should be processed (later, when we implement the code to update a User Defined
Field in SAM with the TroubleTicketID field in SAM, we will get additional messages corresponding to those updates. To avoid an infinite loop of message updates, we
must ensure we don’t update notifications based on update messages from our own previous updates. In this case, a non-blank TroubleTicketID will mean that we’ve
already processed the notification and shouldn’t process it again.)
The integration could be extended to handle other kinds of updates as well, perhaps updating the trouble ticket based on other state transitions, like to OPEN,
CLOSED, or ARCHIVED, but in our example we’ll only handle new notifications.
23
UPDATING NOTIFICATIONS WITH THE SMARTS EDAA
INTRODUCTION
This section documents using the Smarts EDAA to write changes to a notification object in the EMC Service Assurance Manager. The use case described below
provide an example to update a User Defined Field in a notification for the purpose of writing a TroubleTicketID from a 3rd party ticketing system, however, the
process applies to writing any changes to a notification via the Smarts EDAA.
When we create trouble tickets based on new notifications, the trouble ticket system will assign some identifier to the new ticket. The next step in developing our
custom integration is to update the ICS_Notification object in SAM so that the TroubleTicketID field will contain this new identifier.
Using the approach below, we could also make other updates to the notification. Perhaps the trouble ticket system assigns a unique URL for each ticket as well. We
could update one of the User Defined fields with this URL, or with other information from the ticketing system. However, for purposes of our example, we’ll focus on
the TroubleTicketID field alone.
The AMQP publisher in SAM is an outbound-only interface, intended as an efficient way for other programs to track notifications. We can’t modify notifications by
publishing messages into RabbitMQ. To update the TroubleTicketID field, we need to use the REST interface. EMC has a standard that describes how RESTful
interfaces should work, called the EMC Data Access API (EDAA) standard. The application that implements an EDAA compliant interface to EMC Smarts domain
managers is called the Smarts EDAA.
Conveniently, the notification messages published by SAM include relevant Smarts EDAA URLs. In particular the field Extensibility.Notification is the Smarts EDAA URL
for the corresponding ICS_Notification object in SAM.
Content-Type: application/json
Accept: application/json
The first tells the EDAA that we will be sending JSON data. The second tells the EDAA that we want the results back in JSON as well. (it’s actually possible to mix XML
input and JSON output, or vice versa, if that was desirable for some reason).
For a PUT with JSON, the body of the request should contain a nested JSON document. The outer object should contain as a key the class name of the kind of object
we’re sending – in this case, ICS_Notification. The value of that key should be a subobject, containing the key-value pairs to update. Putting it all together, and
assuming the trouble ticket ID that we wish to assign is “ISSUE-12345”, the resulting request should look something like this:
{
"ICS_Notification" : {
"TroubleTicketID" : "ISSUE-12345"
}
}
24
INVOKING OPERATIONS
A simple PUT request is sufficient to update attributes of most classes in a Smarts domain. However, the design of ICS_Notification requires that an operation called
“changed” be invoked for any updates to take effect, and for any subscribers to the notification to see the update.
The “changed” operation takes a single argument, “waitForNotify”. Currently, the value of this argument should always be zero.
Fortunately, the Smarts EDAA also offers a way to invoke operations. We do this using a POST to the URL “action/changed”, relative to the instance’s URL, and
providing a request body that contains the arguments. Again we will choose JSON input and output, so our second request might look like this:
POST /smarts-edaa/msa/INCHARGE-SA-PRES/instances/ICS_Notification::NOTIFICATION-
Router_XYZ_Down/action/changed HTTP/1.1
Accept: application/json
Content-Type: application/json
{
"waitForNotify" : 0
}
(Note: The POST with URL and HTTP/1.1 indication is all on one line, although it has been wrapped above for space constraints)
AUTHENTICATION
This section explains handling the authentication aspect with writing updates via the Smarts EDAA. The Smarts EDAA is almost always deployed in a configuration
that protects it with CAS (Common Authentication Service) authentication provided by EMC M&R (Watch4Net).
For security reasons, EMC strongly recommends creating a login in EMC M&R for each EDAA-using application, rather than sharing accounts or using the admin
account for such purposes. Consult the SAM and EMC M&R configuration guide for details on creating MONITORING AND REPORTING user accounts with SAM access.
CAS is a ticket-based “single sign on” system. To authenticate to a CAS-protected application, a user logs into the CAS service by presenting credentials (username and
password). Once authenticated to CAS, the user can obtain tickets to access particular applications. The ticket is presented to the application, and if accepted, the
application establishes a session, and from that point access is controlled by the session cookie, until it expires.
The normal CAS login flow usually involves several redirects. See the CAS documentation (http://www.jasig.org/cas) for details. For REST clients, these redirects may
not be convenient. For this reason, EMC’s CAS implementation in the EDAA supports some extension headers that cause errors to be returned rather than redirects.
Often, these are easier to hook in HTTP client libraries.
Implementation details will vary greatly between different HTTP client libraries, but in general to implement a CAS-aware client, follow these rules:
Ensure that the client is configured to implement cookies
With each request, send the header “X-EMC-MAC-USE-401: TRUE” and “X-EMC-MAC-NO-TICKET-REDIRECT: TRUE”. With this, if not authenticated, the EDAA will
return a HTTP 401 Unauthorized response, rather than a redirect to the CAS login screen.
When a 401 Unauthorized is received, log in to CAS, and repeat the request. The Location header in the 401 response will contain the URL of the CAS service.
25
Repeat the original request, adding the ST as the query parameter “ticket”.
Login should not be needed on subsequent requests until the session cookie expires, at which point a 401 Unauthorized will be returned again.
Shell Example
The following example shell script demonstrates the sequence of calls using the “curl” utility:
#!/bin/bash
EDAA_URL=http://itops-dev-120.lss.emc.com:8080/smarts-edaa/msa/
# X-EMC-MAC-USE-401:TRUE.
# This will result in a 401 response with the Location header at the
# CAS location
# STEP 2: Retrieve the TGT URL from CAS by posting a username and password
wk '{print $2}')
# STEP 4: Retry original URL and pass in ST. Use cookies to preserve
# the session for future calls. Use X-EMC-MAC-NO-TICKET-REDIRECT:TRUE
# for REST calls to avoid extra unneeded 302 after ST validation.
26
# STEP 5: Make more calls using same cookies
#
curl -H "X-EMC-MAC-USE-401:TRUE" -b /tmp/cookies.txt.$$ -c /tmp/cookies.txt.$$ ${EDAA_URL}
27
CONCLUSION
Using the techniques described, developers can easily integrate with EMC Smarts Service Assurance Manager to listen for notifications, act on them, and make
updates in the SAM system. The techniques are language and platform independent, and applicable to a variety of integration scenarios.
The example code is in filename “notif_demo.py” and is available on ECN (EMC Community Network) located in the Smarts Community > Smarts Support Forum in
the Content tab as a downloadable file from the following location: https://community.emc.com/docs/DOC-36437
28