You are on page 1of 36

November 2013

Oracle Enterprise Manager System Monitoring


Plug-in for Oracle ZFS Storage Appliance
Installation Guide
v1.4.1

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance, Installation Guide v1.4.1

Table of Contents
Introduction.............................................................................................................................. 1
About the Oracle ZFS Storage Appliance Plug-in................................................................. 2
Software Requirements........................................................................................................... 2
Prerequisites for Monitoring Oracle ZFS Storage Appliances.............................................. 3
Configuring the Oracle ZFS Storage Appliance.................................................................... 3
Configuring for Clustered Appliances.................................................................................... 4
Configuring Database-to-Appliance Mappings ..................................................................... 4
Plug-in Management in the Oracle Enterprise Manager Cloud Control 12c Environment..5
Deploying the Plug-in............................................................................................................. 5
Adding Instances for Monitoring............................................................................................ 7
Adding Clustered Systems for Monitoring............................................................................. 8
Viewing Metrics and Reports................................................................................................. 8
Upgrading the Plug-in........................................................................................................... 10
Un-deploying the Plug-in...................................................................................................... 10
Metric and Collection Information....................................................................................... 11
Metric Collections on Datasets............................................................................................. 17
Metric Collections on Pools, Projects and Shares............................................................... 18
Reports................................................................................................................................... 18
Default Threshold Values..................................................................................................... 20
Stateless Alerts...................................................................................................................... 24
Oracle ZFS Storage Appliances Alerts................................................................................. 25
Communicating Performance Issues to the Appliance Administrator.................................25
Troubleshooting Monitoring Deployments........................................................................... 26
Metric Collection Errors....................................................................................................... 29
Oracle ZFS Storage Appliance Audit Logs......................................................................... 31
Host Agent Logs .................................................................................................................. 32
Additional Resources............................................................................................................ 33

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide v1.4.1

Introduction
The Oracle ZFS Storage Appliance Plug-in extends Oracle Enterprise Manager Cloud Control
to add support for monitoring Oracle ZFS Storage Appliances. This guide provides an overview
of the plug-in and how to install and deploy it for monitoring Oracle ZFS Storage Appliances.
This guide is written for the Oracle Enterprise Manager Cloud Control administrator. As Cloud
Control administrator, you will communicate closely with the Oracle ZFS Storage Appliance
storage administrator to discuss performance and analysis of database deployments on the
Oracle ZFS Storage Appliance.
If you need additional information about the operation of plug-ins for the Oracle Enterprise
Manager Cloud Control product, see the Oracle Enterprise Manager Concepts Guide. You can
download the document, as well as other related documentation, from the Oracle Web site
(see the section at the end of this document).
Important: Oracle Enterprise Manager Cloud Control was previously called Oracle Enterprise
Manager Grid Control (versions 10g and 11g). These older versions are no longer being
supported, but users familiar with them should note that references to grid control are now
replaced with cloud control. These legacy versions' documentation still reflects the old
terminology.
Also note that the Sun ZFS Storage Appliance is now called the Oracle ZFS Storage
Appliance, and similarly to the Oracle Enterprise Manager software, legacy Appliance Kit
software versions and their documentation may still reflect the former name. References to
Sun ZFS Storage Appliance, Sun ZFS Storage 7000, and Sun ZFS Backup Appliance all refer
to the same family of Oracle ZFS Storage Appliances.

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

About the Oracle ZFS Storage Appliance Plug-in


The Oracle ZFS Storage Appliance plug-in for Oracle Enterprise Manager Cloud Control
provides first-class monitoring for all Oracle ZFS Storage Appliance models. The plug-in
provides the following primary features:

Gathers and presents storage system, configuration, and performance information


for Oracle ZFS Storage Appliances

Raises alerts for pre-selected configuration and monitoring data

Provides out-of-the-box reports that complement Oracle ZFS Storage Analytics


reports

Ties together Oracle databases deployed on NFS shares from Oracle ZFS
Storage Appliances

Supports monitoring by remote agents in the Cloud Control environment

When you configure the Oracle ZFS Storage Appliance to be monitored by Oracle
Enterprise Manager Cloud Control, an analytics worksheet and datasets are enabled. The
worksheet and datasets help bridge a Cloud Control administrator's view of the appliance
to the storage administrator's more detailed view. This enables both administrators to be
fully conversant in discussing performance or analysis of database deployments on the
appliance.

Software Requirements
Deployment of the Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS
Storage Appliance requires that the following software versions already be installed:

Oracle Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3)

Oracle ZFS Storage Appliance 2010 Q3.4.4 software or later

Note: Additional software requirements, such as the correct Java version in Cloud Control,
are met through the required applications listed, provided the correct versions are
installed. The plug-in can be installed on any operating system in which Enterprise
Manager Cloud Control is running.

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

Prerequisites for Monitoring Oracle ZFS Storage


Appliances
Before you can deploy and monitor Oracle ZFS Storage Appliances, you must:

Install the Oracle Enterprise Manager Cloud Control environment. Be sure to pay
attention to security sections in the Oracle Enterprise Manager Administrator's
Guide.
Configure the Oracle ZFS Storage Appliances for monitoring using the workflow
included with Oracle ZFS Storage Appliances, as identified in the next section.

Configuring the Oracle ZFS Storage Appliance


To set up each appliance for monitoring from Cloud Control, enter the Oracle ZFS Storage
Appliance Browser User Interface (BUI) as root or another user with the proper role to
create worksheets, enable datasets, and create users and user roles on the appliance.
1. Log in to the BUI and go to the Maintenance/Workflows tab.
2. Run the Configure for Oracle Enterprise Manager Monitoring workflow, entering
and recording the password for the oracle_agent user. You can rerun the
workflow later to reset the worksheet, re-enable data sets, or recreate the user.
The following occurs when the workflow is run:
A restricted role user named oracle_user is created for use by
the monitoring agent. The restricted access role associated with the
oracle_agent user is also named oracle_agent.

An analytics worksheet is automatically created to ensure smooth


communication between the Cloud Control administrator and the storage
administrator tasked with administering the target Oracle ZFS Storage
Appliance.

3. Communicate the oracle_agent password to the Cloud Control Administrator


along with the following information:
IP address for the network interface that you want Cloud Control to
use to access management information (this interface must allow
administration).

Full DNS name that is used to locate the system and corresponds
to the IP address.

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

Port that is used by the BUI (typically 215 unless your company
has a requirement to change management ports).

Important: If you are going to deploy/upgrade to v1.4.x of the plugin, you must carry out
the following additional steps:
1. Log in to the BUI and go to the Configuration/Users tab.
2. Click to edit the Oracle Agent User name field.
3. Uncheck Kiosk user if it is checked.

Configuring for Clustered Appliances


If the Oracle Enterprise Manager Cloud Control will monitor resources available on a
clustered Oracle ZFS Storage Appliance, you must configure both cluster heads. The
oracle_agent user and worksheet will automatically be created on the peer head, but
datasets will not become active on the peer. If the head containing the monitored
resources fails, Oracle Enterprise Manager Cloud Control monitors the resources on the
head that takes over the exported resources from the failed head. If the configuration step
was not run on this head, the datasets might not be active, resulting in Metric Collection
errors with the message workflow parameters are invalid. (See Table 5: Common Metric
Collection Errors .)
Note: When running the Configure for Oracle Enterprise Manager Monitoring workflow on
the second head, the oracle_agent user does not have to be "Recreated." If you choose
the option to Recreate the user, be sure to use the same password as the peer head set
(the oracle_agent password will always be the same on both heads of the appliance).

Configuring Database-to-Appliance Mappings


In order for reports that include database-to-appliance mappings to work, the database
and the host on which the database resides must be monitored by Oracle Enterprise
Manager Cloud Control. For instructions on monitoring host storage and database
instances, refer to the documentation provided with Oracle Enterprise Manager Cloud
Control software (see the References section at the end of this guide ).

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

Plug-in Management in the Oracle Enterprise Manager


Cloud Control 12c Environment
This section describes how to deploy and use the Oracle Enterprise Manager System
Monitoring Plug-in in the Oracle Enterprise Manager Cloud Control 12c environment. If
you need to upgrade your Oracle Enterprise Manager Cloud Control 11g environment to
Cloud Control 12c, please follow the instructions at:
http://www.oracle.com/technetwork/oem/grid-control/downloads/oem-upgrade-console502238.html
The Oracle Enterprise Manager Cloud Control Upgrade Guide is located at:
http://download.oracle.com/docs/cd/E24628_01/upgrade.121/e22625.pdf
For further detailed instructions on the Oracle Enterprise Manager's Plug-in Manager and
the plug-in deployment process, see section 18 "Plug-in Manager" of the Oracle
Enterprise Manager Cloud Control 12c Administrator's Guide at:
http://docs.oracle.com/cd/E24628_01/doc.121/e24473/plugin_mngr.htm

Deploying the Plug-in


Deploying the Oracle Enterprise Manager System Monitoring Plug-in in the Oracle
Enterprise Manager Cloud Control 12c environment is a two-step process. First, the plugin is imported into the Oracle Enterprise Manager Cloud Control 12c environment using
the emcli command. Then the Oracle Enterprise Manager console is used to deploy the
plug-in by installing it on the Oracle Enterprise Manager's Management Service and
Management Agent, and adding the Oracle ZFS Storage Appliance as a target. The
emcli command can also be used to monitor the installation of the plug-in. These steps
are described in more detail in the following instructions.
Note: The Plug-in must be deployed on Oracle Enterprise Manager's Management
Service before being deployed on Management Agents. The Oracle ZFS Storage
Appliances can only be added as targets for monitoring once the Plug-in is deployed on
both the Management Service and the Management Agent.
To import the plug-in into the Oracle Enterprise Manager Cloud Control 12c environment,
complete these steps:

1. Set up the software library path (if you have not already done this).

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

(a) From a terminal window on the server hosting Oracle Enterprise Manager 12c ,
create a new directory for the software library; for example:
/home/oracle/swlib1.
(b) In the Oracle Enterprise Manager console, go to Enterprise>Provisioning and
Patching> Software Library>Actions>Administration and click Add Button.
(c) In the popup window, enter

and /home/oracle/swlib1.

swlib1

(d) Wait a minute or so. Then, on the Oracle Enterprise Manager host, verify that
the following directory structure was created under /home/oracle/swlib1:
[oracle@aie-4200i swlib1]$ ls
0

11

14

17

22

25
12

28
15

30
18

33

66

69

71

9 1

53

56

59

61

64

67

72 10

20

43

46

49

51

54

57

62

36
23

39

65

13

26

41

68

29

16
70

44
31

19

21

47
34
24

5
37
27

52
4

55
42

32

58
45
35

60
48
38

63
50
40

2. Install the emcli kit.


(a) Download emclikit.jar by following instructions available on the Enterprise
Manager Console: Setup>Command Line Interace.
(b) Copy the downloaded kit to a new folder; for example: /home/oracle/emcli.
(c) Install the kit and set it up to work with your EM by entering:
java -jar emclikit.jar client -install_dir=/home/oracle/emcli
./emcli setup -dir='/home/oracle/emcli' -url='https://<machinename>:port/em' -user=sysman

(d) Download the plug-in file ( *.opar file) to a directory; for example:
/home/oracle/12g_plugins
(e) Upload the plug-in using the emcli command as shown in the following
example:
./emcli import_update -file=
"/home/oracle/12g_plugins/12.1.0.1.0_oracle.sun.oss7_2000_0.opar" -omslocal

Note: On a Windows system, the emcli command does not require any quotes.
To deploy the plug-in on the Oracle Enterprise Manager using the Oracle Enterprise
Manager console, complete these steps:

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

1. Go to Setup > Extensibility > Self Update. Click on Plugin. Make sure Oracle ZFS
Storage Appliances is listed as Downloaded. Select it and click on Actions > Readme
for the details of the plug-in.
2. Click on the Plug-in tab or Setup > Extensibility > Plugin. Expand the Servers,
Storage and Network category.
3. Select the plug-in Oracle ZFS Storage Appliances from the table displayed in step
2 and select: Actions > Deploy On > Management Servers.
4. In the popup window, enter the Repository Password and click Continue.
5. Wait until the prerequisite checks are done. Click Next and then Deploy.
An emcli command is displayed that can be run to monitor the status of the deployment
and let you know when it has completed successfully. For example:

emcli get_plugin_deployment_status -plugin_id=oracle.sun.oss7

To install the Oracle Enterprise Manager System Monitoring Plug-in on a local


Management Agent, complete the following steps:
1. Select Setup > Extensibility > Plugins.
2. Click and highlight the plug-in that you installed on the Management Server.
3. Select Deploy On > Management Agent.
4. Click Add.
5. In the Target Type drop-down list, select Agent and highlight the local agent
displayed in the table, then click Select.
6. Click Continue.
7. Wait until the prerequisite checks are done. Click Next and then Deploy.
An emcli command is displayed that can be run to monitor the status of the
deployment and let you know when it has completed successfully. For example:
emcli get_plugin_deployment_status -plugin_id=oracle.sun.oss7

Note: It will take about 3 minutes to complete this process.

Adding Instances for Monitoring


To install the Oracle Enterprise Manager System Monitoring Plug-in on a local
Management Agent, complete the following steps:
1. Select Setup -> Add Target -> Add Targets Manually.

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

2. Select Add Targets Declaratively by Specifying Target Monitoring Properties .


3. In the Target Type drop-down list, select Oracle ZFS Storage Appliance.
4. For Monitoring Agent, click Search. In the window that is displayed, for Target Type
drop-down list, select Agent and from the table below, click and highlight the agent you
want to use for monitoring your target. Click Select.
5. Click Add Manually.
6. Add the Target details and click OK. Wait until you see the confirmation and click
OK.
7. To find the target, click on Targets->All Targets.
8. Select the Oracle ZFS Storage Appliance added above to go to the appliance home
page.

Adding Clustered Systems for Monitoring


Clustered systems require consideration by the Oracle Enterprise Manager Cloud Control
administrator. Unless Cloud Control is being used as a general purpose Configuration
Management Database (CMDB) or general purpose asset and resources tracking, only
the head or heads containing active exported resources in use by other monitored
software resources should be monitored by Oracle Enterprise Manager Cloud Control
(even though the configuration workflow must be run on both cluster heads).
If a storage head fails and an alternate head takes over the resources, the new head
owning the logical resources will become monitored (if the takeover head was properly
configured). Within the metric reports, the status of the cluster is given. The cluster status
is important, as shared resources in an active/active environment will become even more
pressured and the actual event of a cluster failover can cause disruption in quality of
service objectives.

Viewing Metrics and Reports


The two primary ways to gather information about monitored instances of an Oracle ZFS
Storage Appliance are viewing metrics and accessing reports. These are each described
in the following sections.
IMPORTANT: By default, all metrics that collect information from datasets on Oracle ZFS
Storage Appliances are disabled upon installation. Likewise, certain metrics for pools,
projects, and shares are disabled too. Further details on types of metrics as well as their

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

settings and how to manage them are included in the section Metric and Collection
Information later in this document.

Viewing Metrics

As a rule, more point in time information is available in raw metric information than in
reports. Reports are focused on information that database administrators can use to
analyze database performance and storage capacity.
To view the raw information collected from an Oracle ZFS Storage Appliance:
1. Log in to Enterprise Manager Cloud Control.
2. Go to Targets > All Targets and select the Oracle ZFS Storage Appliance as the
target.
3. From the target's home page, select Oracle ZFS Storage Appliance > Monitoring >
All Metrics.
4. View the categories and information collected from the last collection interval.
The raw metric information that you have access to can be found in Table 1: Metric and
Collection Information.
Accessing Reports

To view reports for an Oracle ZFS Storage Appliance:


1. Log in to Enterprise Manager Cloud Control.
2. Go to Targets > All Targets and select the Oracle ZFS Storage Appliance as the
target.
3. From the target's home page, select Target > Information Publisher Reports.
The information to which you have access is the same reports as those in the Enterprise
Manager 10g/11g environment. Please refer to Table 2: Reports
Reports accessed using a target instance differ from reports accessed from the Reports
tab within Oracle Enterprise Manager Cloud Control. All instance reports are available
under the tabs Reports >Storage>Network Attached Storage. When accessed from the
Reports tab, reports include configurable time spans for graph display where applicable.
Note: Administrators can clone and customize all system reports from the Reports tab.

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

Upgrading the Plug-in


Ensure the following steps are performed on the Oracle ZFS Storage Appliance.
1.

Log in to the BUI and go to the Configuration/Users tab.

2.

Click to edit the Oracle Agent user field.

3.

Uncheck Kiosk user if it is checked.

Note: If you are upgrading the Plug-in from v1.x to v1.4.x, you must ensure that you are
on Oracle Enterprise Manager 12.1.0.3.0 and the following steps are performed.
After importing v1.4.x of the Plug-in and deploying it on an Oracle Enterprise Manager
Management Service and Management Agent, you must restart Oracle Enterprise
Manager's Management Service. The steps to do so are:
1.
Log in to the host where Oracle Enterprise Manager's
Management Service is residing.
2.
Navigate to where Oracle Enterprise Manager's Management
Service is installed (typically in a folder called Middleware).
3.
Navigate to Middleware/oms/bin and run the following two
commands:
emctl stop oms
emctl start oms

Un-deploying the Plug-in


Oracle Enterprise Manager System Monitoring Plug-in can be un-deployed from the agent
and Oracle Enterprise Manager. By un-deploying the plug-in from the agent, all the
associated targets on the Oracle ZFS Storage Appliance that are monitored by the agent
are automatically removed.
To un-deploy the plug-in from the agent and Oracle Enterprise Manager:
1. Go to Setup > Extensibility > Plug-ins.
2. Under the Servers, Storage and Network category, select Oracle ZFS Storage
Appliances.
3. Go to Undeploy From > Management Agent > Add.
4. Select the Agent from which the plug-in is to be un-deployed and click Continue.

10

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

An emcli command that you can run from the command prompt to monitor the status of
the undeployment activity is displayed.
5. To remove the plug-in from the management servers, repeat preceding steps 1-4
while selecting Management Servers in step 3.
To permanently delete the plug-in:
1. Go to Setup > Extensibility > Self Update > Plug-ins.
2. Under the Servers, Storage and Network category, select Oracle ZFS Storage
Appliances.
3. Select Actions > Remove. This will remove the plug-in permanently from the
system.

Metric and Collection Information


Oracle Enterprise Manager Cloud Control displays a direct mapping of information to
datasets and information collected in the target Oracle ZFS Storage Appliances.
Note: Information collected by the System Attributes from the Workflow dataset indicates
items that cannot be enabled/disabled by an administrator. This information is collected
through scripts on each Oracle ZFS Storage Appliance.
TABLE 1: METRIC AND COLLECTION INFORMATION

METRIC NAME

COLUMN (*=KEY FIELD,

ORACLE ZFS STORAGE

POLLING

UPLOAD FREQUENCY

! = CONDITIONS ENABLED)

APPLIANCE DATASET

INTERVAL

TO EM REPOSITORY

(MINUTES)

(X=DISABLED BY DEFAULT)

Response

Status

N/A (request for host name)

Every polling cycle.

Monitoring Prerequisite

Status!

N/A

30

Upload when there is a


severity change on the

Message

thresholds.
X

Appliance Logical Status

Share Identifier*

System Attribute from


Workflow

Available (GB)
Total Allocated (GB)

20

Every 72nd polling cycle


(or once per day) or
whenever there is a
severity change on the
thresholds.

Data (GB)!

11

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC NAME

COLUMN (*=KEY FIELD,

ORACLE ZFS STORAGE

POLLING

UPLOAD FREQUENCY

! = CONDITIONS ENABLED)

APPLIANCE DATASET

INTERVAL

TO EM REPOSITORY

(MINUTES)

(X=DISABLED BY DEFAULT)

Share Percentage Used (%)!


Quota (GB)
Reserved (GB)!
Unused Reserved (GB)!
SCSI LUN
X

Appliance Project Status

Project Identifier*

System Attribute from

20

Workflow

Every 72nd polling cycle


(or once per day) or

Available (GB)

whenever there is a
severity change on the

Total Allocated (GB)

thresholds.

Project Data (GB)!


Project Percentage Used!
Quota (GB)
Reserved (GB)!
Unused Reserved (GB)!
Total Share Quota (GB)!
Project Over Allocated!
Configuration Service Status

Service Name*

System Attribute from

60

Workflow

Every 24th polling cycle


(or once per day) or

Service Status!

whenever there is a
severity change on the
thresholds.

Appliance Capacity Totals

Storage Pool*

System Attribute from


Workflow

Pool Owner
Pool Status

20

Every 72nd polling cycle


(or once per day) or
whenever there is a
severity change on the
thresholds.

Data Profile
Available (GB)

12

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC NAME

COLUMN (*=KEY FIELD,

ORACLE ZFS STORAGE

POLLING

UPLOAD FREQUENCY

! = CONDITIONS ENABLED)

APPLIANCE DATASET

INTERVAL

TO EM REPOSITORY

(MINUTES)

(X=DISABLED BY DEFAULT)

Total Used (GB)


Pool Used Percentage (%)!
Total Capacity (GB)
Data Disks
Spare Disks!
Log Disks
Cache Disks
Total Project Quota (GB)!
Quota Allocation (GB)!
Reservations (GB)!
Pool Over Allocated!
X

X
X

Cache ARC

Size (MB)!

arc.size

30

Every polling cycle.

Cache L2ARC

Size (MB)!

arc.l2_size

30

Every polling cycle.

CPU

Utilization %!

cpu.utilization

30

Every polling cycle.

Disk I/O Ops

Operations per Second!

io.ops

30

Every polling cycle.

Disk I/O Ops Latency

Operations per Second!

io.ops[latency]

30

Every polling cycle.

Disk I/O Bytes

KB per Second!

io.bytes

30

Every polling cycle.

Network Device

KB per Second!

nic.kilobytes

30

Every polling cycle.

NFSv2 Ops

Operations per Second!

nfs2.ops

30

Every polling cycle.

NFSv3 Ops

Operations per Second!

nfs3.ops

30

Every polling cycle.

NFSv3 Ops Latency

Operations per Second!

nfs3.ops[latency]

30

Every polling cycle.

NFSv4 Ops Latency

Operations per Second!

nfs4.ops[latency]

30

Every polling cycle.

NFSv4 Ops

Operations per Second!

nfs4.ops

30

Every polling cycle.

13

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC NAME

COLUMN (*=KEY FIELD,

ORACLE ZFS STORAGE

POLLING

UPLOAD FREQUENCY

! = CONDITIONS ENABLED)

APPLIANCE DATASET

INTERVAL

TO EM REPOSITORY

(MINUTES)

(X=DISABLED BY DEFAULT)
X

SMB Ops

Operations per Second!

smb.ops

30

Every polling cycle.

SMB Ops Latency

Operations per Second!

smb.ops[latency]

30

Every polling cycle.

iSCSI Ops

Operations per Second!

iscsi.ops

30

Every polling cycle.

iSCSI Ops Latency

Operations per Second!

iscsi.ops[latency]

30

Every polling cycle.

Protocol*

System Attribute from

60

Every 24th polling cycle

Initiator And Target Groups

Workflow

(or once per day).

Group Name*
Address
Initiator Or Target
X

Cache ARC Activity

Cache L2ARC Activity

Cache ARC Activity by Share

Cache L2ARC Activity by

Activity*

arc.accesses[hit/miss]

Accesses per Second!

arc.accesses[hit/miss]

Activity*

arc.l2_accesses[hit/miss]

Accesses per Second!

arc.l2_accesses[hit/miss]

Share*

arc.accesses[share]

Accesses Per Second!

arc.accesses[share]

Share*

arc.l2_accesses[share]

Accesses Per Second!

arc.l2_accesses[share]

NFS V3 Share*

nfs3.ops[share]

Operations Per Second!

nfs3.ops[share]

NFS V4 Share*

nfs4.ops[share]

Operations Per Second!

nfs4.ops[share]

Direction*

nic.kilobytes[direction]

Kilobytes Per Second!

nic.kilobytes[direction]

Device*

nic.kilobytes[device]

30

Every polling cycle.

30

Every polling cycle.

30

Every polling cycle.

30

Every polling cycle.

30

Every polling cycle.

30

Every polling cycle.

30

Every polling cycle.

30

Every polling cycle.

Share
X

NFS V3 Operations By Share

NFS V4 Operations By Share

Network I/O Direction

Network I/O by Device

14

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC NAME

COLUMN (*=KEY FIELD,

ORACLE ZFS STORAGE

POLLING

UPLOAD FREQUENCY

! = CONDITIONS ENABLED)

APPLIANCE DATASET

INTERVAL

TO EM REPOSITORY

(MINUTES)

(X=DISABLED BY DEFAULT)

Network Interface

Kilobytes Per Second!

nic.kilobytes[device]

Interface*

net.kilobytes[interface]

Kilobytes Per Second!

net.kilobytes[interface]

Cluster State!

System Attributes from

30

Every polling cycle.

60

Every 24th polling cycle

Performance

Physical Component Status

Workflow

(or once per day) or

Cluster State Description

whenever there is a
severity change on the

Cluster Peer State

thresholds.

Cluster Peer State Description


Appliance Information

Reported Node Name

System Attributes from

Product Model

Workflow

60

Every 24th polling cycle


(or once per day).

Appliance Serial Number


Chassis Serial Number
Appliance Software Version
Install Time
Last Update Time
Power On Time
Up Time
Clustered
Cluster Peer
X

Appliance Logical Information

Share Identifier*

System Attributes from


Workflow

60

Every 24th polling cycle


(or once per day).

Share Type
Mount Point
Data Profile
Data Compression
Cache Usage
NFS
CIFS

15

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC NAME

COLUMN (*=KEY FIELD,

ORACLE ZFS STORAGE

POLLING

UPLOAD FREQUENCY

! = CONDITIONS ENABLED)

APPLIANCE DATASET

INTERVAL

TO EM REPOSITORY

(MINUTES)

(X=DISABLED BY DEFAULT)

IP Network Interfaces

Interface*

System Attributes from

Every 24th polling cycle

60

Workflow

(or once per day).

IP Address*
CIDR
IP Version
Management Enabled
IP Network Interface to Datalink

Interface*

Relationship

System Attributes from

60

Every 24th polling cycle

Workflow

(or once per day).

Datalink*
Jumbo Frames
IP Datalink to Device

Datalink*

Relationship

System Attributes from

60

Every 24th polling cycle

Workflow

(or once per day).

Link*
Class
IP Network Devices

Network Device*

System Attributes from

Device Speed!

Workflow

60

Every 24th polling cycle


(or once per day) or
whenever there is a

Device Up!

severity change on the


thresholds.

Alert Log Contents

Problem ID*

System Attributes from

Time Stamp

Workflow

20

Every polling cycle.

20

Every polling cycle.

Alert Description!
Alert Log Problem Type!
Fault Log Contents

Problem ID*

System Attributes from

Time Stamp

Workflow

Description!
Fault Log Problem Type!
Problem Log Contents

UUID*

System Attributes from

Description!

Workflow

20

Every polling cycle.

20

Every polling cycle.

Problem Severity!
Problem Log Code!
Replication Activity

Problem ID*

16

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC NAME

COLUMN (*=KEY FIELD,

ORACLE ZFS STORAGE

POLLING

UPLOAD FREQUENCY

! = CONDITIONS ENABLED)

APPLIANCE DATASET

INTERVAL

TO EM REPOSITORY

(MINUTES)

(X=DISABLED BY DEFAULT)

Time Stamp

System Attributes from

Replication Description!

Workflow

Replication Log Problem Type

Metric Collections on Datasets


By default, all Metrics that collects information from datasets on Oracle ZFS Storage
Appliances are disabled. If you are interested in any particular dataset, please enable the
Metric Collection for that dataset.
There are two methods to enable metric collection for datasets on Oracle Enterprise
Manager.
Method 1:
1. Go to the Target Homepage.
2. Click on the Oracle ZFS Storage Appliance Menu > Monitoring > All Metrics.
3. Select the Metric that you wish to enable.
4. Click on the Modify button.
5. Click on the Enabled radio button for Data Collection and Click OK.
Method 2:
1. Go to the Target Homepage.
2. Click on the Oracle ZFS Storage Appliance Menu > Monitoring > Metric and
Collections Setting.
3. Select All Metrics under the View drop-down menu.
4. Find the Metric that you wish to enable.
5. Select the Disabled link under the Collection Schedule column.
6. Click on the Enabled button under Data Collection and click Continue.
7. You will be brought back to the Metric and Collections Settings page with a
notification box. Ensure that you click on OK.

17

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

You can use the previous steps to disable a dataset collection at any time.
Note: Once you enable the metric collection for any dataset, you must ensure that the
storage administrators also enable the same dataset on the Oracle ZFS Storage
Appliances; otherwise you will start receiving metric collection errors.

Metric Collections on Pools, Projects and Shares


The following metrics that collect information about the pool, projects, and shares
information are disabled by default:

Appliances Capacity Total

Appliances Project Status

Appliances Logical Status

Appliances Logical Configuration

Please follow the same steps shown previously for enabling dataset metrics if you would
like to collect information on these metrics.
Note: Metrics and information presented on charts can take up to 60 minutes to populate
as a result of polling intervals for relatively static information. Also, due to differing polling
cycles between statistics, some information can populate prior to other information.

Reports
The following table lists available reports and the information they can provide on the
Oracle ZFS Storage Appliances status.

18

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

TABLE 2: REPORTS
REPORT

DESCRIPTION

PURPOSE

Monitored System Home Page

Contains a series of graphs that give a high-level view Provides at-a-glance status of the system.
of system throughput and clients accessing the
system. It is important to note whether a system is Up
or Down. Up means the management interface is
available and simple properties can be queried that
do not require login. Down means the management
interface is not available. If a system is set up with
more than one interface that allows access to the
data path, it could be Down when the data path
interface is still accessible. The management
interface being flagged as Down is sufficient to
warrant attention to the system.

Database to Appliance Mapping

Shows mapping of databases using NFS to access

Provides a detailed view of how an Oracle

Report -- available from the

data residing on an Oracle ZFS Storage Appliance.

ZFS Storage Appliance is being accessed

Reports tab or from a

Uses Detail Report links to view deeper information

using Oracle databases. Provides easy

preselected Target Report's tab

on mappings and component activity. Detail Reports

navigation to drill into the configuration

provide information such as:

and activity of Oracle ZFS Storage

Database file mapping to exports residing

Appliance components that relate to the

on appliances

database deployment.

Export data on which files reside


(including capacity consumption and
cache activity)

The appliance on which a share resides

The network interface and its activity


through which a database file is accessed

Storage Capacity Report --

Gives detailed information about how storage is used

Shows how capacity is allocated on an

available from the Reports tab

on an individual Oracle ZFS Storage Appliance.

Oracle ZFS Storage Appliance.

Network Interface Information

Provides details of network configuration and

Provides a detailed view of how ports on

Report -- available from the

generalities about network activity over the past 24

an Oracle ZFS Storage Appliance are

Reports tab or from a

hours.

configured. Shows aggregate activity of

or from a preselected Target


Report's tab

preselected Target Report's tab

the interfaces.

19

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

REPORT

DESCRIPTION

PURPOSE

Federated Database to

Similar to the Database to Appliance Mapping Report, Provides a detailed view of how all

Appliance Mapping Report --

but applicable to all monitored Oracle ZFS Storage

monitored Oracle ZFS Storage Appliances

available from Reports tab>

Appliances.

are being accessed for information

Storage>Network Attached

through Oracle databases. Provides easy

Storage group

navigation to drill into the configuration


and activity of individual Oracle ZFS
Storage Appliances and their components.

Federated Capacity and

Gives high-level information about all monitored

Provides a side-by side-comparison of all

Configuration Report -- available Oracle ZFS Storage Appliances.

Oracle ZFS Storage Appliances (and

from Reports tab>Storage>

software installed on them) to help you

Network Attached Storage group

make decisions on where to allocate


capacity.

Default Threshold Values


You can set custom thresholds for some metrics within Oracle Enterprise Manager Cloud
Control. The alerts received are contained within the product and are not set as Alerts and
Thresholds on the Oracle ZFS Storage Appliance itself.
The following table shows the metrics that have thresholds set with their default values.
TABLE 3: METRICS AND DEFAULT THRESHOLD VALUES
METRIC/COLUMNS

COMPARISON

WARNING

CRITICAL

PURPOSE

N/A

N/A

Issued when the Alert Description Matches (via

OPERATOR

Alert Log Contents/Alert Description

Matches

Regular Expression) the thresholds.


Alert Log Contents/Alert Log Problem

Matches

N/A

N/A

Type

Issued when the Alert Log Problem Type Matches


(via Regular Expression) the thresholds.
Problem Type can be Minor/Major/Critical.

Appliances Capacity Totals/Pool Over

N/A

YES

Allocated
Appliances Capacity Totals/Pool Used

the Pool's Total Capacity (GB).


>=

80

90

Percentage
Appliances Capacity Totals/Quota
Allocations

Issued when the Total Project Quota (GB) crosses

Issued when the total capacity available on this


appliance crosses the threshold.

>=

N/A

N/A

Issued when the Quota Allocations crosses the


thresholds.

20

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC/COLUMNS

COMPARISON

WARNING

CRITICAL

PURPOSE

N/A

N/A

Issued when the Reservation crosses the

OPERATOR

Appliances Capacity

>=

Totals/Reservations
Appliances Capacity Totals/Spare

thresholds.
<=

Disks

Issued when the number of spare disks for a


storage pool goes to one or zero, indicating a loss
of a disk is a critical occurrence.

Appliances Capacity Totals/Total

>=

N/A

N/A

Project Quota
Appliances Logical Status/Reserved

Issued when the Total Project Quota crosses the


thresholds.

N/A

N/A

Issued when the Reserved value is equal to the


thresholds.

Appliances Logical Status/Share Data

>=

N/A

N/A

Issued when the Share Data crosses the


thresholds.

Appliances Logical Status/Share

>=

80

90

Percentage Used

Any share issues these warnings. A Cloud Control


administrator can make specific shares trigger on
other values.

Appliances Logical Status/Unused

<=

N/A

N/A

Reserved

Issued when the Unused Reserved falls below the


thresholds.

Appliances Project Status/Project Data >=

N/A

N/A

Issued when the Project Data crosses the


thresholds.

Appliances Project Status/Project Over =

N/A

YES

Allocated

Issued when the Total Share Quota (GB) of any


project crosses the corresponding Project Quota
(GB).

Appliances Project Status/Project

>=

80

90

Percentage Used
Appliances Project Status/Reserved

Issued when the Project Percentage used crosses


the thresholds.

N/A

Issued when the Reserved value is equal to the


thresholds.

Appliances Project Status/Total Share

>=

N/A

N/A

Quota
Appliances Project Status/Unused

the thresholds.
<=

N/A

N/A

Reserved
Cache ARC

Issued when the Total Share Quota used crosses

Issued when the Unused Reserved falls below the


thresholds.

>=

N/A

N/A

Issued when Cache ARC (MB) crosses the


thresholds.

21

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC/COLUMNS

COMPARISON

WARNING

CRITICAL

PURPOSE

N/A

N/A

Issued when Cache ARC Activity (Accesses per

OPERATOR

Cache ARC Activity

>=

Second) crosses the thresholds.


Cache ARC Activity by Share

>=

N/A

N/A

Issued when Cache ARC Activity by Share


(Accesses per Second) crosses the thresholds.

Cache L2ARC

>=

N/A

N/A

Issued when Cache L2ARC (MB) crosses the


thresholds.

Cache L2ARC Activity

>=

N/A

N/A

Issued when Cache L2ARC Activity (Accesses per


Second) crosses the thresholds.

Cache L2ARC Activity by Share

>=

N/A

N/A

Issued when Cache L2ARC Activity by Share


(Accesses per Second) crosses the thresholds.

Configuration Service Status

N/A

N/A

This threshold is enabled but left off because a


service that is down does not necessarily indicate a
problem (the service might not be configured or
operating by plan).

CPU Utilization %

>=

80.00%

90.00%

Issued after three separate polling intervals. See


the CPU Utilization above this percentage. Overuse
of CPU can result in lower performance.

Disk I/O Bytes

>=

N/A

N/A

Issued when Disk I/O Bytes (KB/S) crosses the


thresholds.

Disk I/O Ops

>=

N/A

N/A

Issued when Disk I/O Ops (Operation per Second)


crosses the thresholds.

Disk I/O Ops Latency

>=

N/A

N/A

Issued when Disk I/O Ops Latency (Operation per


Second) crosses the thresholds.

Fault Log Contents/Description

Matches

N/A

N/A

Issued when the Fault Description Matches (via


Regular Expression) the thresholds.

Fault Log Contents/Fault Log Problem

Matches

N/A

N/A

Type

Issued when the Fault Log Problem Type Matches


(via Regular Expression) the thresholds.
Problem Type can be Minor/Major/Critical.

IP Network Device/Device Speed

<=

N/A

N/A

Issued when Device Speed (KB/S) falls below the


thresholds.

22

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC/COLUMNS

COMPARISON

WARNING

CRITICAL

PURPOSE

N/A

N/A

This threshold is enabled but left off because a

OPERATOR

IP Network Device/Device Up

device that is down does not necessarily indicate a


problem (the device might not be configured or
operating by plan).
iSCSI Ops

>=

N/A

N/A

Issued when iSCSI Ops (Operation per Second)


crosses the thresholds.

iSCSI Ops Latency

>=

N/A

N/A

Issued when iSCSI Ops Latency (Operation per


Second) crosses the thresholds.

Monitoring Prerequisite/Status

N/A

Issued when the credentials provided are incorrect.


Once this alert is triggered, all metric collection will
stop until this alert is cleared.

Network Device

>=

N/A

N/A

Issued when Network Device (KB/S) crosses the


thresholds.

Network I/O by Device

>=

N/A

N/A

Issued when Network I/O by Device (KB/S) crosses


the thresholds.

Network Interface Performance

>=

N/A

N/A

Issued when Network Interface Performance


(KB/S) crosses the thresholds.

NFSv2 Ops

>=

N/A

N/A

Issued when NFSv2 Ops (KB/S) crosses the


thresholds.

NFSv3 Ops

>=

N/A

N/A

Issued when NFSv3 Ops (KB/S) crosses the


thresholds.

NFSv3 Ops by Share

>=

N/A

N/A

Issued when NFSv3 Ops by Share Ops (KB/S)


crosses the thresholds.

NFSv3 Ops Latency

>=

N/A

N/A

Issued when NFSv3 Ops Latency (KB/S) crosses


the thresholds.

NFSv4 Ops

>=

N/A

N/A

Issued when NFSv4 Ops (KB/S) crosses the


thresholds.

NFSv4 Ops by Share

>=

N/A

N/A

Issued when NFSv4 Ops by Share Ops (KB/S)


crosses the thresholds.

NFSv4 Ops Latency

>=

N/A

N/A

Issued when NFSv4 Ops Latency (KB/S) crosses


the thresholds.

23

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC/COLUMNS

COMPARISON

WARNING

CRITICAL

PURPOSE

AKCS_

AKCS_

State of the cluster. OWNER means this head is

OWNER

STRIPPED

the owner of the cluster and that resources are

OPERATOR

Physical Components Status/Cluster

States

likely in contention. STRIPPED means that the


head's resources are owned by its peer and must
have the takeover redone.
Problem Log Contents/Description

Matches

N/A

N/A

Issued when the Problem Description Matches (via


Regular Expression) the thresholds.

Problem Log Contents/Problem Log

Matches

N/A

N/A

Code
Problem Log Contents/Problem

Issued when the Problem Log Code Matches (via


Regular Expression) the thresholds.

Matches

N/A

Severity

Major|

Issued when the Problem Severity Matches (via

Critical

Regular Expression) the thresholds.


Problem Type can be Minor/Major/Critical.

Response/Status

N/A

Down

Provides information to the Home Page report on


availability of the system's management path.

Replication Activity/Replication

Matches

N/A

Description
SMB Ops

>=

N/A

Replication

Issued when the Replication Description Matches

of.*

(via Regular Expression) the thresholds.

N/A

Issued when SMB Ops (Operation per Second)


crosses the thresholds.

Stateless Alerts
Most of the alerts that are set in the preceding table are stateful alerts. They will be
automatically cleared once the underlying conditions are cleared. F or example, if a
threshold of 20 is set for SMB Ops, an alert will be triggered once the value crosses 20,
and cleared once the value drops below 20.
However, there are a few special cases where the alerts cannot be cleared automatically.
In these scenarios, these alerts are defined as stateless and you must manually clear
them. All alerts from the following metrics are stateless alerts:

Alert Log

Fault Log

24

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

Problem Log

Replication

To clear stateless alerts, follow these steps:


1. Click on the Alert either from the Homepage or Incident Manager.
2. Click on the Clear button.
3. Click on Yes to permanently clear the alert.

Oracle ZFS Storage Appliances Alerts


If real-time monitoring for performance is required, the storage administrator should create
appliance alerts and thresholds on the Oracle ZFS Storage Appliances user interfaces.
This allows the storage administrator to view performance issues using real-time
indicators, as opposed to the slower polling interval required by Oracle Enterprise
Manager Cloud Control.

Communicating Performance Issues to the Appliance


Administrator
The most common use of the Oracle Enterprise Manager Cloud Control Plug-in (besides
simple capacity monitoring and high-level information collection) is analysis of application
performance degradation relating to a database. Using the information within Oracle
Enterprise Manager Cloud Control, you can provide to the Oracle ZFS Storage Appliance
administrator information related to performance degradation for an application on the
appliance.
An Oracle ZFS Storage Appliance administrator has the advantage of real-time appliance
analytics that can be viewed in various time spans (including initial configuration),
monitored by Oracle Enterprise Manager Cloud Control. Some datasets are available from
the first power-on of the appliance.
When you present performance information to the storage administrator, you can start the
analysis process by using the Oracle Enterprise Manager worksheet. This worksheet
displays a down to the second view of the data compared to the less useful view of the
data available to Oracle Enterprise Manager Cloud Control (often a 10- or 20-minute

25

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

collection cycle averaged over the period, resulting in a much less dramatic performance
curve).
Using the worksheet and/or other datasets available to the storage administrator, you can
analyze the cause down to the hardware or software component level. In the event of a
resource contention issue, you can study levels of client access to determine how and
when individual clients are accessing a system, along with the resources they are
accessing. These can include network interfaces, logical resources (such as shares), and
more.
When initial communication is complete, the storage administrator should complete the
analysis of the problem and propose a way to resolve a disruption in quality of service. If
the problem is not likely associated with the Oracle ZFS Storage Appliance, the storage
administrator can provide this information.

Troubleshooting Monitoring Deployments


A variety of reasons could cause a monitored instance in the Oracle Enterprise Manager
Cloud Control to return information improperly. These issues are typically associated with
misconfiguration, incorrect permissions, or changing the configuration within one of the
products.

26

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

TABLE 4: APPLIANCE PROBLEMS AND SOLUTIONS


PROBLEM

SOLUTION

Appliance does not appear to be Up

If the appliance previously reported as Up but is now indicating it is Down, this is an indication

after being added as an instance to

that the management path is no longer available on the target Oracle ZFS Storage Appliance.

be monitored.

The data path might still be available on the appliance, but this critical problem must be
addressed.
If this is the first time an appliance was added and it has never reported as being Up, an error
could have been made when setting up the instance for monitoring. Check the following:

Is the designated DNS name or IP address correct in the asset's Monitoring


Configuration?

Is the port number correct? The Oracle ZFS Storage Appliance default port is 215.

Is the named asset accessible from the Agent system?

Note: The Up/Down indicator does not require authentication to the target Oracle ZFS
Storage Appliance, so the user ID and password are not relevant here.
Checking the value of the Response Metric should show 0. This indicates that the Agent
properly invoked the metric collection program, but the metric collection program could not
connect to the target Oracle ZFS Storage Appliance given the parameters in the Monitoring
Configuration.
After determining 0 is shown in the field, ensure the appliance is available at the configured IP
address in the monitoring configuration. However, if the value of Response Metric is a 1, the
appliance is likely available. In this case, refresh the Cloud Control page reporting that the
appliance is down. It will likely return to "Up".
Appliance is Up, but all metrics are

Check the metric collection errors. (See Table 4: Common Metric Collection Errors.) The most

not being collected.

common problem is misconfiguration of the password for the oracle_agent user within
the Oracle Enterprise Manager Cloud Control environment.
For most metric collection errors, a summary message and Java Stack Trace is generated
and viewable by clicking the Metric Collection Errors link under the Home Page reports, then
clicking through the text Exception. The initial summary message will contain information
about the problem.

27

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

PROBLEM

Appliance is Up, but some metrics

SOLUTION

Determine what metrics are not being collected. Most likely, it is a group of metrics such as:

are not being collected.

ARC Activity by Share

L2ARC Activity

L2ARC Activity by Share

NFS V3 Operations by Share

NFS V4 Operations by Share

There are usually three reasons for selective metrics not being collected:

A component does not exist on a system This is common with the L2ARC metrics. In
this case, you will not receive a metrics collection error; rather, the Last Upload time on the
All Metrics page will remain empty (or include a -).

The Configure for Oracle Enterprise Manager Monitoring workflow was not run on
the target Oracle ZFS Storage Appliance. This problem embodies in metrics that
are not collected by default on a target appliance, such as those listed above. In
this case, Metric Collection Errors will occur within the Oracle Enterprise Manager
Cloud Control until the target Oracle ZFS Storage Appliance is correctly
configured (See Table 4: Common Metric Collection Errors.)

A storage administrator either disabled or removed datasets on the target system.


Ask the storage administrator to re-enable the dataset name that is in error. See
Metric Information in Table 1: Metric and Collection Information.

The list of shares in the Appliance

The layout of filesystems and their values are collected at different intervals than the capacity

Logical Configuration and the

information for them. In the worst case, a metric could be as much as 60 minutes out of sync

Appliance Logical Status appear to

with a storage appliance. If you have a need for real-time information, refer to the Oracle ZFS

be out of sync.

Storage Appliance BUI.

A Report or Metric Detail is not

Metrics and information about system values are collected in varying intervals. In the worst

showing a change made on the

case, a metric could be as much as 60 minutes out of sync with a storage appliance. If you

system.

have a need for real-time information, refer to the Oracle ZFS Storage Appliance BUI.

Unexpected Workflow Errors in

Be sure you are running the configuration script as a user with root access (or equivalent)

running the Configuration Workflow

with respect to creating and manipulating datasets and users.

on the Oracle ZFS Storage


Appliance, such as aksh: uncaught
error exception (code EAKSH_
GENERIC_DATASET_
INVALID)

28

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

PROBLEM

SOLUTION

Database mapping reports do not

The host, database, or both are not being monitored by Cloud Control. Install the proper

show database artifacts that are

agents on the host to ensure that storage attached to the system is being monitored.

mapped to NFS v3 or v4 shares on

Refer to the Oracle Enterprise Manager Cloud Control documentation for information on

an Oracle ZFS Storage Appliance.

monitoring host and database assets. See the Oracle Enterprise Manager Cloud Control
documentation listed in the Additional References section at the end of this document .

Database mapping reports do not

iSCSI and Fibre Channel mappings are not supported in this release.

show database artifacts that are


mapped to iSCSI or Fibre Channel
shares on an Oracle ZFS Storage
Appliance.

Metric Collection Errors


Some common configuration problems manifest as Metric Collection Errors. In most
cases, these error messages contain two parts a short string followed by a longer Java
Stack Trace. The first string usually contains enough information to isolate the problem.
The Java Stack Trace is used by the Support Center (if the problem is complex). Common
Metric Collection problems are as follows:
TABLE 5: COMMON METRIC COLLECTION ERRORS
METRIC COLLECTION ERROR

PROBLEM / SOLUTION

login denied

Problem: The user ID/password combination on the target Oracle ZFS Storage Appliance is
incorrect.
Possible causes:

The Monitoring Configuration in Oracle Enterprise Manager Cloud Control is


using the incorrect user ID/password. The user ID should be oracle_agent
and the password should be the one provided by the storage administrator.

The target Oracle ZFS Storage Appliance does not have the proper user ID and
role.

Solution:
Change the monitoring configuration to use the proper user ID/password provided by the
storage administrator, or reconfigure the target Oracle ZFS Storage Appliance to use the
correct oracle_agent password.

29

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC COLLECTION ERROR

PROBLEM / SOLUTION

workflow parameters are invalid

Problem: The parameters passed to the Oracle ZFS Storage Appliance are invalid.
Possible causes:

A dataset on the Oracle ZFS Storage Appliance is not enabled.

Resources from a monitored cluster head failed to the secondary head, for which
the storage administrator did not run the Configuration workflow.

An internal error occurred and the Agent is calling an incompatible workflow.

Solution:

Run the Configuration workflow on the target Oracle ZFS Storage Appliance and
recreate the worksheet.

Ask the storage administrator to ensure that all of the datasets in the created
worksheet are being collected properly.

(dataset name) is not enabled on the

Problem: A dataset cannot be found on the Oracle ZFS Storage Appliance.

filer. Please enable the dataset on the

Possible causes:

filer or disable the collection of this

metric.

A dataset on the Oracle ZFS Storage Appliance is not enabled.

Solution:

Ask the storage administrator to enable the dataset on the Oracle ZFS Storage
Appliance.

Disable the collection of this dataset metric with the following steps:

1.

Navigate to Metric and Collection Settings of the target.

2.

Change the Drop-down View option from All Metrics.

3.

Look for the metric name you wish to disable.

4.

Click on the Collection Interval (such as Every 30 minutes).

5.

Click the Disable button to disable collection of this metric.

30

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

METRIC COLLECTION ERROR

PROBLEM / SOLUTION

cannot find specified workflow

Problem: A workflow that collects and returns metrics to the Agent cannot be found on the
Oracle ZFS Storage Appliance.
Possible causes:

The oracle_agent user (or user that you configured in the Monitoring
Configuration) does not have workflow.*.* read authorization.

An Oracle ZFS Storage Appliance is being used without the workflow on it, either
because the version of the Oracle ZFS Storage Appliance software is incorrect,
or the workflow was removed by a storage administrator.

Solution:
1.

Run the Unconfiguration workflow on the target Oracle ZFS Storage Appliance
and remove the oracle_agent user.

2.

Run the Configuration workflow on the target Oracle ZFS Storage Appliance and
recreate the oracle_agent user.

3.

Ask the storage administrator to ensure the Configure/Unconfigure workflows are


on the system.

4.
workflow failed

Contact the Support Center if the Agent fails to start collecting the dataset again.

Problem: The Agent is accessing an Oracle ZFS Storage Appliance, but the workflow is
failing in some way.
Possible causes:

A problem in the metric collection logic is causing it to fail on the Oracle ZFS
Storage Appliance.

Solution:
Run the Configuration workflow on the Oracle ZFS Storage Appliance and recreate the
worksheet.
If you are still receiving this error after recreating the worksheet, contact the Support Center.

Oracle ZFS Storage Appliance Audit Logs


For detailed debug information, use the audit log on the target Oracle ZFS Storage
Appliance to determine which actions occurred with the workflows. Creation and deletion
of artifacts (or the failure to create or delete an artifact) are traced in the audit log.
Note: Debugging the reason that an Agent is not properly collecting statistics from an
Oracle ZFS Storage Appliance can be more difficult and should only be done by service
personnel. Contact the Oracle Support Center (see Additional Resources at the end of this
document).

31

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

Host Agent Logs


Most problems (other than version mismatches) occur during the collection of metrics or
response information. Look for the following logs on the Agent host. The Agent host may
differ from the management station location. In the following file locations, %AGENT_LOCATION
% indicates the home directory of the Agent, typically similar to
/export/home/oracle/OracleHomes/agent10g. An asterisk (*) indicates there are several files
with additional extensions.
TABLE 6: HOST AGENT LOGS

Agent Log Location

Description

%AGENT_HOME%/sysman/emd/oracle_zfs_plugin/logs/

Contains errors that occurred below the Oracle Enterprise

{target_ip}/sun_storage_7000_pipe.log*

Manager Cloud Control framework during attempts to


communicate with the Oracle ZFS Storage Appliance. These files
will typically contain fine-grained details on connection problems
with an Oracle ZFS Storage Appliance or failures that occur while
retrieving information.

%AGENT_HOME%/sysman/log/emagent.trc*

These logs also contain connection exceptions as well as any


information on using the data returned from the system to
populate database tables.

%AGENT_HOME%/sysman/log/*

Remaining logs will contain finer-grained details on various


elements in the Cloud Control Framework and 90% of the issues
can be diagnosed with the above.

Note: When interacting with the Support Center or developers within your own help desk,
include the sun_storage_7000_pipe.log* files and emagent.trc* files for reference.

32

Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

Additional Resources

Oracle Support Center


http://www.oracle.com/support

Patches and updates downloads from My Oracle Support (MOS)


https://support.oracle.com/CSP/ui/flash.html#tab=PatchHomePage
%28page=PatchHomePage&id=g1ws8maj%28%29%29

Oracle Unified Storage Systems Documentation


http://www.oracle.com/technetwork/documentation/oracle-unified-ss-193371.html

Oracle Enterprise Manager Concepts Guide

http://docs.oracle.com/cd/B10501_01/em.920/a96674/toc.htm

Management of Heterogeneous Data Centers Using Oracle Enterprise Manager


Plug-ins,
An Oracle White Paper
http://www.oracle.com/technology/products/oem/pdf/dc_mgmt.pdf

Oracle Enterprise Manager Cloud Control 12c documentation, including Oracle


Enterprise Manager Concepts and Oracle Enterprise Manager Administration
Guide
http://docs.oracle.com/cd/E24628_01/index.htm

Oracle Enterprise Manager Cloud Control Upgrade Guide


http://download.oracle.com/docs/cd/E24628_01/upgrade.121/e22625.pdf

Oracle Enterprise Manager, Cloud Control Extensibility Programmers Reference,


12c Release3 (12.1.0.3)
http://docs.oracle.com/cd/E24628_01/doc.121/e25161/title.htm

33

Oracle Enterprise Manager System Monitoring

Copyright 2013, Oracle and/or its affiliates. All rights reserved. This document is provided for

Plug-in for Oracle ZFS Storage Appliance,

information purposes only and the contents hereof are subject to change without notice. This document is

Installation Guide

not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally

November 2013, Version 1.4.1

or implied in law, including implied warranties and conditions of merchantability or fitness for a particular

Oracle Application Integration Engineering

purpose. We specifically disclaim any liability with respect to this document and no contractual obligations
are formed either directly or indirectly by this document. This document may not be reproduced or

Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.

transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior
written permission.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be
trademarks of their respective owners.

Worldwide Inquiries:

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks

Phone: +1.650.506.7000

are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD,

Fax: +1.650.506.7200

Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced
Micro Devices. UNIX is a registered trademark licensed through X/Open Company, Ltd. 0611

oracle.com

You might also like