1/3(W)

EMC Knowledgebase

"VPLEX: How to report and open a VPLEX performance call or service request"
ID:

emc247209

Usage:

15

Date Created:

07/28/2010

Last Modified:

01/25/2013

STATUS:

Approved

Audience:

Support

Knowledgebase Solution
How to report and troubleshoot a VPLEX performance issue

Question:

What is needed by PREM and Field before opening a Performance call with Engineering

Question:

VPLEX: How to report and open a VPLEX performance call or service request

Environment:

EMC SW: GeoSynchrony 4.0

Environment:

EMC SW: GeoSynchrony 4.0.1

Environment:

EMC SW: GeoSynchrony 4.1

Environment:

EMC SW: GeoSynchrony 4.2

Environment:

EMC SW: GeoSynchrony 5.0

Environment:

EMC SW: GeoSynchrony 5.0.1

Environment:

EMC SW: GeoSynchrony 5.0.1 P 1

Environment:

EMC SW: GeoSynchrony 5.0.1 P 2

Environment:

EMC SW: GeoSynchrony 5.1

Environment:

EMC SW: GeoSynchrony 5.1 P 1

Environment:

EMC SW: GeoSynchrony 5.1 P 2

Environment:

EMC SW: GeoSynchrony 5.1 P 3

Environment:

Product: VPLEX VS1

Environment:

Product: VPLEX VS2

Environment:

Product: VPLEX Local

Environment:

Product: VPLEX Metro

Environment:

Product: VPLEX Geo

Problem:

Performance problem involving VPLEX

nR

eg

is

te

re

Question:

Before opening a Performance Service request the Customer, EMC Field and Support team need to please read and take action on points
through 6 below.
Commonly reported VPLEX performance problems
1. "Performance is low"

This is a very generic description. Please ask your customer to express the performance issue in terms of response time, IOPS, or
bandwidth (KB/s).
In order to avoid unnecessary investigations,a clear statement of the perceived problem and acceptable solution should b
formulated before undertaking performance troubleshooting.
It is good to ask:
Fix:

Do you have established performance requirements for applications in the form of Service Level Agreements
you tracked and analyzed the achieved performance
to ensure that those requirements are met? In other words, do you have a known good baseline performance data?
Has anything changed recently in the load, such as growth in the number of users, or increased demand from existing user
These changes can cause previously acceptable
application performance to degrade and reduce the ability of the system to satisfy the established SLA. How does the curre
compare to your baseline?
3. "Application performance got worse after introducing VPLEX"

A commonly reported problem is that a back-up operation or particular batch job now takes longer under VPLEX than it did before a
VPLEX.
Ask if the baseline metric is still applicable to the current environment. For example:

http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=Def... 2013/4/21

2/3(W)

How old is the target baseline? Has the current load on the new VPLEX system changed since the non-virtualized
measurement was taken?
Is the target performance metric measured on the same system before VPLEX was added to it, or was it on another system
there any differences between the baseline
configuration and the current VPLEX configuration (other than VPLEX of course)?
If feasible, re-measure the native (host to storage-array) baseline on the same system that VPLEX was added
5. Understand the implications of VPLEX Metro
It is important to understand the implications of write through cache mode and VPLEX Metro.Writes to VPLEX distributed
written to the storage-array at each cluster before the operation is acknowledged to the host. There will be additional WAN round
added to write operations that were not applicable to the baseline configuration which was likely to a purely local single data center
configuration.
6. "Bandwidth (MB/s) or IOPS is lower than expected" or "Response time is higher than expected"
On the host Check the I/O size:

Bandwidth: Maximum throughput performance is achieved with large I/O sizes, typically 64KB or larger. EMC recommend
VPLEX to not exceed 128KB I/O sizes. Anything larger results in serialized I/O operations of 128KB chunks from within
the back-end storage-array, and increases the observed host response time.
IOPS: Maximum IOPS performance is achieved with smaller I/O sizes. EMC recommends for VPLEX to test with 4KB I/O siz
since this results in whole VPLEX cache page operations. Response time: Large IO sizes result in higher response times.
When you gather the information outlined in 1-4 please proceed.
8. Provide a detailed problem description including:

re

a. Server Type(s), OS version(s) and HBA type(s).BE Array Type(s).

b. I/O profile with Details on FE paths, BE paths, # of volumes and storage volumes, size, and layout.
c. Performance timelines when it started/ended and how i/o traffic was varied.
d. The Time difference between devices. These vary from system to system and knowing the differences helps with debugging

nR

eg

is

te

9. When troubleshooting issues, gather logs from all components in your surrounding storage environment.
Collect-diagnostics from both VPLEX clusters as per EMC Knowledgebase solution emc238545 "VPLEX: How to collect log
a VPLEX instance".
Host Logs- EMC Grabs
Collect Host Logs such as emcGrab,emcreports,esxgrab and application logs.
Get timestamps from Hosts having issues
Collect any host related performance data (SAR,IO Meter,IO Stat, Perfmon,other).
FC fabric logs - supportsaves, Show tech-support details
Brocade log collection reference emc42373 and emc93016
Cisco log collection reference emc242289
WAN device logs (similar to 3 for Metro)
Storage Array logs
Engage the PSE Lab or CLARiiON support to review the BE storage. Reference emc118640 and emc36745
emc161922 "How to gather the necessary information for a CLARiiON or VNX performance analysis"
emc113166 "What to collect for Symmetrix performance issues"
Get timestamps from Arrays having issues.
The VPLEX tools used to gather performance data are "VPlexCLI monitors" and "VpPexGUI Performance dashboard"

VPlexCLI monitors
a. Perpetual monitors (gathered with the base collect-diagnostics)
b. User defined monitors: -monitor create/add-file-sink/destroy commands and Individual VPLEX object monitors(need to be
VPlexGUI performance monitor dashboard (New for GeoSynchrony 5.1)
Coming soon!!

Perpetual monitors
The perpetual performance monitors are an "always on" set of performance monitors gathering a standard set of performance
cannot be turned off by the normal user.
The perpetual monitor files are included as part of a base collect-diagnostics.
They live in /var/log/VPlex/cli/ and have the string "PERPETUAL" in their file name (ex. director-1-1-A_PERPETUAL_vplex_sys_perf_mon.log)
There is one file per director, each director's files are capped at 10MB, and up to 10 files are retained.
The data collection frequency is every 30 seconds, so roughly one to two months worth of performance data is stored in the 10 files, depend
upon how busy the cluster happens to be
The perpetual monitors are recorded locally to a cluster. When collecting performance information from a system, it's impor
to grab a collect-diagnostics from both cluster's management stations.

NOTE: NDU-ing to any version of 5.0 requires that the management station VPlexCli process be restarted, in order for the perpetual monitors
begin
running Execute "sudo /etc/init.d/VPlexManagementConsole restart" on each cluster's management station.
User defined monitors:
Read the VPLEX CLI guided for more information on setting up user defined monitor
VPLEX collects and displays performance statistics using two user-defined objects:

monitors - Gather the specified statistic.

monitor sinks - Direct the output to the desired destination. Monitor sinks include the console, a file, SNMP, or any combination of th
three.

http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=Def... 2013/4/21

3/3(W)

Creating a monitor includes the following general steps:

1. Determine the type of statistic to collect from the target object.
2. Use the monitor create command to create a monitor, its parameters and the type of statistics to collect and report.
3. Use the monitor add-sink commands to add one or more sinks to the monitor to direct the monitor ofs output.
4. Use the monitor stat-list command followed by the <Tab> key to display the statistics sub-categories.
Tech Support will be happy with the Perpetual Monitors but we may request additional monitors depending on the
Here is a Working Example of setting up user defined monitors
1. Create monitors on each director.
VPlexcli:/> monitor create --name director-1-1-A_stats --period 10s --director director
stats director.*,cache.*,directory.*,fe-director.*,storage-volume.write-latency,storage
latency
2. Add a file sink to the monitor using the following command:
VPlexcli:/> monitor add-file-sink -m director-1-1-A_director-1-1-A_stats/ f /var/log/VPLEX/cli/director-1-1-A_stats.CSV
3. Log onto the management server and check/scp the output files.

Support Center:

re

4. Repeat step 1-3 for all directors in the cluster.

service@ManagementServer:/var/log/VPLEX/cli> ls -al director-1-1-A_stats.txt

-rw-r--r-- 1 service users 2462 2010-06-24 18:42 director-1-1-A_stats.txt

Fix:

This is on Wiki and http://speed.corp.emc.com

te

For Troubleshooting please review the "VPLEX Performance Troubleshooting Guide (vspeed edition) drafted Feb 2012"

is

Issue affecting performance can be Unaligned and subpage wites , Copy Operations for Data Mobility and Mirror
Synchronization/Resynchronization , Multi-pathing , Array cache state,VPLEX cache Switch ISL hops,Switch buffer credits,Switch FC negotiatio
will often not auto-negotiate the highest speed)

1. Host to VPLEX target front-end performance - The front-end fabric

2. 2.VPLEX initiator back-end to storage-array performance - The back-end fabric
3. For Metro/Geo, VPLEX inter-cluster communication performance - The FC WAN fabric forMetro-FC, and the IP WAN network for Metr
and Geo

nR

Notes:

eg

The three main areas in the VPLEX data path are:

A bottleneck in any one of these three areas will show itself to the host application as a performance problem. It is importan
know where to focus your troubleshooting efforts.
EMC Confidential

Notes:

http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=Def... 2013/4/21