10GigEthr-02 (iexgbe) B.11.31.

1303
Ethernet Driver Release Notes (Edition 2
update)
HP-UX 11i v3

Abstract
This document contains specific information that is intended for users of this HP product.

HP Part Number: 5900-2658

Published: March 2013
Edition: 2

 Copyright 2013 Hewlett-Packard Development Company, L.P.

Legal Notices
Hewlett-Packard makes no warranty of any kind with regard to this document, including, but not limited to, the implied warranties of merchantability
and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or
consequential damages in connection with the furnishing, performance, or use of this material.
Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial
Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under
vendor's standard commercial license.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall
not be liable for technical or editorial errors or omissions contained herein.
Trademark Acknowledgments
Unix is a registered trademark of The Open Group.
Revision history

Document
Edition number
manufacturing part
number

Publication date Changes made to content

5900-2658

Edition 2

March 2013

Added defect fixes originally documented as fixed in Web 1301

release; the Web 1301 release was replaced by the equivalent AR
1303 release.

5900-2500

Edition 1

March 2013

Performance enhancements and documentation of fixes introduced in

the AR1303 release and known problems found in this and earlier
releases.

1 10GigEthr-02 overview
The 10GigEthr-02 bundle contains the PCIe 10 Gigabit Ethernet iexgbe driver. This driver supports
the following 10 Gigabit Ethernet products and LOMs:
PCIe 10 Gigabit Standup Ethernet
adapters

PCIe 10 Gigabit Mezzanine Ethernet

adapters

HP Integrity Server LOM

AM225A 10GbE-SR (fibre) adapter

NC532m (part number 467799-B21)

BL860c i2

AM232A 10GbE-LR (fibre) adapter

BL870c i2

AM233A 10GbE-CR (copper) adapter

BL890c i2
Integrity Superdome 2

NOTE: The HP-UX ioscan utility shows AM225A, AM232A and AM233A adapters all as
AM225A. This is because the only difference between the adapters is the type of SFP module
that is used. Since the adapters are functionally equivalent at the operating system level, ioscan
does not distinguish between them.
For more information about installing the supported PCIe 10 Gigabit Ethernet and mezzanine
adapters, see the following documents:

HP-UX and Windows OS Installation Guide for HP NC532m Dual Port Flex-10 10Gb Ethernet
Multifunction BL-c Adapter

HP PCIe 2 port 10Gb Ethernet-SR, LR, and CR Adapter Installation Guide

These documents can be found at the following website:

http://www.hp.com/go/10-gigabit-ethernet-docs
For more information about HP Integrity Server LOM installation, see the respective server installation
documentation at the following locations:
www.hp.com/go/blades-docs
www.hp.com/go/integrity_servers-docs
For more information about firmware requirements and how to obtain the latest firmware for your
product, see Firmware requirements (page 12).

1.1 Terminology
The following terms are used in this document:
Adapter
The whole card.
AVIO

Accelerated Virtual Input/Output. An I/O protocol that improves virtual

I/O performance for network and storage devices used within the HP
Integrity VM environment.

Card

The adapter, most often in reference to the hardware or to a specific

form factor (such as PCIe mezzanine).

Device

The adapter as a whole, or an HP-UX device.

DIO

The direct I/O networking feature that allows virtual machines to

directly control I/O devices, minimizing device emulation overhead
incurred with AVIO.

Flex-10

HP Flex-10 allows each 10 Gigabit Ethernet adapter to be configured

by HP-UX as eight LAN devices. Each port in a dual-ported mezzanine

1 10GigEthr-02 overview

or LOM represents a single, wired connection to network hardware

as follows:

In a Blade Virtual Connect environment, the NIC is configured in

Flex-10 mode. Each port is logically divided into up to four
individual network devices which share the total bandwidth of the
network connection. For more information about HP Virtual
Connect, see http://h18004.www1.hp.com/products/blades/
virtualconnect/index.html.

In other environments, the NIC is configured as a single network

device per port.

Function

A PCIe function. Each function can be configured by HP-UX as a single

device. In a Flex-10 environment, HP-UX can configure a dual-ported
NIC with up to 8 PCIe functions (up to four per port).

LOM

LAN on motherboard. A chip or chipset capable of network connections

that has been embedded directly on the motherboard of a server. For
example, server blades include LOMs that are dual-port 1 or 10
Gigabit Ethernet interfaces.

mezzanine adapter

Also known as a daughterboard or daughtercard. Plugs directly into

the motherboard or another plug-in adapter to extend functionality. It
usually fits on top of and parallel to the board or adapter it plugs into.

NIC

Network Interface Card. A function or device that is configured as a

network or LAN device.

PCIe

Peripheral Component Interconnect Express. A computer expansion

bus standard that replaces older (such as PCI) bus standards to
improvements such as a higher maximum system bus throughput, a
smaller physical footprint, better performance scaling for bus devices,
more detailed error detection and reporting, and native hot-plug
functionality. More recent revisions of the PCIe standard support
hardware I/O virtualization.

Port

The physical connection to the network. The NIC typically has two
physical connections; this configuration is often described as
dual-ported.

1.2 Features
The iexgbe driver is a PCIe 10 Gigabit Ethernet driver that can support the standup, mezzanine,
and LAN on motherboard (LOM) technologies for a variety of platforms. It supports the HP Virtual
Connect Flex-10 interface on server blade platforms. Before HP Virtual Connect was introduced,
only two interconnect choices were available for connecting server blades to a network
passthrough devices and switches. Passthrough devices are simple but require too many cumbersome
cables and create complexity. Blade switches reduce the number of cables but add more
management responsibilities for LAN and SAN administrators. In both cases, multiple people are
needed to perform very simple server tasks. Only HP offers the third choice HP Virtual Connect
which reduces cables without adding switches to manage and provides many other advantages.
For more information about HP Virtual Connect, see the following website:
http://h18004.www1.hp.com/products/blades/virtualconnect/index.html
Other notable features provided by the iexgbe driver include:

Transmit Checksum Offload (CKO) and TCP Segmentation Offload (TSO) for IPv4

Receive CKO for IPv4

Flex-10 bandwidth allocation

Multiple receive queues

Receive Side Scaling (RSS) with TCP and UDP

TCP segment reassembly in driver

Jumbo Frames

Multicast and Promiscuous mode

IPv6

IEEE 802.1Q VLAN tagging and stripping in hardware

Interrupt migration

Interrupt coalescing

Link handling

DLKM

APA aggregate mode1

APA failover mode (LAN Monitor)

HP Serviceguard

PCI OLRAD

PCI Error handling and recovery

HP Integrity Virtual Machines (HPVM)

vPars V6

DIO (direct I/O)

HP-UX LAN provider

MIB and driver statistics

Configurable using SMH or nwmgr

nettl tracing/logging

2 New and changed features in this release

2.1 New features
This release of 10GigEthr-02 introduces the following new features:

HP Smart Update Manager (HP SUM) support for online updating of I/O adapter firmware
HP SUM is the SPP deployment tool. It can be used to perform online updates of the I/O
adapter firmware on Integrity server systems running HP-UX. The 10GigEthr-02 driver is now
compatible with HP SUM, allowing online update of the firmware on supported I/O adapters.
The nwmgr firmware update command displays appropriate advisory messages describing
the online update. For example:
$ nwmgr -s --hp -A fw_dwnld=firmware_file.frm -c lan12
Firmware file: firmware_file.frm
Firmware file loaded and validated successfully
Proceed with update (Y if so)? Y
Please do not interrupt until complete (approximately 2-3 minutes)
Downloading firmware image to flash ...

1. Only LACP mode is supported with APA aggregation with 10GbE links. APA aggregate mode is not supported with
server blades over Virtual Connect interconnects. Virtual Connect does not support aggregates (trunks), LACP, or
otherwise on the downlinks side.

2 New and changed features in this release

Firmware update completed

Updated firmware will be active after a system reboot

Boot time reduced significantly for configurations requiring that multiple parameters be set
(QXCR1001149267)
The boot time configuration delay when multiple parameters are being set is now reduced
significantly. Prior to this release, system boot time could be impacted significantly by the
number of parameters being set in the configuration file, especially in system configurations
supporting a large number of iexgbe LAN devices. Each configuration parameter was set
using independent calls to the driver. Additional delays would result if driver resets were
required for the settings to take effect. With this release of the driver, a single call is used to
set multiple parameters, reducing the delay caused by multiple resets.
A new command line option has been added to the lanadmin command to provide this
enhancement:
lanadmin -X boot_parameter [{drv_pr_on | drv_pr_off}] [mtu=size] [{recv_cko_on | recv_cko_off}] [{send_cko_on
| send_cko_off}] [drv_mq=queue-number] [vmtu={0 | 32160}] ppa

For example:
/usr/sbin/lanadmin -X boot_param send_cko_on drv_mq=4 vmtu=32160
recv_cko_on drv_pr_off mtu=1500 9

2.2 Changed features

The iexgbe driver has been enhanced with performance and stability improvements.

2.2.1 Changes introduced with 10GigEthr-02 B.11.31.1205

Tx and Rx flow control are now enabled by default. This flow control feature (which is optional)
has certain limitations documented in Section 4 (page 10).
Table 1 (page 7) lists the Rx and Tx flow control settings and defaults, as determined by the MTU
setting.

Table 1 10GigEthr-02 flow control and Jumbo Frames support matrix

MTU

Rx flow control

Tx flow control

1500 (non-Flex-10 ports)

ON/OFF

ON/OFF

Default setting during driver

initialization before startup config
parameters are applied will be: ON

Default setting during driver

initialization before startup
configuration parameters are applied
will be: ON

9000 (non-Flex-10 ports)

OFF only.

ON/OFF
Default setting during driver
NOTE:
initialization before startup
1. If previous value is ON, setting
configuration parameters are applied
MTU to 9000 automatically sets Rx will be: ON
flow control to OFF.
2. If the MTU value is changed back
to 1500, the original Rx flow
control setting (rx_fctrl) prior
to changing to 9000 MTU is
restored.

1500 (Flex-10 ports)

Only OFF

Only ON

9000 (Flex-10 ports)

Only OFF

Only ON

For non-Flex-10 interfaces, Tx and Rx flow control will be enabled by default. Setting Jumbo Frames
MTU disables Rx flow control; Rx flow control cannot be enabled if the interface has a Jumbo
Frames MTU.
For Flex-10 interfaces, Tx flow control is always enabled and Rx flow control is always disabled.
Table 2 (page 7) lists new configuration file parameters for non-Flex-10 interfaces. These
parameters are added to the configuration file /etc/rc.config.d/hpiexgbeconf at installation
if the file has not been previously modified by the customer. If the file has been previously modified,
the configuration file is not automatically modified with the new parameters, and the configuration
file must be updated manually. The new version will be available in file
usr/newconfig/etc/rc.config.d/hpiexgbeconf.
Table 2 Configuration file parameters for non-Flex-10 interfaces1
Command

Function

Valid values

Default

HP_IEXGBE_TX_FLOW_CONTROL

Enable or Disable the

adapter's Tx-side flow
control

ON/OFF

ON

HP_IEXGBE_RX_FLOW_CONTROL Enable or Disable the

adapter's Rx-side flow
control

ON/OFF

ON

For Flex-10 interfaces, the configuration parameters described in this table will fail if changed from the default settings
listed in column 4. The defaults are permanent settings.

The following warning messages have been added to the nwmgr command:

Setting MTU size greater than 1500 (enabling Jumbo Frames) for a non-Flex-10 interface
brings up the warning message shown in the following example:
server2#nwmgr -s -A mtu=9000 -c lan3
WARNING: For non-flex10 interface, setting Jumbo MTU will automatically turn OFF Rx flowcontrol
lan3 current values:
New MTU = 9000

Attempting to turn off Tx flow control for a Flex-10 interface brings up the warning message
shown in the following example:
server2#nwmgr -s -A tx_fctrl=off -c lan3

2 New and changed features in this release

WARNING: For flex10 interface, the Tx flow control will be always on and the Rx flow control will be always
off
lan3 current values:
New Transmit Flow Control = Off

NOTE: The warning message incorrectly displays the Tx flow control state as Off when it
should be On. This is a known issue (see Known problems and limitations (page 10)).

Attempting to turn on Rx flow control for a Flex-10 interface on which the MTU size is greater
than 1500 (enabling Jumbo Frames) brings up the warning message shown in the following
example. The Rx flow control cannot be turned on in this instance, so the specified value (on)
is invalid; Rx flow control remains turned off.
server2#nwmgr -s -A rx_fctrl=on -c lan3
WARNING: For flex10 interface, the Rx flow control is always on
For non-flex10 interface, if Jumbo MTU is set, the rx flow control cannot be turned on
lan3 current values:
New Receive Flow Control = ERROR: Invalid value

3 Problems fixed in this release

The following defects were fixed in this release:
Table 3 HP-UX 11i v3 January/March 2013 fixes 10GigEthr-02 B.11.31.1301 and B.11.31.1303
SR number
QXCR1001233679

Description
Link status is incorrectly reported for LAN ports connected to a switch or pass-through
When LAN ports are connected to a switch or pass-through, the iexgbe driver might fail to
detect a cable connect or disconnect. As a result, the LAN interface link status on the system
side is incorrectly displayed by nwmgr and lanadmin. For example, after a cable disconnect,
the status is incorrectly reported as UP instead of DOWN.

QXCR1001222146

Port unavailable, queues fail to initialize on Superdome

While doing stress testing with heavy traffic along with periodic port resets, olrad, and
ifconfig plumb/unplumb, the port might be unable to initialize the maximum number of
queues supported. In such cases for drivers with versions AR1209 and earlier, the port
transitions to the DEAD state. To make the port usable, a system reboot is required. The issue
is rarely reproducible; reproducing the issue requires the above-mentioned stress testing
sustained over a long duration.

Problems fixed in recent versions

Table 4 HP-UX 11i v3 September 2012 fixes 10GigEthr-02 B.11.31.1209
SR number
QXCR1001202586

Description
The host encountered a rare hang situation as the intctl command gets stuck within the
kernel after repeated interrupt migrations are run on the host backing interface. The cause is
the driver failing to perform an appropriate handshake with the kernel module responsible
for completing the interrupt migration event under certain rare circumstances.

Table 5 HP-UX 11i v3 May 2012 Fixes 10GigEthr-02 B.11.31.1205

SR number

Description

QXCR1001209171

AVIO: High outbound discards on HPVM guest with iexgbe interface. TCP connections (such
as rlogin and telnet) from a remote system to the guest do not work and the guest hangs
intermittently.

QXCR1001182096

When packets with an invalid LLC, IP or TCP header length are sent through the HPVM guest
igssn interface, the iexgbe driver will drop such packets. But the guest is not aware of
such drops and it continues to expect send completion for those packets. This situation results
in a gradual reduction of the number of usable descriptors for igssn, which causes
performance to degrade and, eventually, the igssn interface to hang.

QXCR1001192266

For non-Flex-10 (non-VC) interfaces, when using customized iexgbe configuration file settings
that enable Jumbo Frames (MTU size greater than 1500), problems occur at startup. This
problem surfaced after Tx and Rx flow control support was introduced to the iexgbe driver
(QXCR1001093086). The workaround in the preceding release was to disable flow control
by default. The B.11.31.1205 release sets Tx and Rx flow control to ON by default. Any
request to turn on Jumbo Frames will override the Rx flow control settings and automatically
set Rx flow control to OFF.

Table 6 HP-UX 11i v3 March 2012 Fixes 10GigEthr-02 B.11.31.1203.01

SR number
QXCR1001192266

Description
Disables flow control by default. This removes the possibility of bringing up issues introduced
with the B.11.31.1201 release (QXCR1001093086). For more information about the issues,
see Section 4 (page 10).

3 Problems fixed in this release

4 Known problems and limitations

The following problems and limitations are known to exist in version B.11.31.1303:

QXCR1001270195 iexgbe 1303 versions introduced issues due to missing firmware lock
The following symptoms might be seen because of this defect:

iexgbe driver continuously sends UP/DOWN notification events to the WBEM provider.
(QXCR1001268673) The WBEM events seen by the user are of these types:
DL_HP_EXT_LINK_DOWN_IND and DL_HP_EXT_LINK_UP_IND.

(QXCR1001267062) The nettl log becomes filled with link EPHY8073 link up
notification messages.
The following is an example of the messages seen:
<4006> 10GBase-SR in path 0/0/0/3/0/0/5
Link Event.
EPHY8073 link up or
<4006> 10GBase-SR in path 0/0/0/3/0/0/5
Link Event.
EPHY8726 link up

(QXCR1001268375) Speed mismatch error frequently recorded in syslog.

The driver can present the following message in syslog during DLKM load/unload,
boot, or other control path operations:
internal_link_settings_status: 10G speed mismatch 0x300

(QXCR1001268375) Ports for which the adapter reports the incorrect speed might
occasionally cause APA formation to fail during startup.

iexgbe m45write and m45read message logged on the console.

(QXCR1001267051) During DLKM load/unload, the following message might be seen
on the console:
...m45write: Setting PHY address to write failed, phyreg 0xffe4, devad 0x7, val 0x1e0
...m45read: Setting PHY address to read failed 0x2227ffe4 0x1

(QXCR1001265402) During installation or deinstallation of the 10GigEthr-02 bundle,

the following message might be seen:
...m45read: Read of PHY register failed, addr 0x2227ffe4 init_done 1,
status 0x2e270000

QXCR1001267697 Problem with nwmgr -s -A all --sa --fr cu -c command

The command does not properly store the parameter settings in the
/etc/rc.config.d/hpiexgbeconf configuration file.

QXCR1001247823 iexgbe driver intermittently stops collecting/incrementing link statistics

The iexgbe driver intermittently stops collecting and updating link statistics. This does not
affect data traffic; it only affects statistics. However, this can cause false link failure detection;
systems with MC/Serviceguard might exhibit a system Transfer Of Control (TOC), thereby
inhibiting Serviceguard recovery from a link outage or network connectivity problem.
Severity: Medium
Workaround: Resolve the problem by resetting the link. Statistics will then be collected and
updated properly.

10

Incorrect warning message after attempt to turn off Tx flow control for a Flex-10 interface
When an attempt is made to turn off Tx flow control for a Flex-10 interface, a warning message
indicates correctly that Tx flow control always stays on for Flex-10 interfaces; however, the
warning message displays the current value of Tx flow control as Off instead of On.
Severity: Minor
Workaround: Disregard the displayed state of Tx flow control.

QXCR1001265320 The iexgbe driver might cause TCP/IP startup to fail.

During system boot, the system might fail when bringing up TCP/IP. An error message such
as "route add net : Network is Unreachable" is seen.
Severity: Medium
Workaround: Include a waitfor $(driverState) line in the init script
/sbin/init.d/hpiexgbe, as shown in the following example (line 407):
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415

#echo "$boot_cmdstr"
echo "${HPLANADMIN} -X boot_param $boot_cmdstr $PPA"
emsg=`${HPLANADMIN} -X boot_param $boot_cmdstr $PPA 2>&1`
if [[ $? -ne 0 ]]; then
errmsg; rval=${ERROR}
fi
waitfor $(driverState)

# SAM script would have attempted a MTU change earlier; this is 2nd try
#configure_mtu
#if (( $mturet != 0 )); then
#
errmsg # emsg was set by configure_mtu
#
rval=$ERROR
#
fi

Rx flow control must be disabled when Jumbo Frames are used.

Due to limitations of the hardware supported by the iexgbe driver, Rx flow control must be
disabled when Jumbo Frames (MTU value greater than 1500) are enabled. This limitation is
enforced by the iexgbe driver. Therefore, if the current configuration has Rx flow control
enabled while Jumbo Frames are used, the iexgbe driver will not honor changes made to
flow control settings. Setting Jumbo Frames will override and disable the Rx flow control
settings.
NOTE:

For the Flex-10 interface, Rx flow control is always set to OFF.

For the non-Flex-10 interface, you can change the flow control and MTU size as needed.
However, when Jumbo Frames are enabled, if Rx flow control is already set to ON, the driver
automatically resets RX flow control to OFF. Rx flow control and Jumbo Frames cannot both
be in operation at the same time.
Severity: Medium
Workaround: If you want Rx flow control, disable Jumbo Frames.

Rx flow control is disabled in Flex-10 mode

In Flex-10 mode (Virtual Connect), Rx Flow control is disabled by default. Attempts to enable
Rx flow control are not honored by the driver, because the driver and hardware cannot
determine whether Jumbo Frames are disabled across all FlexNICs.
Severity: Medium
Workaround: No workaround necessary.

Using ncweb to change number of receive queues causes an error message.

Severity: Medium

4 Known problems and limitations

11

When you use the ncweb or SMH interface to change the number of receive queues, an error
message appears.
Workaround: Use nwmgr instead of ncweb to change number of the queues. The command
syntax is:
nwmgr -s -A drv_mq=<new#> -c lan<instance>
where
<new#>

The number of queues that you want to enable.

<instance>

Instance number of the LAN interface; for example, lan1.

5 Compatibility and installation requirements

This section describes the compatibility information and installation requirements for this release.

For specific installation instructions, see the installation procedure or overview document for
your adapter.

For detailed information on supported server platforms, operating systems, and firmware level
requirements, see the Ethernet Support Matrix.

These documents are available at:

http://www.hp.com/go/10-gigabit-ethernet-docs

5.1 Operating system and version compatibility

This release is specific for HP-UX 11i v3 (B.11.31).

5.2 Hardware requirements

This version of 10GigEthr-02 runs with the HP-UX 11i v3 (B.11.31) operating system on all HP
Integrity server blades and rack mounted systems. For more information about supported server
platforms, see the Ethernet Support Matrix at:
http://www.hp.com/go/10-gigabit-ethernet-docs

5.3 Firmware requirements

IMPORTANT: To ensure full functionality, you must install the latest adapter firmware and Extensible
Firmware Interface (EFI) driver for your adapter.
The 10GbEthr-02 (iexgbe) requires the latest firmware for supported devices. For more information,
see the latest HP-UX Ethernet Card Support Matrix at:
http://www.hp.com/go/10-gigabit-ethernet-docs
To determine and download the latest adapter firmware and EFI driver for your Integrity server
product, follow these steps:
1. Go to http://www.hp.com.
2. Click the Support & Drivers link on the main page.
3. Click the Drivers & Software tab.
4.
5.
6.
7.
8.

12

Enter your product name (for example, NC532m or, for LOMs, a server name such as
Superdome 2) or enter the product number and click Search.
Click the Cross operating system (BIOS, Firmware, Diagnostics, etc.) link.
Click EFI and FWs for use on Integrity under the Description heading for the EFI download.
To review firmware versions and release notes, click the Readme tab. To download the
firmware, click the Download button.
Download the firmware.

9.

To install the firmware update, follow the procedure supplied with the downloaded update
package.

6 Related information
The latest documentation for Ethernet adapters is available in English at:
http://www.hp.com/go/10-gigabit-ethernet-docs

Installation guides are listed alphabetically in the Setup and install general section.

Release notes are listed alphabetically in the Getting started section.

Support guides are listed alphabetically in the User guide section.

Support Matrix documents are listed alphabetically in the General reference section.

7 Software availability in native languages

The 10GigEthr-02 product is available only in the English language.

8 Documentation feedback
HP is committed to providing documentation that meets your needs. To help us improve the
documentation, send any errors, suggestions, or comments to Documentation Feedback
(docsfeedback@hp.com). Include the document title and part number, version number, or the URL
when submitting your feedback.

6 Related information

13