How to Use mlxlink to Diagnose NVIDIA ConnectX-6 NICs

How to Use mlxlink to Diagnose NVIDIA ConnectX-6 NICs

NVIDIA’s mlxlink utility provides a detailed view of the physical and operational state of ConnectX network adapters, making it useful for validating high-speed Ethernet links and diagnosing physical-layer issues.

This guide demonstrates how to use mlxlink with a ConnectX-6 (CX-6) NIC to inspect link state, negotiated speed, lane width, FEC, physical-layer error counters, BER, and optical module diagnostics. The same methodology generally applies to other ConnectX generations, although individual fields and supported capabilities may vary by hardware and firmware.

The focus here is diagnostics and status inspection. Configuration and modification of link parameters are outside the scope of this guide.

🔎 Running mlxlink for a ConnectX-6 NIC

#

A comprehensive status report can be obtained with:

mlxlink -d mlx5_0 -m -c

The exact device identifier may differ between systems. In this example, mlx5_0 identifies the target ConnectX device.

The -m option enables module-related information, while -c requests additional cable and physical-link diagnostics.

A typical output contains several sections that can be used to evaluate the health of the NIC, link, and attached optical module.

⚙️ Operational Link Information

#

A healthy 200GbE link may report:

State                            : Active
Physical state                   : ETH_AN_FSM_ENABLE
Speed                            : 200G
Width                            : 4x
FEC                              : Standard RS-FEC (544,514)
Loopback Mode                    : No Loopback
Auto Negotiation                 : ON

These fields provide the first indication of whether the port is operational and how the physical link has been established.

State

#

State : Active

Active indicates that the port has an operational link.

If the state is not active, subsequent physical-layer fields should be examined to determine whether the problem is related to negotiation, cable/module detection, signal integrity, or the link partner.

Physical State

#

Physical state : ETH_AN_FSM_ENABLE

This indicates that Ethernet auto-negotiation is enabled at the physical layer and that the port is operating within its Ethernet Auto-Negotiation state machine.

The precise state reported by mlxlink depends on the current link configuration and negotiation process.

Speed

#

Speed : 200G

The negotiated link rate is 200 Gbps.

This field should be compared with the expected rate for the NIC, transceiver or AOC, switch port, and configured link mode.

Width

#

Width : 4x

The connection uses four physical lanes.

At 200G, this corresponds to a 4 × 50G lane configuration, producing an aggregate line rate of 200 Gbps.

FEC

#

FEC : Standard RS-FEC (544,514)

The link is using Reed-Solomon Forward Error Correction with the (544,514) coding scheme.

FEC allows certain transmission errors to be corrected at the physical layer before they become visible as uncorrected link errors.

For high-speed Ethernet, checking FEC status is particularly important when investigating signal-quality or cable-related problems.

Loopback Mode

#

Loopback Mode : No Loopback

No hardware loopback is currently enabled, so the port is communicating with an external link partner.

Auto Negotiation

#

Auto Negotiation : ON

Auto-negotiation is enabled, allowing the endpoints to negotiate compatible link capabilities.

When troubleshooting an unexpected link speed, this field should be considered alongside the enabled and supported speed fields.

📋 Supported Link and Cable Speeds

#

The supported-speed section may look like:

Enabled Link Speed (Ext.)        : 0x000017f2(200G,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G)
Supported Cable Speed (Ext.)     : 0x000017d0(200G,100G_2X,100G_4X,50G_1X,50G_2X,25G,10G)

Two different concepts are represented here: what the NIC is configured to support and what the attached cable or module advertises.

Enabled Link Speed

#

Enabled Link Speed (Ext.) lists the link rates enabled by the NIC.

In this example, the adapter supports multiple operating modes, including:

• 200G
• 100G × 2
• 100G × 4
• 50G × 1
• 50G × 2
• 40G
• 25G
• 10G
• 1G

The presence of a mode in this field does not necessarily mean that the current link is operating at that rate. The actual negotiated rate is shown separately under Speed.

Supported Cable Speed

#

Supported Cable Speed (Ext.) describes the speeds supported by the attached cable or optical module.

This distinction is useful during troubleshooting. A NIC may support a particular speed while the connected module, cable, or remote endpoint does not.

🩺 Troubleshooting Status

#

The troubleshooting section provides a high-level interpretation of the current link state:

Status Opcode                    : 0
Group  Opcode                    : N/A
Recommendation                   : No issue was observed.

Status Opcode

#

Status Opcode : 0

A status opcode of 0 indicates that mlxlink has not identified an active error condition requiring attention.

Recommendation

#

Recommendation : No issue was observed.

This is a useful high-level health indicator, but it should not replace examination of the physical counters and module diagnostics when investigating intermittent problems.

📊 Physical Counters and BER

#

The physical diagnostics provide more granular information:

Time Since Last Clear [Min]      : 18285.8
Effective Physical Errors        : 0
Effective Physical BER           : 4E-15
Raw Physical BER                 : 1E-10
Raw Physical Errors Per Lane     : 2667124,2619081,190455,106349

These counters are particularly important for high-speed links because a port can remain operational while accumulating physical-layer errors that may eventually affect reliability.

Time Since Last Clear

#

Time Since Last Clear [Min] : 18285.8

This indicates how long the physical counters have been accumulating since their last reset or clear operation.

Counter values should therefore always be interpreted together with this time interval.

Effective Physical Errors

#

Effective Physical Errors : 0

A value of zero indicates that no uncorrected physical-layer errors have been recorded in the reported interval.

This is a strong indicator that FEC is successfully handling the observed raw errors without allowing them to propagate into uncorrected link errors.

Effective Physical BER

#

Effective Physical BER : 4E-15

The effective BER after correction is extremely low.

For operational troubleshooting, the important point is that the link has not accumulated uncorrected physical errors despite the presence of raw errors at the lane level.

Raw Physical BER

#

Raw Physical BER : 1E-10

Raw BER represents the observed bit error rate before FEC correction.

A non-zero raw BER does not automatically indicate a failing link. High-speed Ethernet systems are designed to tolerate and correct a certain level of physical-layer errors through FEC.

The trend over time is often more informative than a single snapshot.

Raw Physical Errors Per Lane

#

Raw Physical Errors Per Lane : 2667124,2619081,190455,106349

The values represent raw physical errors observed on each lane.

The first two lanes have accumulated substantially more raw errors than the other two lanes in this example. However, because Effective Physical Errors remains zero, those errors have been successfully corrected.

For troubleshooting, an increasing error rate concentrated on one lane can be particularly useful because it may point toward a lane-specific signal-integrity problem.

🔌 Optical Module Diagnostics

#

The module section provides information about the attached cable or transceiver:

Identifier                       : QSFP28
Compliance                       : Active Optical Cable with 50GAUI,100GAUI-2 or 200GAUI-4 C2M. Providing a worst BER of 10^(-6) or below
Cable Technology                 : 850 nm VCSEL
Cable Type                       : Active cable(active copper / optics)
OUI                              : Mellanox
Vendor Name                      : Mellanox
Vendor Part Number               : MFS1S00-H010E
Vendor Serial Number             : MT2007FT07419
Rev                              : A7
Attenuation(5g,7g,12g) [dB]      : N/A
FW Version                       : 37.50.322
Wavelength[nm]                   : 850
Transfer Distance [m]            : 10
Digital Diagnostic Monitoring    : Yes
Power Class                      : 5.0 W max
CDR RX                           : ON,ON,ON,ON
CDR TX                           : ON,ON,ON,ON
LOS Alarm                        : N/A
Temperature[C]                   : 62 [-10..80]
Voltage [mV]                     : 3270[3100..3500]
Bias Current[mA]                 : 7.396,7.392,7.4,7.4[5.492..8.5]
Rx Power Current[dBm]            : 0,0,0,0 [-14..6]
Tx Power Current [dBm]           : 0,0,0,0 [-12..6]

This section is especially useful when diagnosing issues that may originate outside the NIC itself.

Module Identifier and Compliance

#

Identifier : QSFP28

The attached module identifies itself as a QSFP28-class device.

The compliance field indicates support for high-speed Ethernet interfaces including 50GAUI, 100GAUI-2, and 200GAUI-4 configurations.

Cable Technology

#

Cable Technology : 850 nm VCSEL

The module uses 850 nm Vertical-Cavity Surface-Emitting Laser technology.

VCSEL-based optical technology is commonly used for short-reach data-center connectivity.

Vendor Information

#

The reported vendor information identifies the module as Mellanox:

Vendor Name        : Mellanox
Vendor Part Number : MFS1S00-H010E
Rev                : A7

Mellanox became part of NVIDIA, so older Mellanox-branded ConnectX hardware and optical accessories remain common in NVIDIA networking deployments.

The part number and revision are useful when checking compatibility, firmware requirements, or known hardware-specific behavior.

Firmware Version

#

FW Version : 37.50.322

The reported firmware version provides a reference point when comparing behavior across systems or investigating compatibility issues.

When diagnosing production problems, recording the firmware version is important because ConnectX behavior can depend on both NIC firmware and the software stack installed on the host.

Transfer Distance

#

Transfer Distance [m] : 10

The attached active optical cable is rated for a reach of up to 10 meters.

Cable length and optical specifications should always be considered together with the negotiated link speed and lane configuration.

🌡️ Module Health and Electrical Diagnostics

#

Several fields provide real-time operating information for the module.

Digital Diagnostic Monitoring

#

Digital Diagnostic Monitoring : Yes

Digital Diagnostic Monitoring (DDM) allows operational parameters such as temperature, voltage, and laser bias current to be monitored.

This is valuable for identifying thermal or electrical abnormalities before they develop into link failures.

Temperature

#

Temperature[C] : 62 [-10..80]

The reported temperature is 62°C, within the module’s specified operating range of -10°C to 80°C.

Although the module is operating within specification, 62°C is relatively high compared with a lower-temperature baseline. A sustained upward trend may warrant investigation of airflow, chassis temperature, or cable placement.

Voltage

#

Voltage [mV] : 3270[3100..3500]

The module is operating at 3.27V, within the reported 3.1V–3.5V range.

This indicates that the module’s supply voltage is currently within specification.

Bias Current

#

Bias Current[mA] : 7.396,7.392,7.4,7.4[5.492..8.5]

The four channels report approximately 7.4 mA, within the specified 5.492–8.5 mA range.

Monitoring this value over time can help identify degradation or abnormal behavior in optical transmit components.

Receive and Transmit Power

#

Rx Power Current[dBm] : 0,0,0,0 [-14..6]
Tx Power Current [dBm] : 0,0,0,0 [-12..6]

The reported values are within the displayed operating ranges.

However, optical power fields should be interpreted according to the specific module implementation and mlxlink firmware output. A zero reading does not necessarily imply that the optical signal itself is absent, particularly for an active cable implementation.

🧪 How to Interpret the Complete Output

#

A single mlxlink field rarely provides enough information to diagnose a high-speed Ethernet problem.

A more reliable approach is to correlate several groups of measurements:

1. Check State to confirm that the link is operational.
2. Check Speed and Width to verify the expected link configuration.
3. Check FEC to confirm that the expected correction mechanism is active.
4. Compare enabled and supported speeds to identify capability mismatches.
5. Review Effective Physical Errors for uncorrected physical-layer failures.
6. Review raw BER and per-lane errors for emerging signal-integrity problems.
7. Inspect module temperature and voltage for environmental or electrical abnormalities.
8. Verify cable and module identity when compatibility or hardware issues are suspected.
9. Compare counters over time rather than relying exclusively on a single snapshot.

This layered approach is particularly useful for 100G and 200G Ethernet links, where FEC can mask underlying physical-layer errors until conditions deteriorate.

🛠️ Practical Troubleshooting Workflow

#

When a ConnectX link is unstable, unexpectedly slow, or repeatedly dropping, mlxlink can be used as part of a systematic diagnostic workflow.

First, establish whether the problem is a link-state problem or a physical-quality problem. If the port is inactive, inspect negotiation, supported speeds, module detection, and the remote endpoint.

If the link is active but experiencing errors, examine the effective and raw BER values and compare error counts across lanes.

A rapidly increasing error count on one lane is more suspicious than evenly distributed low-level errors. Temperature should also be monitored because optical modules operating near their thermal limits can experience degraded signal characteristics.

The module part number is another important diagnostic variable. If replacing the cable resolves the issue while the NIC configuration remains unchanged, the original module or cable becomes a strong candidate for the root cause.

For persistent production incidents, capture mlxlink output periodically so that changes in BER, temperature, bias current, and other counters can be correlated with traffic conditions and link events.

✅ Summary

#

The example ConnectX-6 link is operating normally at 200 Gbps using a 4 × 50G lane configuration.

The diagnostic output shows:

• 200G link speed
• 4-lane physical configuration
• RS-FEC enabled
• Auto-negotiation enabled
• No uncorrected physical errors
• Extremely low effective BER
• A 10-meter Mellanox active optical cable
• 850 nm VCSEL technology
• DDM support
• Module temperature and voltage within specification

The most important lesson is that mlxlink provides considerably more information than a simple link-up or link-down status. Its combination of negotiated link parameters, FEC statistics, BER counters, per-lane errors, and module telemetry makes it a useful low-level diagnostic tool for NVIDIA ConnectX networking hardware.

For high-speed Ethernet deployments, regularly checking these physical-layer indicators can help identify signal-quality degradation before it develops into packet loss, link flapping, or complete connectivity failure.