The Problem

This week I had to connect an additional NetApp Storage System to my existing UCS environment through a different path than a similar shared storage platform that we utilize here. This shouldn’t be a big deal but there were a few caveats:

• The Storage System was attached to a dedicated Nexus 5k for this customer
• The VLAN configured on the customer switch collided with one configured in UCS so VLAN translation was necessary through access ports.
• The traffic takes a different switching path from normal NFS traffic in this environment

I configured everything as one normally does when connecting to IP storage, jumbo frames and all. There was only one problem:

~ # vmkping -s 1400 -d 10.0.0.253
PING 10.0.0.253 (10.0.0.253): 1400 data bytes
2508 bytes from 10.0.0.253: icmp_seq=0 ttl=255 time=0.232 ms
2508 bytes from 10.0.0.253: icmp_seq=1 ttl=255 time=0.198 ms
2508 bytes from 10.0.0.253: icmp_seq=2 ttl=255 time=0.265 ms

--- 10.0.0.253 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.198/0.232/0.265 ms
~ #

~ # vmkping -s 2500 -d 10.0.0.253
PING 10.0.0.253 (10.0.0.253): 2500 data bytes

--- 10.0.0.253 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ #

DOH!

Now, normally this is just a simple issue of the MTU being set incorrectly somewhere along the traffic path but as I dug deeper into this issue that turned out to not be the case, it was something much stranger and more interesting. The IPs, VLANs and hostnames have been changed or obscured to protect the innocent. Onward!