Packet De-duplication
When connecting a network monitoring tool (aka Sniffer) to a network environment through mirror or span ports, it is very likely that duplicated packets will be captured.
The generation of duplicated packets is usually due to incorrect switch configuration. For example, both ingress and egress packets are captured, but the inter-VLAN traffic result double packets records for those packets leaving and entering both monitored VLAN.
For some older equipments like Cisco 6509 Catalyst OS, duplicated packets always mirrored/spanned to the monitor tool.
From the Ethernet level, it is hard to distinguish whether the frame is duplicated or not. Let’s focus on a level 2 packet, such as ARP request. There is neither sequence number nor packet ID to identify the differences between two packets with the same content. However, as the packet may be still correct, we cannot simply filter them, because similar ARP packets may point to an ARP storm, and if we remove that duplication, mistakes will follow closely.
Duplicated packets are a nightmare to analyzers. All of the response time, TCP round-trip, retransmission detection, and application level responses are messed up regarding on the packet duplicated condition. Therefore, for the analysis procedure, it’s critical to identify packet duplication and remove those unimportant ones.
To assess packet duplication, the best way is go to the layer 3. The Layer 3 we are talking about here actually means the IP Layer, which provides a very good field to ensure passive packet duplication. The key is the IPID field. When a machine sends IP packets, the OS will automatically add the number of IPID count and put it into the IP Packet. This number will not be changed by the inter-path routers or switches. So if the packets possess the same IPID and pattern, normally, we can confirm that they are from duplication. However, with only 2 bytes, the IPID field only has 65,536 numbers as potential candidates of the IPID value. So when it comes to a very busy server, in a single second, more than 100 K packets may be sent. Then the IPID verification must go together with the content and packet length validation.
Yet, things grow much more complicated in the real world: sometimes, duplicated packets may vary a little bit from each other. Let’s refer to such a case: A Packet received in Port A has no VLAN tag at all and when this packet is forwarded to Port B, an 802.1Q/ISL tag is tagged into this packet by the switch to adapt to Port B Vlan settings. If there are some reasons make the switch mirror both Packets at Port A and Port B to the monitor port, the monitor tool will see a duplicated packet pair with differences in length and byte-to-byte content match.
IPID still works in this case. We need to adopt all of the IP layer data to compare the content and length despite of the difference at frame level.
In another case which packets are transferred from a router to a firewall, both firewall and router are working under load balancing/dynamic routing mode. So the packet received and sent by the firewall/routers will be usually from and to different Mac address, although they are from the same equipment and even the same port. This will bring another kind of packet duplication——all the others are the same except MAC addresses. Luckily, the IPID algorithm with IP Layer byte match is still working under such circumstances.
The IPID measurement has its limitation, for example, it is difficult when handling NAT packets; and it is difficult to handle the fragmented vs. non-fragmented packets for duplication detection purpose. In some heavy loaded environments, the algorithm might result faultily. By adopting continuous detection and duplication count limit algorithm, the error rate can be easily tuned. So basically, the IPID detection works very well in most cases. Whether the IPID based packet duplication will remove the TCP retransmission? ——Definitely not. TCP retransmission is actively generated by communication hosts, so the IPID will be increased for each TCP transmission.
BoneLight implements the IPID based packet duplication as a converter. By mouse right click, a packet de-duplicated trace will be shown instantly.
See more for
Enjoy Now to download a copy of BoneLight!