[Omer Shapira <omer_shapira@xxxxxxxxx>]

Hello,

I want to start a discussion on the proper support for the packet metadata that can be collected on Darwin platforms using `tcpdump -k` flag. For disclosure, I work at Apple at the team that is responsible for all Darwin networking, including the tcpdump, but also all the transport protocols (kernel and user space), as well as routing etc. Recently I got a permission from Apple management to contribute to Wireshark. I believe this combination gives me the opportunity to fix something that has been broken for a long time.

Context:

For the background, the `-k` flag will cause Darwin’s  `tcpdump` command to collect process metadata: the name, the id, and the uuid of both the associated process, and the effective process.  The metadata can be viewed by `tcpdump`, for example:

   tcpdump -n -k -r ~/tmp/tcpdump_k_pktap.pcapng -c 2 

   reading from PCAP-NG file /Users/oesh/tmp/tcpdump_k_pktap.pcapng 

   16:30:58.065667 (en0, proc mDNSResponder:428, eproc :428, svc BE, in, so) IP 17.230.173.255.5353 > 224.0.0.251.5353: 0 TXT (QU)? TyM-bM-^@M-^Ys MacBook Pro._companion-link._tcp.local. (63)

   16:30:58.065680 (en0, proc mDNSResponder:428, eproc :428, svc BE, in, so) IP6 fe80::cb8:176a:f22c:66f9.5353 > ff02::fb.5353: 0 TXT (QU)? TyM-bM-^@M-^Ys MacBook Pro._companion-link._tcp.local. (63)

Support for the process metadata has been added around 2012-2013. This metadata is quite valuable when analyzing and optimizing traffic on mobile devices, where waking the antenna uses a lot of energy. 

The Challenge:

Unfortunately, when the support for the process metadata was added, the team missed the opportunity to do the right thing and to use CUSTOM block for the metadata, and instead used the LOCAL block id (with the MSB set). This was before my time, and as far as I can tell,  this decision was not made in spite, but due to the lack of context and the general sense of urgency.

It seems that the right thing to do is to get Darwin `tcpdump` to encode the process information in a standards-conforming fashion. I am fully supportive of this idea, and will work to make this happen. 

There is, however, an issue with the ossification of the current process metadata encoding. During the 12-13 years since the introduction of the process metadata, there are hundreds of thousands of existing pcap files that contain the metadata in the LOCAL block, and this is just in a single company. It is not realistic to expect the engineers to re-encode the “legacy” pcap files.

Because of that, I believe that while it is possible to “seal” the structure of the current process metadata block, there is no alternative but to continue supporting this, indefinitely.

I would like to mention that I am not super proud of this situation. I cannot change the past, but at least right now I am in the position where I can try to fix the situation, which is why I am making this effort.

Suggested Path Forward:

From the perspective of today, I would like to get to a situation where:

1. Darwin tcpdump uses the standards-conforming representation for the metadata when writing new files.

2. It is possible to gradually extend the metadata in a standards-conforming way.

3. Developers can use Wireshark to filter the traffic by the process metadata.

4. Developers can use Wireshark with both the “legacy” files, and the new files.

Since there are more than billion of devices running Darwin in some shape or form, and since there were more than ten years to collect potentially valuable pcap files on those devices, it is important to preserve the ability to dissect the “legacy” files. It unrealistic to expect that all the legacy files can be converted to a new representation; there is just too much of that legacy.

My understanding is that to support the “legacy” representation of Darwin metadata, Wireshark will need to start treating `0x80000001` as an ossified quasi-standard block type, and to at least attempt to decode this as Darwin process metadata.

Moreover, due to the quantity of the “legacy” pcap files, it might be a pragmatic idea to mention the block 0x80000001 as an “exception” in https://datatracker.ietf.org/doc/draft-ietf-opsawg-pcapng/ , so that future developers would skip this. 

Discussion

I would like to hear the opinions of  the core developers on the above. Again, I am not super proud that mistakes have happened, an d I am trying to fix it going forward, but please keep in mind the backwards compatibility.