On Oct 24, 2020, at 10:16 AM, Harald Welte <laforge@xxxxxxxxxxxx> wrote:
> On Fri, Oct 23, 2020 at 10:21:36AM -0700, Guy Harris wrote:
>> On Oct 17, 2020, at 7:25 AM, Harald Welte <laforge@xxxxxxxxxxxx> wrote:
>>
>>> I'm currently facing a problem where I need to create pcap files
>>
>> (For the third of those, Michael Richardson has been talking about other changes to support faster writing of capture files, including support for asynchronous I/O.)
>
> sidenote: I've just been doing some development work with io_uring
> (using liburing) on modern Linux systems, and it's amazing in terms of
> performance of asynchronous I/O. Might be worth investigating.
Might be, although we'd need to either:
1) figure out a way to do that, while hiding the platform-dependent details, on *all* (currently living) platforms supported by libpcap:
macOS;
the *BSDs;
Solaris;
HP-UX;
AIX;
Windows;
Linux;
which don't all have the same asynchronous I/O mechanisms (POSIX aio on most if not all of the UN*Xes, "overlapped I/O" or whatever Microsoft calls it on Windows)
or
2) arrange that the user may, but need not, provide their own low-level writing code that the new writing APIs call, so they can either use a platform-independent mechanism supplied by libpcap or write their own code. I think Michael Richardson has been thinking of something such as that.
>>> Furthermore, when starting a cooked Linux capture on the Linux 'any' device,
>>> it also appears wireshark is not displaying the information about which netdevice
>>> the message was captured.
>>
>> Yes. Fixing that would be part of the "get additional information when capturing, to put into pcapng files" item above.
>
> Now I'm a bit confused. You indicated libpcap doesn't have those APIs
> yet, but dumpcap uses libwirestap. Now you say that wireshark (which,
> AFAIK, is using dumpcap for captures) suffers from that constraint.
dumpcap uses libpcap for *capturing* and libwritecap (not libwiretap) for *writing capture files*.
>> Currently, dumpcap doesn't work around the absence of support for that in libpcap; it probably *could* do so.
>
> So dumpcap uses libwiretap, and libwiretap uses libpcap? I should go and
> read the code.
No - see previous comment.
What *might* be possible to do, in the absence of new libpcap capture APIs, would be to have dumpcap, when capturing from the "any" device on Linux and writing to a pcapng file:
when the capture starts, write out Interface Description Blocks (IDBs) for all the currently-known interfaces on the system, and make a table mapping from the kernel's interface indices (ifIndexes, in SNMP terms) to interface IDs for those IDBs;
when a packet arrives, look up its interface index of the packet, and:
if it's found, write the packet out with that interface index;
if it's *not* found, write out an IDB for the new interface, add it to the table, and write the packet out with that interface index.
>> It can, although you can also do memory-mapped capture, which is what current versions of libpcap do. The information in question is provided by memory-mapped captures as well.
>
> Thanks for confirming the latter. I was aware of mmap'ed AF_PCAP,
(PF_PACKET/AF_PACKET)
> just not sure libpcap is actually using it.
Yes. The master branch of libpcap *only* uses it - it dropped support for non-memory-mapped capture on Linux.