["Loris Degioanni" <loris@xxxxxxxxxxxxxxxxxxxxxxx>]

Hi.
I am quite astonished by the interest aroused by our tests a year after their publication on the web. We (the Italians, i.e. the network group of the Politecnico di Torino) performed these tests in November 1999 in order to obtain quantitative results for my graduation thesis that was focused on the development of WinPcap. Since, as far as I know, BPF is the best public capture device in the Unix world, we used the latest freebsd distribution of that period (3.3) and compared the results obtained. Since then, we never updated that page. I know that freebsd and its BPF implementation have been updated meanwhile, but winpcap as well has been improved and optimized. This means that the results are outdated, but the comparison is acceptable, because it takes in consideration the best implementations of that period. FYI, we have done other tests in September 2000 comparing the new WinPcap 2.1 with BPF under freebsd 4.1, and the results were quite interesting, but I will speak of this later.
Our goal was to create a suite of tests able to:

- measure the performance of the various components (although this is difficult because of the impossibility to isolate each component from  interacting one with the others and with the OS).
- measure capture performance "as a whole", from the NIC to the hard disk.

I know perfectly that the second point involves measuring the performance of the whole operating system, and not only of the capture driver, but we included it because (as we wrote commenting the tests) in our opinion it is the most interesting from the end-user standpoint. I know also that better capture performance can be obtained tuning the OS (eliminating the unnecessary protocol stacks, setting bigger buffers, choosing a fast file system for the dumps, etc), but we were interested in measuring the usage that the standard tcpdump/windump user makes of these tools, so we installed a fresh win2k professional and a fresh freebsd on the same disk of a P2 400 and launched the tests without modifying anything.
In my opinion, those tests demonstrate that:

- BPF for freebsd 3.3 is more efficient in the CPU usage. On the other side, winpcap 2.02 is more responsive, because it delivers the packets to userland as soon as the application is ready to receive them. Both the problems have been solved by recent releases: BPF now has an 'immediate mode' that can be used to pass the packets to user-mode without the need to fill the hold buffer, winpcap has the 'delayed write' capability that optimizes the CPU usage.
- The filtering and pre-filtering process has about the same impact in Windows and freebsd. However, since the Win32 version of the BPF virtual processor is nearly identical to the original, the Windows kernels (and in particular the one of win NTx) seem sensibly faster in the pre-filtering (interrupt+dma(or equivalent)+nic driver), in particular when al lot of packets are discarded by the BPF filter. This was very surprising for us when we performed the test, because comparing the linear and elegant implementation of the bsd networking with the enormous and complex NDIS we expected exactly the opposite situation. However our recent tests seems to confirm this situation. I think I have an explanation for this, and I will write about it only if somebody is interested.
- When the packets are dumped to disk, the file system plays a very important role. This is quite obvious. Notice for example that, according to our tests, a fat32 partition under Windows 2k is able to save 15-20% more packets than a NFTS partition of the same disk: fat is a very poor file system, but for this reason it is very fast, and not only in Windows. However, Win2k seems to have a very efficient storage management. In my opinion, the block size of the stdio library is not the only parameter that influences the dump perormance. Don't forget that the data that tcpdump saves to disk is already buffered by libpcap (with a 32k buffer in bsd and a 256k buffer in win), so a further buffering is not so essential. FS kernel-level cache (that in Win2k is quite large) is very important too, because it is the last buffer before the physical write. Moreover, the method used by the application to save the data has to be taken in consideration: libpcap requires pcap_dispatch() to be invoked for every saved packet, and is able to save with pcap_dump() only a packet at a time. This is very inefficient, because requires a lot of cpu and doesn't make a good use of the libpcap packet buffer. In my opinion, a pcap_write() function able to directly save the content of the packet buffer would be noticeably faster.

Now, some words on our new tests. We performed them in order to measure the performance of the new version of winpcap (2.1) that currently is in beta on our web site. We compared it to BPF for freebsd 4.1, trying to obtain more accurate and objective results. In particular, we saved every dump (under Win98,Win2k and freebsd) on the same fat32 partition, and tested tcpdump also with a kernel-buffer size of  2*512K. Moreover, we stressed the capture architectures with a more sustained traffic (until 83kpps). I don't attach the results to this message because they are in an 88k excel file, but I have not problems to send them to the people interested.
They show that the cpu performance of winpcap for winNTx is now comparable with the one of freebsd and that freebsd has still lower dump performance, but saving on fat32 it is able to obtain better results and in a case (22000 500bytes packets per second) it is better than win2k. The really strange thing is that BPF seems not to perform noticeably better if a 1 megabyte buffer is used instead of the standard 64k buffer, and in some cases the 32k buffer performs even better. I wrote a message to the tcpdump mailing list (http://www.tcpdump.org/lists/workers/2000/msg01135.html) to ask about this behaviour, but I obtained no satisfactory answer.

Finally, I would like to say that since I know we can't be considered very impartial, we would like to see other tests and evaluations on this subject, because we think it is a very important matter. We had to make a comparison with bpf because we didn't find any quantitative value on this subject before. If anyone obtained different results, please let us know.

Loris.


From: Dragos Ruiu <dr@xxxxxxx>
To : <tcpdump-workers@xxxxxxxxxxx>; <ethereal-dev@xxxxxxxxxxxx>; <snort-devel@xxxxxxxxxxxxxxxxxxxxx>; <freebsd-hackers@xxxxxxxxxxx>; <tech@xxxxxxxxxxx>
Subject: [Ethereal-dev] Fwd: kyxtech: freebsd outsniffed by wintendo !!?!?

> 
> (Hurm.... Wintendo outperforming unix???!??  Something's
>  improper about this, and it ought to be fixed...  :-) 
>  Comments?  Other OS numbers: more recent 
>  FreeBSD versions? Solaris? Tru64? Optimization
>  patches? Can those OO MSDN lobotomies actually
>  be good things? Hurm... The Italian gauntlet has
>  been thrown down....   --dr :-)
>