["Ronnie Sahlberg" <ronnie_sahlberg@xxxxxxxxxxxxxx>]

----- Original Message ----- 
From: "Martin Heroux"
Sent: Friday, November 21, 2003 7:59 AM
Subject: Re: [Ethereal-users] tcpdump vs ethereal


>
> One thing I did which works well, is that I created a RAM disk of 500MB
> which I mount under /tmp
> So ethereal capture and write it in RAM... I have 1GB of RAM so I can
> affort a 500MB ramdisk

If the data you capture is less than the available RAM you should get
EXACTLY the same performance by
writing the data to disk as by writing to a RAM disk.

There is NO speed difference at all by writing data to a buffer that will,
at the operating systems convenience,
sometime later, then the CPU is idle be flushed to disk
compared  to writing some data to a buffer that will not be flushed to disk.

In your example above, there should be NO difference whatsoever in
performance between writing to ramdisk or writing
to physical disk.

The difference is when the data captured is larger than the available memory
then
   a RAM disk will fail completely
   a physical disk will make the capture process slow down  but still work
with degraded performance.


If there is a performance/speed difference when you test this on your box,
then your OS has a broken buffer cache.  Upgrade to Linux or any other
OS that works.

>
> The amount of packet drops dropped, but still not 0
>
> I don't think that we could do an home made sniffer that competes Dolch or
> the like mainly because (that's what I have been said) they capture
> directly on board and then slowly and quietly download their buffer to an
> hard drive.

Yes you can.

1, use one internal disk with the filesystem and the Linux install on it.
This sits on a separate disk on a separate IDE controller.
2, if possible, get a motherboard with multiple pci buses.
3, get as many multiport high end high performance IDE controllers as
possible.
4, get 8 modern IDE disks and distribute them across the IDE controllers.
    Do not use Master/Slave configurations if possible.  1 disk ONLY per bus
if possible.
5, hdparm is your friend, Make sure DMA is enabled and push the
controllers/drives up to the highest possible mode they support.
6, create a striped device across these 8 disks.   Experiment with different
stripe sizes from 32kb to a few MByte.
7, Do NOT create a filesystem or mount this stripe device. Filesystems make
access non sequential when metadata is accessed.
8, get 2 high CPUs  that are as fast as possible.
9, put as much and as fast as possible RAM in the box.
10, google for the pcap patch that reduces copying of data.
11, compile your own latest and greatest Linux kernel.
12, remove and disable ALL services and daemons that are not essential.
13, do NOT use X. Use the commandline/console that god gave us.
14, run tcpdump as a realtime process and assign it the highest possible
priority.
15 make sure tcpdump writes the data to the striped device and NOT to a
file.

when finished, use dd or something to pull the data back from the striped
disk into a file on the linux disk.



>
> The way we are doing it, the packet needs to go through a couple of layers
> before being written on the disk, it is not right tru.

It goes to buffer cache.   Sometime later buffercache will be paged out to
disk.
Write to the device. Do not use a filesystem,   this to guarantee that the
writes will be sequential.

>
> However what I am surprised is that modern SCSI drive turning at 7500 RPM
> are quite faster than 1Gbps card isnt ? So why does the storage unit can
be
> a bottleneck ?

No.   A 1GbE link is many times faster than any single spindle disk.
A modern single spindle can not do more than ~30MByte/s.
That disk can ONLY even closely approach that 30MByte/s number if all
accesses are purely sequential.
To have the slightest chance to go above a maximum of 30MByte/s for optimal
pure sequential access you MUST
satripe the data across multiple spindles.

A saturated full duplex link is >200MByte/s

>
> Still, I don't think a 100K sniffer solution (NAI high-end distributed
> sniffer solution for example) worth its price... I, for one, can live with
> a 0.05% percent of packet loss :-)
>

Build the box as I specified and see.