["McNutt, Justin M." <McNuttJ@xxxxxxxxxxxx>]

> Here's my perl code to do the split. It's tested on a 
> smallish trace file,
> although I can't guarantee it's efficiency when re-reading a 2GB file
> multiple times.

Yes, editcap is inefficient that way, but the code is sound.  Thanks!

[code snipped]

We do indeed need something more like the command-line "split" command.  An app that would split a given capture file into chunks of NN packets, MM kB, or XX seconds.  Tcpslice and editcap can give you a *single* chunk of time or certain packet numbers, but they don't really split the file.

> > 2)  I need to be able to use the ring buffer feature with a 
> > ten-second autostop (rather than specifying a file size).
> 
> Can't help you here, although given the volume of traffic you 
> appear to
> anticipate collecting I can't see why you wouldn't settle for 
> a frame count
> based solution. Start a new file every 1000 frames if it pleases you.

This is based on what the capture is meant to do.  Since we want to capture a large data stream for after-the-fact analysis, we want to be able to search quickly for the most appropriate data.  If each capture file represented a ten-second or one-minute window in time, it would be easy (easier) to locate the right piece of the stream based on the files' timestamps.

Using a certain number of files for a given period of time also means that we'll always have a constant number of files that hold three days of data.  If I roll over based on file size or frame count, I'll never be able to guarantee that I have three days of data, and that's the goal.  When we come in on Monday, we want to be able to analyze a piece of the data stream from Friday night.

> > 3)  I need to be able to use at least 1000 files in the ring 
> > buffer (although about 60,000 would be much better).  This 
> > one is by far the most important, since if I can get past the 
> > 10 file limitation I can worry about item 1) above and make 
> > do, but with only 10 files in the ring buffer I'm screwed.
> 
> Ethereal doesn't maintain any locks on the other files in the 
> ring buffer.
> Perhaps you could copy (or even move) the files to your own 
> "ring buffer"
> directory. Should be a trivial perl script to sleep every 
> 100ms and check
> for any new files and FIFO data older than 3 days. 

Sleeping for <1s in Perl is non-trivial, but not impossible.  I posed this question, though, because it seemed that Ethereal shouldn't need much modification to allow for this, based on the segment of a thread prevously posted to ethereal-dev.

But yes, one script could do the file copy and roll-over, and another (quite trivial) script could watch for files older than three days and remove them.

In fact,

find . -type f -ctime +3 -exec rm {} \;

takes care of the old files.

But as I said before, the code is already in Ethereal.  I don't know that removing the 10-file limit is very difficult, and would remove an awful lot of scripting on my end.

> > The deal is that I need to run a perpetual packet capture on 
> > a 75+ Mb link and I need to buffer to hold at least 3 days 
> > worth of data.  I have the disk space and the server hardware 
> > to do this, but I'm limited by Ethereal.
> 
> This is certainly a non-trivial task!

Nah.  It's just expensive to buy that much disk space.  :-)

> Perhaps you could consider other technologies such as ntop or 
> Netflow to collect this for you?

Those apps collect statistics.  We want the data itself.

> What are you planning on doing with the data when you collect it?  
> 
> Do you need the full frame count or just the src and dst IP addresses?
> 
> Are you just looking for a particular event? Ethereal can't 
> do this (yet),
> but you can get NG Sniffer to start storing a capture to disk 
> when it sees a particular trigger frame.

We want the entire data stream for after-the-fact analysis for both intrusion and network performance reasons.  On multiple occasions it would have been extremely useful to us to "go back in time" and see what data triggered an event.

We don't know in advance what the data will be, so we can't program in a trigger frame.  We need the entire data stream, because in some cases we need to prove that the traffic contained a certain type of data (SRC/DST ports aren't proof).

Ethereal - modified only slightly - has all of the pieces we need, but I've got to get around the current limitations in the ring buffer.

--J