[Loris Degioanni <loris.degioanni@xxxxxxxxx>]

If I understand well, what you ask is a way to "group" a set of packets 
to quickly navigate among them. In this case, ntar already has a 
solution to the problem: the file format subdivides the capture in 
sections, whose number and size is controlled by the calling program (or 
library, in case of libpcap) using the ntar_create_section() and 
ntar_close_section(). It is possible, for example, to close the current 
sections and create a new one after 10.000 packets. When reading the 
file ntar_get_next_section() or ntar_get_previous_section() can be used 
to navigate through the sections and reach the desired one.

Regarding advanced ways to speed-up Ethereal capture handling, this is 
the precise goal of the directory block 
(http://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html#anchor15), 
which however is not been implemented yet.

Loris


ronnie sahlberg wrote:
> The draft describes two different blocks for holding a captured packet:
>
> Packet Block: it contains a single captured packet, or a portion of it. 
> Simple Packet Block: it contains a single captured packet, or a
> portion of it, with only a minimal set of information about it.
>
>
> I often work with very very large capture files and often want to only
> extract a very small subset (packets captured between time X and time
> Y).
> This is very very slow with the current fileformats doe to the massive
> amount of data that has to be processed.
>
>
> I would propose to introduce a new type of block  and encapsulate both
> PacketBlock and SimplePacketBlock in it :
>
>
>
> Packet
> ======
> This the block you refer to in the draft as PacketBlock. A Packet MUST
> be encapsulated inside a PacketBlock.
>
>
> SimplePacket
> ============
> This the block you refer to in the draft as SimplePacketBlock. A
> SimplePacket MUST be encapsulated inside a PacketBlock.
>
>
> Then introduce a new PacketBlock :
>
> PacketBlock
> ===========
> A block that contains one or (preferrably) more Packet or SimplePacket
> structures.
> Format of PacketBlock (in units of 4 bytes):
>
> Version: 1 byte,  version of packetblock structure.
>
> Num packets: 3 bytes,  number of Packet or SimplePacket structures in
> this PacketBlock, or zero (==use offline tool to calculate this and
> populate the field)
>
> Length : 8 bytes,  total length of this block in bytes.
>
> TSofFirstPacket: timestamp of the first packet or simplepacket in this
> PacketBlock or zero (==run some offline tool to populate this field if
> you need it)
>
> TSofLastPacket: timestamp of last packet or simplepacket in this
> PaketBlock or zero.
>
>
>
> A PCAP writer should aggregate PacketBlocks/SimplePacketBlocks into
> blocks of approximately 10MByte.
> This default approximate block-size should be controllable by the user
> but i think 10MByte should be reasonable.
>
>
>
> Assuming we have a 10GByte capture file, then this would make it possible to :
> 1, parse the cap file and collect all PacketBlock headers, timestamps
> and number of packets in each block.  fseek() to the next block
> instead of parsing it packet by packet.
> 2, present to the user a list :
> Block 1: start:x:y stop:x:y  packets:z
> Block 2: ...
> ...
> 3, Allow the user to ONLY read/extract a subset of the PacketBlocks in
> teh capture file.
>
>
> Having this kind of encapsulation structure would allow users to use
> large captures (and skip the portions that are of no interest) while
> the current design would require the user to read the entire 10GB
> file.
>
> 1GB and larger capture files are becoming more and more standard to
> many users and reading through this massive amount of data is slow and
> painful.
>
>
>
> On 6/25/05, Gianluca Varenni <gianluca.varenni@xxxxxxxxx> wrote:
>
> > Hi all.
> >
> > This mail is to announce the birth of the NTAR project. NTAR stands for
> > Network Trace Archival and Retrieval library, and is an implementation of
> > the PCAP next generation dump file format, that was proposed and discussed
> > last year by several folks on the libpcap/tcpdump and WinPcap mailing
> > lists.
> > The library is released under the 3-clause/BSD license.
> >
> > The URL of the project is:
> >
> > http://www.winpcap.org/ntar
> >
> > On this website you can find
> > - the source file of the library (both for windows and *nix)
> > - the HTML documentation of the API (generated with doxygen from the
> >  commented source files), both for the user  and for someone wanting to
> >  extend it. The docs contain some tests/examples that you can look to
> >  get an idea of how the library works.
> > - An updated version of the PCAP draft specifying the file format.
> >  The original draft of the file format is available at
> >     http://www.tcpdump.org/pcap/pcap.html
> >
> >
> >
> > I'm seeking contributors to improve the library in terms of
> > - testing the library on different platforms and operating systems
> > - reviewing the API
> > - implementing new extensions to the library.
> >
> > A mailing list, ntar-workers@xxxxxxxxxxx, has been created for NTAR-related
> > discussions. People interested in this project are welcome to join it, the
> > mailman web interface to subscribe is available at
> >
> > https://www.winpcap.org/mailman/listinfo/ntar-workers
> >
> > Have a nice day
> > Gianluca Varenni
> >
> > _______________________________________________
> > Ethereal-dev mailing list
> > Ethereal-dev@xxxxxxxxxxxx
> > http://www.ethereal.com/mailman/listinfo/ethereal-dev
>
>
> _______________________________________________
> Ethereal-dev mailing list
> Ethereal-dev@xxxxxxxxxxxx
> http://www.ethereal.com/mailman/listinfo/ethereal-dev