Ethereal-dev: Re: [Ethereal-dev] large-file support?

Note: This archive is from the project's previous web site, ethereal.com. This list is no longer active.

From: Guy Harris <guy@xxxxxxxxxxxx>
Date: Wed, 23 Jul 2003 23:45:23 -0700

On Wednesday, July 23, 2003, at 10:42 PM, Naveen Kumar Kaushik wrote:

I have non zipped larger than 2 GB file captured by some other tool (ie
Wiretap and zlib limitation does not come in to picture).

The zlib limitation doesn't come into the picture, as the file isn't gzipped.

However, the Wiretap limitation does come into the picture, because you're trying to open the file in Ethereal - *ALL* capture files, *REGARDLESS* of the tool that captured them, are read using the Wiretap library code.

Now By design, all versions of Win32 (Win9x/ME/NT/2000) provide full support
for 64bit pointer file operations ,

Are you certain of that? Windows NT (NT 3.x, NT 4.0, NT 5.0 a/k/a Windows 2000, NT 5.1 a/k/a Windows XP, NT 5.2(?) a/k/a Windows Server 2003) might support it, but are you certain that Windows OT (95, 98, me) support it?

so can we just change source so that for
unzipped file of size greater than 2 GB can be opened on window.ie limited
LFS support.

As you said in your first sentence - and as I explained in my first reply in this thread - "just chang[ing] the source" isn't necessarily easy. Wiretap can't use "fseek()", for example, as the offset in "fseek()" is a "long", and, on Windows *AND* ILP32 UNIXes, a "long" is 32 bits. Unfortunately, the "fsetpos()" alternative uses an "fpos_t", which is *not* guaranteed to be an integral data type; equally unfortunately, Wiretap and its callers expect that seek pointers in files *are* integral data types.

*If* fpos_t is an integral data type on Windows and remains so, and if it's also an integral data type on those UNIX systems that support it and remains so, *then* we could, on platforms that support "fsetpos()" *and* where "fpos_t" is an integral data type greater than 32 bits *and* where "fopen()" can open a file >2GB and let you seek on a file >2GB (i.e., where you don't, for example, have "fopen()" not opening with O_LARGEFILE and O_LARGEFILE being necessary to read past the 2GB point), support reading capture files greater than 2GB in size (assuming the capture file format supports it - libpcap format doesn't have any file offsets in the file, so it should support it, but Network Monitor format has 32-bit file offsets, so it could, at best, support files 4GB in size).

On other platforms, however, you're out of luck.

(BTW, the Single UNIX Specification, Version 3, section on "Standard I/O Streams" seems to imply that C99 requires that an "fpos_t" has to include some junk to handle "wide-oriented" streams, which suggests that any system whose C library conforms to C99 will almost certainly *not* have "fpos_t" be an integral data type. Perhaps there aren't many systems now that conform to C99, but there might be some in the future....)

This can possibly extended to Linux since future kernel releases of linux
might have LFS support (since Linux group is working on it).

*Current* kernel releases of Linux support it; did you not read Ben Greear's mail, in which he said he had a program that read capture files >2GB on Linux? Current GNU libc also supports it, as far as I know.

But Linux is far from being the only interesting non-Windows platform for Ethereal; it works on all the BSDs, and the BSDs have supported files >2GB since 4.4BSD, and Solaris, on which Ethereal also runs, has supported it since Solaris 2.6. Ethereal also runs on Digital UNIX, but "long" is 64 bits there so it supports files >2GB for free.

Unfortunately, few of them make it easy to support reading those files with Wiretap the way it currently works. We might change it to do its own buffering at some point, in which case it might be easier to support it on Windows and on UNIX platforms that either natively support 64-bit file offsets (e.g., the BSDs) or support the Large File Summit APIs, but that's probably not going to happen soon, so, unfortunately, you'll continue to face the problem of handling very large capture files for a while.