[Joseph Laibach <jlaibach@xxxxxxxxxxxxx>]

Doug,

                I was thinking of going that route, but I really only need the current days worth of traffic. I changed the ring buffer limit and re-complied Wireshark. I now have the capture running and it got past the 10000 limitation and currently is moving towards the 20000 mark now. For someone like me that had no idea on how to compile code the change was very easy and didn’t require me to re-invent the back-end processes that we use to find sequence number of the market data that we are capturing. I built a whole web console on a linux machine that searches multiple sniffer machines and uses tshark to read the capture files and parse out the sequence number that we are looking for. The last piece of this puzzle was get the data captured for the whole entire day, instead of 4 hours at a time. The next thing I’m looking to figure out is a dissector for the market data that we capture, ie: NASDAQ, NYSE, ARCA, CME, etc…

 

Thanks

 

Joe

 

From: wireshark-users-bounces@xxxxxxxxxxxxx [mailto:wireshark-users-bounces@xxxxxxxxxxxxx] On Behalf Of Douglas Ross
Sent: Thursday, May 20, 2010 6:26 PM
To: Community support list for Wireshark
Subject: Re: [Wireshark-users] tshark or dumpcap ring buffer limitations

 

Joseph,

 

Have you considered compressing the capture files?

If two of your concerns are the huge amount of disk space and consequent network traffic you need to manage this data?

 

I'm very new to Wireshark, but have used Ethereal in the past, and periodically compressed capture files and deleted the originals.

(eg. WinRar achieves a compression ratio of about 10/1)

 

Anyway, I made a script to automatically compress daily files into two archives per month. This not only reduced disk usage, but also dramatically reduced the number of files/folders.

 

If 1000 ring files is the hard coded limit, then auto compress and delete after every 500 or 800.

If "0" ring files implies no limit, then auto compress after whatever number is most convenient.

 

I used the capture file dts (date.time stamp) to determine which half-month archive it should be put in. eg:

...20100501... to ...20100515... I'd put in archive ...201051

...20100516... to ...20100531... I'd put in archive ...201052

(undoubtedly you'd make a finer split, perhaps into one or two archives per day ..)

(capture file name includes start of capture dts; system dts is at close of file (= start of next))

 

Hopefully, that will help solve the problems of volume of files, and waste of disk space and network capacity.

 

Similarly, I used scripts to help decompress whichever file I needed, based on dts.

 

Hope this helps for a relatively quick fix, at least to give you some ideas.

 

Good luck

Regards

Doug

 

PS. while you're scripting this (if you go down that route) you could consider doing first pass analysis, and filter out the stuff you're not interested in, and/or split the capture into known good and useful stuff; definite rubbish(discard); and possible trouble...

 

 

---

From: Jeff Morriss <jeff.morriss.ws@xxxxxxxxx>
To: Community support list for Wireshark <wireshark-users@xxxxxxxxxxxxx>
Sent: Fri, 21 May, 2010 3:34:54 AM
Subject: Re: [Wireshark-users] tshark or dumpcap ring buffer limitations

Joseph Laibach wrote:
> All,
>
>                I’m running a continuous capture of data. I’m trying to
> use a ring buffer of 25000 files with an 8mb file size. The problem is
> that the ring buffer starts overwriting after 10000 files. I’ve tried it
> with dumpcap and tshark. The command is using the –b files:25000 –b
> filesize:8192. Is there a limitation to the size of the ring buffer for
> dumpcap and/or tshark?

Turns out that if you specify the number of files as 0 then
dumpcap/*shark will create an unlimited number of files.  I don't know
if that's acceptable or if you really need it to roll over at 25,000,
but it's an option.
___________________________________________________________________________
Sent via:    Wireshark-users mailing list <wireshark-users@xxxxxxxxxxxxx>
Archives:    http://www.wireshark.org/lists/wireshark-users
Unsubscribe: https://wireshark.org/mailman/options/wireshark-users
            mailto:wireshark-users-request@xxxxxxxxxxxxx?subject=unsubscribe

 

This communication is for informational purposes only.  It is not intended as an offer or solicitation or as an official confirmation.  Market prices and other information are not guaranteed as to completeness or accuracy and are subject to change without notice.  Schonfeld Group reserves the right to monitor and review the content of all messages sent to or from this e-mail address.