On Mon, Dec 10, 2018 at 8:11 PM Guy Harris <guy@xxxxxxxxxxxx> wrote:
>
> On Dec 10, 2018, at 5:16 PM, Richard Sharpe <realrichardsharpe@xxxxxxxxx> wrote:
>
> > It has taken 20 minutes or more to load this monster (looks like about
> > 5M packets)
>
> "It has taken 20 minutes or more..." as in "it took 20 minutes or more to load it" or as in "it has, so far, taken 20 minutes or more to load it, and it's still loading it"?
The latter.
> > and I dread making searches through it.
> >
> > The Wireshark process is only consuming about 1.7GB of memory at the
> > moment and 17% of CPU on my laptop with NVMe and oodles of memory and
> > a 4-core Xeon.
>
> "At the moment" as in "after it finished loading, it's only consuming..." or "while it's loading, it's consuming..."?
While it's loading it's consuming ...
> The dissection process is currently not multi-threaded, so extra cores don't help. Making it multi-threaded is possible, but it would involve a lot of work to parallelize an in-principle-sequential process, including finding dependencies (e.g., a case where dissecting packet N involves looking up something in a table that might be modified by the dissection of packet M, for M < N, in which case the dissection of packet N has to block waiting for packet M to be dissected).
>
> Just out of curiosity, what happens if you read the file with TShark without the -2 flag, sending the output to the null device, and then with the -2 flag, again sending the output to the null device? What's the difference in the memory consumption of TShark in those cases, and how does it differ from the memory consumption of Wireshark for the same file? (That could indicate how much memory is used by the frame table - which shouldn't be constructed by TShark without -2, but should be constructed by TShark with -2 and by Wireshark - and how much is used by the GUI display widget showing the packet list, which will only be in Wireshark.)
I will give that a try.
However, I have found a way to make it more tractable. Since there are
two TCP streams (connections) in the capture, I am using tshark to
split the capture into those two streams, one of which I am interested
in (I don't know which at this stage). They should be around 2.9GB
each.
--
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)(传说杜康是酒的发明者)