Wireshark-dev: Re: [Wireshark-dev] New packet list: Optimize memory usage

From: didier <dgautheron@xxxxxxxx>
Date: Mon, 13 Jul 2009 16:44:09 +0200
Le dimanche 12 juillet 2009 ᅵ 21:48 +0200, Jakub Zawadzki a ᅵcrit :
> Hi, 
> 
> This patch (Proof of Concept) removes allocating memory for columns data,
> and makes them 'dynamic' (packets redissected when column data needed)
> 
> Getting column data is done in packet_list_record_get_column()
> I'd be grateful if someone could give some hints how it could be optimized.
> (Anyway I think it'd be nice if we have some column cache for some previous and next rows, or e.g. tcp.stream related)
> 
> Unfortunetly resorting for large dumps is PITA - we could also cache some columns (configurable by user?)
>From my own test (cf http://wiki.wireshark.org/Development/Optimization)
I would say that you always want to cache columns so it's only slow the
first time but:
- you don't need to cache some of them, frame number and times.
- use a cache for addresses 
- don't copy constant data (protocol and col info use a lot of them).

Memory usage is not that bad:
my patched version with a 53MB file (250k packets)
virt res
137  84 MB
after sorting
157 106 MB

svn (compiled without WANT_GUARD_PAGES and DEBUG_USE_CANARIES)
368  315 MB
but there's the copy of col_expr in frame data. 

And on a bigger file (doesn't load in svn Wireshark):
557 MB 1.4M packets

virt res
470MB 420MB
after sorting
600MB 518MB

Didier