Marc,
Yes, Jaap is correct. Enabling the /3GB switch mentioned in
http://wiki.wireshark.org/KnownBugs/OutOfMemory is what made my WireShark work better with my XP Pro and 4 GB RAM. I can open more of my 512MB capture files OK but others it crashes at around 65%. What the packets are in the packet capture directly relate to how much memory it takes. At home I have 8GB RAM and a Core2 Quad with Windows 64-bit Vista and I have even less (not zero) Wireshark memory crashes.
If you find a memory leak or a memory conservation solution, let us know.
Thanks!
Brian