["Kevin Cohen" <kcohen@xxxxxxxxxxxxxx>]

OK,

I wrote a simple script that is based on the information below (I'd be happy
to post it if anyone is interested).  What I did, was I took 40 GIGs of
binary data, extracted all of the SMTP data into binary format (which now
turns out to be only 1 GIG of binary data split up into about 50 files).

My new script parses through these 50+ files, in probably over 3 days time
(Today is the first day and only 20% done since last night), to spit out new
binary files that represent communication between two separate IP addresses.

I guess my question is what can I do to cut down the time to crunch through
this.  I don't think that 1 gig should take 3 days....  Is speed based on
RAM or CPU speed?  Do I need to consider a cluster farm, better equipment or
what?  Any pointers would help.

Thank you,

Kevin


-----Original Message-----
From: ethereal-users-admin@xxxxxxxxxxxx
[mailto:ethereal-users-admin@xxxxxxxxxxxx]On Behalf Of Visser, Martin
(Sydney)
Sent: Monday, July 08, 2002 10:19 PM
To: ethereal-users@xxxxxxxxxxxx
Subject: RE: [Ethereal-users] Batch capabilities?


How about something along these lines?

1. Write a script to parse out all the source and destination addresses. The
following generates output like this

16.153.16.21,16.153.16.255
16.153.16.82,255.255.255.255
16.153.176.249,16.172.128.29
16.153.208.249,16.172.128.29
16.153.208.86,16.172.128.29
16.167.0.1,16.167.0.255

tethereal.exe -V -R ip -r your_capture_file.cap | perl ipaddress.pl | sort |
uniq

ipaddress.pl looks like this :-

#!/usr/bin/perl
# ipaddress.pl parses verbose tethereal output to give ipsrc,ipdst
while(<>) {
  chop;
#   Looking for
#  Internet Protocol, Src Addr: 10.1.0.1 (10.1.0.1), Dst Addr: 10.1.0.2
#  (10.1.0.2)
  if (/^Internet Protocol, Src Addr: (\S+).*Dst Addr: (\S+)/ ) {

    $ipsrc = $1;
    $ipdst = $2;
    print "$ipsrc,$ipdst\n";
  }

(You could also print out the file and frame number which might come in
useful later)

2. Now you have a list of unique source-dest conversations.

This could then be used again in another script (left as an exercise for the
reader :-) iteratively using doing something like tethereal -R
"ip.src==1.2.3.4 && ip.dst=4.3.2.1" -w saved_capture_by_conversation.pcap

Alternative you might find it less CPU intensive to use editcap specifying
the frame numbers (that you parsed out on the first pass)

Hope this helps with a few ideas

Martin Visser
Network Consultant - Global Services
COMPAQ, part of the new HP

3 Richardson Place
North Ryde, Sydney NSW 2113, Australia
Phone *: +61-2-9022-1670    Mobile *: +61-411-254-513
   Fax 7: +61-2-9022-1800     E-mail * : martin.visserAThp.com




-----Original Message-----
From: Kevin Cohen [mailto:kcohen@xxxxxxxxxxxxxx]
Sent: Tuesday, 9 July 2002 7:05 AM
To: ethereal-users@xxxxxxxxxxxx
Subject: [Ethereal-users] Batch capabilities?


Hi all,

I'm a newbie with Ethereal, so I'm not sure what all of its capabilities
are.  I'm hoping someone can help me out with my problem.

I have about 40gigs of captured data in binary format.  I have the files
separated into about 100MB files, so they are not too large (a 2GIG file
takes forever to open in Ethereal).  I would like to write a batch script
that can go through the files and pull out source or destination addresses
and put them into one file.  I would also like to batch based on port and or
protocol.

Currently, I've been able to use Snort and manipulate packets based on IP's
if I convert the Binary to ASCII.  I then do not know how to convert the
ASCII back to
Binary, or if that is possible.

If anyone has any suggestions, I would very much appreciate it.

Kevin Cohen, CISSP
President
Data Triage Technologies Inc.
kcohen@xxxxxxxxxxxxxx
310.478.2113


_______________________________________________
Ethereal-users mailing list
Ethereal-users@xxxxxxxxxxxx
http://www.ethereal.com/mailman/listinfo/ethereal-users

_______________________________________________
Ethereal-users mailing list
Ethereal-users@xxxxxxxxxxxx
http://www.ethereal.com/mailman/listinfo/ethereal-users