["Ronnie Sahlberg" <sahlberg@xxxxxxxxxxxxxxxx>]

Hi,

First let me state that I have read each and every followup on the initial
post
carefully and have gained many valuable insights and ideas.
I especially want to thank Joerg Mayer for his first post. It gave me a lot
of insights on
things I had not thought through properly.

I will not respond to any post individually but will address the result of
applying the more importnant
(as i thought) points in this very mail. I may have forgotten some points,
sorry for that.

I will refer to every extension as graphs, eventhough it may be used for
other things as well.

First some short responses:

Why not update the graph whenever a new packet arrives?
Arbitrary desicion to reduce CPU load. I think it is enough if the graph is
updated once every second,
if there is new data. I image on live captures one might have 1000s of
packets coming in every second.

Why not use e.g. GNUPlot to print the graphs instead of coding it in GTK?
Actually an excellent idea. The rest of the proposed draft design will be
based on using something that opens a pipe to write commands into GNUPlot
and let gnuplot do all the drawing (which will then make sure the graphs
will always look really nice and professional).
Another advantage is that we could then start GNUPlot with nice-ing it to
very low priority on each and every (unix) platform so that the graph
updates will not steal valuable cpu cycles if ehtereal needs it for
something more important. To me its preferable to have jerky updates of the
graph than to get higher packetloss if capturing.

Why a separate thread to process data for the graphs and why low-priority?
A thread would run in the same memory space as the main ethereal
application, this means it would
be able to take simple and cheap pointers as arguments to the datacollection
subroutine. This graph
data collection routine would also be able to follow pointers if it needed
to if we were to pass
pinfo or whatever to it.
I.e. we would get very cheap ipc (since we only pass a pointer and dont need
to marshal anything) and
we would have maximum flexibility. We pass *pinfo and a *void, the
collection thread could potentially
access anything and everything inside ethereal. We thus do not create any
restrictions limitations
on what kind of data we can handla and what we can not.
Why low-priority?  Well, data processing to update a graph window once every
second or so is not as timecritical as the rest of ethereal like capturing
packets and updating the packetlist. Lowpriority
would allow us to make sure the important things get the cpu when they need
it and the graph thingy
gets the cpu if nothing else is happening.

Different types of filtering?
I thought without analyzing too closely on if the current implementation
would allow it, that new packets/events
would only be fed to whatever graph thingy if it first passed the
displayfilter.
Something like if(tree && !pinfo->fd->flags.visited){
This would mean that the graphs would only be based on whatever packets the
displayfilters would pass.
Entering the displayfilter "ssh"  would automatically change all graphs to
only show statistics for "ssh"
since only these packets would be sent to the graph thingies.
No new filtering, just apply proper displayfilters and the graphs will do
the right thing.

We should not write too much new code!
Agreed. Especially lotsa code for one specific toolkit and at the same time
killing off KDE/Qt/W32native/Curses
ports. Definitely. I am ashamed I did not see this, perhaps the most
important requirement.
I would therefore propose that NO widget specific code is implemented in
ethereal, instead the graphies should
only process the data and push it down a pipe to GNUplot and let GNUPlot
draw it.
This also applies to the request to not add lots of new bloat to ethereal.
By offloading all the graphics manipulation to external tools (that does
these thing really well) the amount
of bloat will be limited. I still will aim at keeping the interface clean so
it can easily be changed to dlopen() stuff
for thse that dont want any bloat at all.



Ok, new proposal:
All graphs are managed by separate GNUPlot processes. Ethereal will
communicate to teh command interface
by writing commands to a pipe to the cli of gnuplot. This pipe is one-way.
Ethereal graphie thing writes and gnuplot reads.


The dissectors are updated in the following way:
-------------------------------------------------
Each dissector (as we need to) has the following two new changes:
proto_reg_handoff_xxx
     will also do   tap_handle=register_packet_tap("xxx");
dissect_xxx
     the main dissection function for the protocol will add, after the
entire packet is dissected (and all structures
     updated and all child dissectors have returned) something like this
just before they return :
     if(tree && !pinfo->fd->flags.visited){
           tap_packet(tap_handle, packet_info *pinfo, void
*protocol_specific_structure);
     }     /* better names for the functions are WELCOME */


register_packet_tap() is used to tell the infrastructure that this dissector
can be tapped.
tap_packet() will then call each datacollector subscribing to this tap in
turn. For this it is importnant to
keep these datacollectors as lightweight as possible.

So, the API between the dissector and the wanted infrastructure is the two
functions
tap_handle *register_packet_tap(char *tap_name)
void tap_packet(tap_handle *hnd, packet_info *pinfo, void
*protocol_specific_data)

We can enhance dissectors on a need basis, only addint this functionality to
it whenever there is an extension that needs it.


magic infrastructure inside ethereal
-----------------------------------
the infrastructure will make all of it work and is what we must implement.
it is not important though what it looks
like. what is important is the interface to use it so people can use it.


The datacollection and graph processing units. I.e. the extensions
-----------------------------------------------------------------
The extensions are just a bunch of subroutines. These are not implemented as
threads. If need be for separate threads running in parallell to the main
ethereal thread this fact can be hidden in the infrastructure above in that
case.
The following functions are required for an extension (No other calls across
main ethereal nor global variables are allowed to make it easy to dlopen()
convert it later to reduce bloat)
(you dont like the name graph? neither do I, come up with something better)

The extencion .c file should contain a function
register_graph_xxx()
    used to initialize the extension. This function is called in some manner
(initially similar to register.c since it is
    easy?) once when ethereal starts to initialize all state and prepare the
extension for action.
    One thing it would need to do would be to register itself as a
datacollector with the infrastructure with
register_data_collector(
    "FrameSizeHistogram",  /* Name of this extension as it will appear in
the tools menu where on/off is toggled */
    void(*start_collecting)(void),
    void(*stop_collecting)(void),
    void(*reset_counters)(void),
    void(*draw_graph)(void)
);

The first two functions are called whenever the menuitem is ticked or
unticked:
start_collecting()   will clean up all state variable (by calling
reset_counters?) and then do things like
    starting GNUPlot with an open pipe to ewwrite commands to it.
    When all state is prepared we will here also register which taps we want
to listen to by calling
    open_tap(
        "frame",   /* name of the tap when it was created in teh dissector
above */
        col_handle=(*collect_data)(packet_info *pinfo, void
*protocol_specific_data)
    )
    collect_data is what is called whenever tap_packet() is called in the
frame dissector.
stop_collecting() will kill gnuplot and clean up all state.
    It will also remove all taps we have placed by calling
    close_tap(col_handle);
    which will stop tap_packet() from sending more stuff out way.

We can thus have multiple collect_data() functions in one and the same
extension, in case we want to
assemble data from multiple protocols in teh same thingy. (protocol
statistics replacement?)
collect_data() is called whenever tap_packet() is called from the
dissector(s) we have registered
    we want to subscribe to.
    For each and every frame that passes the display filter, we will get one
call to collect_date() for each
    protocol we have opened the tap to.
    collect_data() should be as cheap as possible not to interfer with the
important parts of ethereal.

void (*reset_counters)(void)
   this function is called whenever the infrastructure wants us to reset all
state and all counters (but let whatever
   GNUPlot process or pipe we have remain active. To be called when the
capture is restarted or rescanned.

For ALL the functions above, the infrastructure will guarantee that they
will never be called concurrently.
The following, last, function may be called concurrently to any of those
above.
void (*draw_graph)()   If there has been packets coming from a specific tap
during the last second or so, then
   the infrastructure will call this function to get it to (ask gnuplot to)
redraw the window.
   This function would be called for anyone that has had packets coming
their way about once a second or so.
   This part of the infrastructure is a prime candidate to implement as a
low-priority thread.




Thats the updated design proposal. Its difficult to explain properly since
its all in my head but i think it is improving a lot compared to the initial
design.
This design feels quite good and right.
I will await anyones comments on this new design.
If it gets clean and easy enough to use we might get a lot of people
producing extensions.

As a test to see if the final design will work I propose we do two simple
extensions such as
FileSizeHistogram
and something that prints similar output as
nfsstat
does.
These need not look great or be used at all in other ways than to just
verify that the design works and the API
is easy to use.


best regards
    ronnie sahlberg