[Paul Offord <Paul.Offord@xxxxxxxxxxxx>]

I see several problems with doing dumpcap first:

 

• Once the dumpcap code is finished you’d still not be able to do anything new and, probably more importantly, you’d have no way of testing that the code is working correctly
• I already have a dissector that works which I can quickly submit to the project once the TSDB piece (and perhaps TRB piece) is done
• Developers (or users) will be able to use TribeLab Workbench to create sample files to test the TSDB, TRB and dissector code
• I think getting dumpcap to do the job is going to take quite some effort
  • Not impossible but lots to consider
  • Workbench uses Data Descriptor XML files to describe the log file layout (see below)
  • I imagine we would want dumpcap to do some level of heuristic discovery of record layouts rather than need descriptors like this

 

Best regards…Paul

 

<?xml version="1.0" encoding="utf-8" standalone="yes"?>

<source>

               <header headerline="false" skipheaderlines="0">

                              <description>Descriptor file for Apache access log in common format</description>

                              <generator>Babel 3.0</generator>

                              <gendate>2017-10-20</gendate>

                              <gentime>19:18:22</gentime>

                              <genzoffset>+1</genzoffset>

                              <owner>Paul Offord</owner>

                              <nativeformat>LogFormat "%h %l %u %t \"%r\" %>s %b" common</nativeformat>

                              <example>192.168.1.87 - paulo [09/Jul/2012:08:25:35 +0100] "GET /Setup.php HTTP/1.1" 200 1824</example>

                              <wsnamespace>apache</wsnamespace>

                              <charencoding>ASCII</charencoding>

               </header>

               <records>

                              <record type="1">

                                             <eols enforce="true">

                                                            <eol>\n</eol>

                                                            <eol>\r\n</eol>

                                             </eols>

                                             <delimiters>

                                                            <delimiter>&nbsp;</delimiter>

                                             </delimiters>

                                             <missingvalues>

                                                            <missingvalue>-</missingvalue>

                                             </missingvalues>

                                             <criteria>

                                                            <criterium type="string" offset="*">*</criterium>

                                             </criteria>

                                             <columns>

                                                            <column>

                                                                           <informat quoted="false">%i</informat>

                                                                           <name>host</name>

                                                                           <abbrev>bds.apache.host</abbrev>

                                                                           <blurb>This is the IP address of the client (remote host) which made the request to the server.</blurb>

                                                                           <type quoted="false">FT_IPvx</type>

                                                                           <display>BASE_NONE</display>

                                                                           <bitmask>0</bitmask>

                                                            </column>

                                                            <column>

                                                                           <informat quoted="false">%s</informat>

                                                                           <name>identid</name>

                                                                           <abbrev>bds.apache.identid</abbrev>

                                                                           <blurb>The identity of the client determined by a request to the identd server on the clients machine.</blurb>

                                                                           <type quoted="false">FT_STRINGZ</type>

                                                                           <display>BASE_NONE</display>

                                                                           <bitmask>0</bitmask>

                                                            </column>

                                                            <column>

                                                                           <informat quoted="false">%s</informat>

                                                                           <name>userid</name>

                                                                           <abbrev>bds.apache.userid</abbrev>

                                                                           <blurb>This is the userid of the person requesting the document as determined by HTTP authentication.</blurb>

                                                                           <type quoted="false">FT_STRINGZ</type>

                                                                           <display>BASE_NONE</display>

                                                                           <bitmask>0</bitmask>

                                                            </column>

                                                            <column>

                                                                           <informat quoted="false" start-bracket="[" end-bracket="]">[%d/%b/%Y:%H:%M:%S %z]</informat>

                                                                           <name>datetime</name>

                                                                           <abbrev>bds.apache.datetime</abbrev>

                                                                           <blurb>The time that the request was received.</blurb>

                                                                           <type>EVENT_DATETIME</type>

                                                                           <display>BASE_NONE</display>

                                                                           <bitmask>0</bitmask>

                                                            </column>

                                                            <column>

                                                                           <informat quoted="true">%s</informat>

                                                                           <name>request</name>

                                                                           <abbrev>bds.apache.request</abbrev>

                                                                           <blurb>The request line from the client is given in double quotes.</blurb>

                                                                           <type>FT_STRINGZ</type>

                                                                           <display>BASE_NONE</display>

                                                                           <bitmask>0</bitmask>

                                                            </column>

                                                            <column>

                                                                           <informat quoted="false">%d</informat>

                                                                           <name>response code</name>

                                                                           <abbrev>bds.apache.response-code</abbrev>

                                                                           <blurb>This is the status code that the server sends back to the client.</blurb>

                                                                           <type>FT_UINT32</type>

                                                                           <display>BASE_DEC</display>

                                                                           <bitmask>0</bitmask>

                                                            </column>

                                                            <column>

                                                                           <informat quoted="false">%d</informat>

                                                                           <name>bytes returned</name>

                                                                           <abbrev>bds.apache.sc-bytes</abbrev>

                                                                           <blurb>This indicates the size of the object returned to the client, not including the response headers.</blurb>

                                                                           <type>FT_UINT32</type>

                                                                           <display>BASE_DEC</display>

                                                                           <bitmask>0</bitmask>

                                                            </column>

                                             </columns>

                                             <infofield>%4 - %5</infofield>

                              </record>

               </records>

</source>

 

 

From: Wireshark-dev [mailto:wireshark-dev-bounces@xxxxxxxxxxxxx] On Behalf Of Roland Knall
Sent: 03 November 2017 16:52
To: Developer support list for Wireshark <wireshark-dev@xxxxxxxxxxxxx>
Subject: Re: [Wireshark-dev] Capture filename not available at plugin init time

 

Quite a few breweries I assume ;-) 

 

The real question here is dumpcap. That should be done first. Over the years, there was an effort to get this done every so months. But most people seem to give up silently.

 

From my own experience of messing with ByteView, I can wholeheartedly understand the reasoning

 

cheers

 

On Fri, Nov 3, 2017 at 5:49 PM, Paul Offord <Paul.Offord@xxxxxxxxxxxx> wrote:

Probably a whole brewery!

 

From: Wireshark-dev [mailto:wireshark-dev-bounces@xxxxxxxxxxxxx] On Behalf Of Graham Bloice
Sent: 03 November 2017 16:48

To: Developer support list for Wireshark <wireshark-dev@xxxxxxxxxxxxx>
Subject: Re: [Wireshark-dev] Capture filename not available at plugin init time

 

 

 

On 3 November 2017 at 16:40, Paul Offord <Paul.Offord@xxxxxxxxxxxx> wrote:

So the plan would be:

 

• Add support to read the TSDB and create the resulting structures
• Add support to read Text Record Blocks (TRBs)

• This is mostly stuff that Guy Harris described a while back
• In my current code the data records are encapsulated in a dummy Ethernet frame

• Add support to mergecap to correctly handle the TSDBs

• Similar to adjusting IDBs when files are merged

• Add the dumpcap code to read text files and produce pcap-ng

 

I think I’ll leave this idea to circulate for a few days before I start writing code.  Maybe I’ll pester the developers next week at SF EU.

 

That sounds like it will need a lot of beer tokens ;-/

 

 

I will also see if there is an easy way to get plugin_if_get_ws_info to work in the init routine as I still believe this will be useful to dissector developers.

 

Thanks and regards…Paul

 

From: Wireshark-dev [mailto:wireshark-dev-bounces@xxxxxxxxxxxxx] On Behalf Of Roland Knall
Sent: 03 November 2017 16:24

To: Developer support list for Wireshark <wireshark-dev@xxxxxxxxxxxxx>
Subject: Re: [Wireshark-dev] Capture filename not available at plugin init time

 

This is a different thing here. If TSDB is a common code block, I think the chances are really good.

 

But still it needs the basic read functionality in dumpcap

 

cheers

 

On Fri, Nov 3, 2017 at 5:18 PM, Paul Offord <Paul.Offord@xxxxxxxxxxxx> wrote:

OK – I understand.

 

If I write the code to read the TSDB and make it available do you think it would be accepted into the main project?  I’m thinking about my syncro experience here.

 

Thanks and regards…Paul

 

From: Wireshark-dev [mailto:wireshark-dev-bounces@xxxxxxxxxxxxx] On Behalf Of Roland Knall
Sent: 03 November 2017 14:15

To: Developer support list for Wireshark <wireshark-dev@xxxxxxxxxxxxx>
Subject: Re: [Wireshark-dev] Capture filename not available at plugin init time

 

Hi Paul

 

You should never assume, that you will be able to read the file, while WS is reading it. If this is working right now, it might be out of pure coincidence, that said, the real thing here should be to get dumpcap to use pcapng as input format, which would give you the tsdb block where you need it to be, during dissection.

 

The support for any pcap-ng extension block is already in Wireshark. The issue still is, to get the block structure through.

 

cheers

Roland 

 

On Fri, Nov 3, 2017 at 3:07 PM, Paul Offord <Paul.Offord@xxxxxxxxxxxx> wrote:

Thanks for responding Roland.

 

I’ve written a tool that reads a log file and converts it to a PCAP-NG with a matching dissector.  The pcap file carries a data descriptor block in a new PCAP-NG block type called as TSDB.  The TSDB carries the information needed to register the header fields.  To add support for the TSDB into core Wireshark is going to be a big job (which I will submit later).  As a quick solution, the dissector gets the information by directly opening and reading it from the PCAP-NG file – hence the need for the filename.

 

The above aside, and I can guess you are thinking I’m just trying to avoid a bigger coding job, it’s not unreasonable to expect plugin_if_get_ws_info to get the filename at init time, and init is called when the filename is known.

 

I have a really kludgie workaround which is to read the TSDB and register the hf structure on the first call to the dissector, but it’s not ideal.

 

Best regards…Paul

 

From: Wireshark-dev [mailto:wireshark-dev-bounces@xxxxxxxxxxxxx] On Behalf Of Roland Knall
Sent: 03 November 2017 13:53
To: Developer support list for Wireshark <wireshark-dev@xxxxxxxxxxxxx>
Subject: Re: [Wireshark-dev] Capture filename not available at plugin init time

 

Hi Paul

 

As far as I know, cf_open can still fail after calling the init-functions. In that case you would get the filename, but the capture is already closed.

 

My question is, why do you need the filename in the first place?

 

Also, you could set the filename at a later point. If you implement a tap-interface, you could set the filename in the first tap-print callback. Makes sense, 'cause you normally only have data at this point anyway.

 

You can raise this as an improvement (do not think it is a bug) if you want to, not really sure though, if it should be changed

 

cheers

 

 

On Fri, Nov 3, 2017 at 2:49 PM, Paul Offord <Paul.Offord@xxxxxxxxxxxx> wrote:

I have a dissector that needs the capture file name at the time my dissector’s init function is called.  I attempt to get the name with plugin_if_get_ws_info(…), not an unreasonable request I think you’ll agree, but unfortunately the filename comes back as a NULL pointer.

 

I’ve traced through the code and this is what happens:

 

• We pass through the MainWindow signal and slot stuff and eventually call cf_open(…) in file.c with the filename as one of the parameters
• cf_open(…) opens the file to test the validity of the filename and then closes with cf_close(cf)
• cf_close(cf) frees the memory holding the filename and NULLs the filename pointer in the cf structure
• cf_open then creates a new epan session with ws_epan_new(cf)
• ws_epan_new(cf) calls epan_new() which calls init_dissection() and this is where eventually my dissector’s init function gets called
• My dissector calls plugin_if_get_ws_info(…) which attempts to get the filename info from the cf structure, which due to the above returns a NULL filename pointer
• Eventually we return back to cf_open(…) and a little later we set up the file name in the cf structure – all too late for my dissector’s init function

 

So my questions are:

 

• Can I raise this as a bug?
• If not, would a solution that made the filename available to plugin_if_get_ws_info(…) at init time be accepted?
• What would be an acceptable solution?

 

Thanks and regards…Paul

______________________________________________________________________

This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev@xxxxxxxxxxxxx>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request@xxxxxxxxxxxxx?subject=unsubscribe

 

______________________________________________________________________

This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev@xxxxxxxxxxxxx>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request@xxxxxxxxxxxxx?subject=unsubscribe

 

______________________________________________________________________

This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev@xxxxxxxxxxxxx>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request@xxxxxxxxxxxxx?subject=unsubscribe

 

______________________________________________________________________

This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev@xxxxxxxxxxxxx>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request@xxxxxxxxxxxxx?subject=unsubscribe

 

--

Graham Bloice

Software Developer

Trihedral UK Limited

______________________________________________________________________

This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev@xxxxxxxxxxxxx>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request@xxxxxxxxxxxxx?subject=unsubscribe

 

______________________________________________________________________

This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________