[Jirka Novak <j.novak@xxxxxxxxxxxx>]

Hello Peter,

>> I tried to run make in top directory and it failed for first try :-) I
>> checked the output and found that something changed in DNS dissector and
>> dns.flags.recdesired .. value="1" was in past and now value="FFFFFFFF".
>> It is probably about signed/unsigned value - not important now.
>> But this issue pointed to me more things:
>> 3) There should be easy way how to describe which wireshark version
>> processed stored PDML file.
>> Probably different outputs should be stored for main branches (2.0, 2.2,
>> ...). It is obvious that each version of wireshark will change (improve)
>> some outputs. Therefore older release will not be able to process it
>> correctly (PDML diff will fail).
>> I think that without output version you can't run regression tests
>> automatically.
> 
> Branches can work, but given that there is only little testdata for now,
> it seems better to track the master branch directly.

I don't think branches are good approach - you are not able to compare
same sample between versions.

>> 5) There should be easy way how to describe which filter should be used
>> for preprocessing PDML. Now only filter.xsl is used.
> 
> XSL is currently the only available format because it is powerful given
> the XML nature. Are there any particular things that are lacking in XSL?

I think that PDML and TEXT output should be stored/compared. TEXT
contains e.g. info column which is not shown in PDML.
Any other format can be added in future - e.g. automated GUI scripts and
outputs as mentioned by Alexis and Graham.

> For now you can file pull requests on the happy-shark project on Github.
> It is still in experimental phase, nothing is set in stone and things
> can change as you see them fit :-)

I prepared scripts and Makefile which express my idea. I attached
modified README.md where I add description of new features. There is
list of files in my test repository to demonstrate future content.
Any comments are welcomed.

./common/copy-all.xsl
./README.md
./Makefile
./tests/dns/qr-1/qr-1.description.txt
./tests/dns/qr-1/qr-1_2.3.pdml
./tests/dns/qr-1/qr-1_2.0.text
./tests/dns/qr-1/qr-1_2.3.text
./tests/dns/qr-1/qr-1.pdml.current
./tests/dns/qr-1/qr-1_2.0.pdml
./tests/dns/qr-1/qr-1.requirements.txt
./tests/dns/qr-1/qr-1.text.current
./tests/dns/qr-1/qr-1.pcapng.gz
./tests/dns/qr-1/filter.xsl
./tests/dns/dns-1/dns-1.requirements.txt
./tests/dns/dns-1/dns-1_2.2.pdml
./tests/dns/dns-1/dns-1_2.0.pdml
./tests/dns/dns-1/dns-1.description.txt
./tests/dns/dns-1/dns-1_2.3.text
./tests/dns/dns-1/dns-1.text.current
./tests/dns/dns-1/dns-1_2.0.text
./tests/dns/dns-1/dns-1.pdml.current
./tests/dns/dns-1/filter.xsl
./tests/dns/dns-1/dns-1.pcapng.gz
./tests/rtp.ed137a/ed137a-1/ed137a-1_2.3.pdml
./tests/rtp.ed137a/ed137a-1/ed137a-1.text.current
./tests/rtp.ed137a/ed137a-1/ed137a-1_2.3.text
./tests/rtp.ed137a/ed137a-1/ed137a-1.requirements.txt
./tests/rtp.ed137a/ed137a-1/ed137a-1.pdml.current
./tests/rtp.ed137a/ed137a-1/ed137a-1.description.txt
./tests/rtp.ed137a/ed137a-1/ed137a-1_2.0.text
./tests/rtp.ed137a/ed137a-1/ed137a-1_2.0.pdml
./tests/rtp.ed137a/ed137a-1/ed137a-1.args
./tests/rtp.ed137a/ed137a-1/ed137a-1.pcapng.gz
./tests/rtp.ed137a/ed137a-1/filter.xsl
./scripts/sample_test.sh
./scripts/sample_make_output.sh
./scripts/sample_verify.sh

						Sincerely yours,

							Jirka Novak

happy-shark
===========
Happy Shark is the regression test framework for Wireshark consisting of a tool
and a collection of capture files.

Running tests
-------------
Simply invoke `make` or `make -j4` for more parallelism.

Adding a new test
-----------------
Create a new subdirectory under `tests/<protocol>/`. <protocol> is name of protocol
subjected to test (e.g. 'dns' or 'rtp.ed137a'). Use same name as display filter in wireshark.
Directory populate with the following files:

 - FOO.pcap.gz or FOO.pcapng.gz - the source file (noncompressed files shall not be used, e.g. 'dns-1.pcapng.gz')
 - FOO.description - description of purpose the file is included (e.g. basic DNS query or RTP header with ED-137A header extension)
 - FOO.requirements - requirements, how tshark/wireshark should process the file (e.g. packet should be decoded by specification of ED-137B, PTT and SQL bits should be shown in packet info column). Requirements can describe non dissector related staff too (e.g. RTP Stream Analysis window should show warning about bad packet sequence, warning should be shown in yellow color).
 - FOO.args - optional file, contains options for tshark to process file as expected (e.g. 'decode as' parameters)
 - FOO_<version>.pdml - the expected processed output from <version> of tshark in PDML format (e.g. `dns-1_2.0.pcapng.pdml`). Only first two levels of version number are used.
 - FOO_<version>.text - the expected processed output from <version> of tshark in TEXT format (e.g. `dns-1_2.0.pcapng.text`). Only first two levels of version number are used.
 - FOO.no_pdml - optional file expressing that PDML output should not be checked (requirements probably describe GUI related requiremens only)
 - FOO.no_text - optional file expressing that TEXT output should not be checked (requirements probably describe GUI related requiremens only)
 - filter.xsl - the post-processor.

Run make outputs to generate .pdml and .text and make verify_repository to check all required files before commit.

When proposing a new test, please include the source of the packet capture file
in the commit message. The source could be a link to https://bugs.wireshark.org/
or https://wiki.wireshark.org/SampleCaptures for example. Try to keep capture
files small and specific to a small number of protocols.

Options and variables to run framework
--------------------------------------

SUPPORTED_VERSIONS - list of versions checked during make or make outputs, when not specified, default in Makefile is used
VERSION - version used for make or make outputs, when not specified, tshark version is used
TSHARK_EXECUTABLE - path to tshark, when not specified, tshark in PATH is used

make or make test - run tests, compare output of latest stored .pdml and .text
make VERSION=2.0 - same as above, compare output with version 2.0 or previous
make test_pdml or make test_text - run tests for PDML or TEXT output only
make verify_repository - check whether each sample contains required files for at least one of checked versions
make verify_repository VERSION=2.0 - same above, check is made for specified version only
make outputs - generate .pdml and .text output for samples where files are missing, version is derived from version of used tshark
make outputs TSHARK_EXECUTABLE=path/tshark - same as above, but you can determine used tshark
make clean - removes temporary files after make test

Architecture
------------
The initial desired features were:

 - Matching fields (the displayed text, byte offsets and length).
 - Take a packet capture file and produce the expected "output".
 - Have a filter that strips layers or just keeps a single layer.
 - Allow preferences to be applied (SSL keys, port numbers, ...).
 - Maybe check both single and second pass mode (tshark -2) to catch issues
   related to maintained state within a dissector.

License
-------
This project including the tools and capture files are provided under the terms
of version 2 of the GPL or any later version.