[Kevin Johnson <kjohnson@xxxxxxxxxxxxxxx>]

FIX FOR NETXRAY TIMESTAMP DISPLAY PROBLEMS

This note is to describe / explain a fairly major rewrite of those portions of netxray.c concerned with assigning the correct timeunits (time ticks).  
The summary explains the actual changes.  This is followed by a more wordy narrative on the reasoning followed and the testing performed.  
Finally a diff of our proposed changes is attached for testing and potential inclusion in the Ethereal release code

SUMMARY:  The current netxray.c file defines the "parsing" instructions for reading files taken with name brand Sniffer products (which have been 
based on NetXray code for some time).  There is a problem with the way the variable "timeunit" is determined, namely that there are about 4 
different pre-selected values that can be assigned based on various features looked for in the capture file, and in other places even more timeunit 
numbers are used in variables that are part of the checking. 

It turns out all this is unnecessary - the actual correct timeunit value is recorded in the capture files by the sniffer.  This is a 32-bit value read in from 
offset 0x4c, reversing the order of the hex bytes as usual for this file (example, displayed as 100f963b but read as 3b960f10). 

Our changes do two things - they read in this value and assign it to the "realtick" variable, and because we've had to break apart the existing 64-byte 
array "xxb[]" to get that value, we have to fix the places where other values from xxb[] were being used.  We ended up only reading in 12 bytes to xxb[], 
then grabbing a 32-bit value for a new variable "realtick", and grabbing the remaining 48 bytes into the new array variable "xxj[]".  Various locations 
referencing xxb[20] had to be changed to reference xxj[4] for example.  The "timeunit" variable is assigned the value of "hdr.realtick" in everywhere possible.

This change fixes the recently reported problems from people trying to read captures taken with newer models of the brand-name Sniffer product, 
especially gigabit units, where Ethereal's display of the timestamps have been off by large amounts.  However, we believe the fix also improves the 
accuracy timestamp rendering for nearly all other Sniffer captures as we are no longer relying on rounded-off or fudged numbers.  This belief is
 bolstered by testing with dozens of captures taken with a variety of hardware and software versions, including ethernet, gigabit ethernet, token 
ring, FDDI, and frame relay captures. 

The only Sniffer captures where timestamp display is not improved are ATM captures made with the older ATM pods - and these are no worse than 
in the existing version of Ethereal - they simply have not changed.

NARRATIVE:  We regularly use distributed Sniffers to capture traffic as we're lucky enough to work for a company with a good budget for them.  
But we also use Ethereal to read the traces quite often for a variety of reasons most of you will understand.  So it was with some disappointment that
 we found Ethereal to be "broken" in its display of packet times when reading traces taken with the new S6040 model Sniffers with gigabit blades.  
Captures would appear to be 1000 times slower than they actually were.  Packets 0.005 seconds apart would be 5 whole seconds apart in the 
Ethereal display.

Reading through some of the previous E-mails on the developer and users mailing lists as well as the source-code comments, we began looking at 
our capture files in a hex editor.  The initial problem seemed simple - netxray.c was looking at a particular byte to figure out what kind of sniffer trace 
this was.  A value of "02" in that byte would have given us a timeunit of a billion (1e9), but our sniffer was inserting a "06" in that byte (this seemed to 
be interpreted as an ISDN HDLC capture).  This gave us a time unit of a million (1e6).  Editing the file to replace the 06 with 02 caused Ethereal to "fix" 
the time problem, but this did not solve the problem.  We would need to find a new way to evaluate the file type, or sniffer type, or *something* to understand 
that the more granular time unit had been used.

It was in reading comments in the source code regarding previous use of the value 1193180 that triggered the discovery of the location of the actual time 
tick value.  In a slightly older capture file I found that value (in the reverse-order hex) encoded at offset 0x04c.  I then checked my new capture files (which 
need a time tick closer to one billion) and found that the hex value there equates to around 999,970,000 (changes a little bit in some captures).

At first we tried retaining most of the "if" logic that is used to set "timeunit", and only altered the part that referred to version 2.002 captures with a gigabit 
pod, but of course in grabbing that value we ended up breaking the 64-byte array xxb[].  This required a little recoding to grab the bytes in before and 
after our timeunit value, and changing places that had referred to specific xxb[] values.  The resulting compiled binary correctly read our new captures 
without resorting to altering the byte which might indicate the capture type.

We then postulated that Sniffer has probably been using the same location for encoding the time ticks value for a long time, and that it might prove fruitful to 
try using that value ALL the time.  We did this, and tested against many captures taken with various versions of the Sniffer code over a 7 year period, and 
including ethernet, gigabit ethernet, token ring, FDDI, and various WAN types.  We do not seem to have broken or worsened Ethereal's ability to display the 
correct time for any of these, and in most cases have at least slightly improved the accuracy (as compared with the times displayed by the brand-name 
Sniffer product).  In the process we have also fixed the issue with newer gigabit captures being off by a factor of 1000.

There was one type of capture that was not improved, and is noticeably "off" compared to the actual Sniffer software, and that was ATM captures.  
Our only examples of these were 7 year old captures used in a Sniffer University course.  However, our "fixed" netxray.c code did not perform any 
differently than the original, so this problem has already existed.

As an interesting side note, there is a check being performed to determine whether or not to honor the FCS bits at the end of each packet.  The check
 is based on the hex values of two bytes - right in the middle of the time stamp.  It turns out these two bytes are only valid if the "real" time unit is
 near the 1193180 value, so it is likely Ethereal has been ignoring the FCS information when some folks think it has been used.  We did not attempt
 to fix this but did our best to duplicate the functionality of the original check.

In addition to the FCS check problem, we need to address some other issues including the actual meaning of the Capture Type byte, which used to 
be xxb[20] and is now xxj[4] - since the new gig sniffers are writing a "6" there, we have some bad assumptions in the code that may have other 
impacts for how we're interpreting the code.  We also need to tackle writing the time ticks into the correct place when Ethereal "saves as" 
the NetXray format.

We will continue to work on this file and hopefully will be able to submit further patches to continue to improve the functionality of Ethereal when 
paired with Sniffer brand sniffers.

We would appreciate comments and ideas on this patch.  We can be reached at either of the addresses below.

Thanks for your time,
James Fields and Kevin Johnson
jvfields@xxxxxxx and kjohnson@xxxxxxxxxxxxxxx

-------------------
BASE Project Lead
http://sourceforge.net/projects/secureideas
http://base.secureideas.net
The next step in IDS analysis!

--- netxray.c	2004-12-14 14:54:03.000000000 -0500
+++ /home/kjohnson/downloads/security/netxray.c	2004-12-30 18:19:07.749191552 -0500
@@ -62,11 +62,13 @@
 	guint32	timelo;		/* lower 32 bits of time stamp of capture start */
 	guint32	timehi;		/* upper 32 bits of time stamp of capture start */
 	guint32 linespeed;	/* speed of network, in bits/second */
-	guint8	xxb[64];	/* other stuff */
+	guint8	xxb[12];	/* other stuff */
+	guint32	realtick;	/* Actual timeticks from the sniffer */
+	guint8	xxj[48];	/* Other unknown stuff */
 };
 
 /*
- * Capture type, in xxb[20].
+ * Capture type, in xxj[4].
  */
 #define CAPTYPE_NDIS	0	/* Capture on network interface using NDIS */
 #define CAPTYPE_BROUTER	1	/* Bridge/router captured with pod */
@@ -320,75 +322,33 @@
 
 	/*
 	 * Figure out the time stamp units and start time stamp.
+	 * 
+	 * --- Below we set the timeunit to the hdr.realtick value
+	 * 	We were able to replace the various different checks on the
+	 * 	2_00x case with a single if due to the tick value being
+	 *	encoded into the capture.
 	 */
 	start_timestamp = (double)pletohl(&hdr.timelo)
 	    + (double)pletohl(&hdr.timehi)*4294967296.0;
 	switch (file_type) {
 
 	case WTAP_FILE_NETXRAY_OLD:
-		timeunit = 1000.0;
+		timeunit = hdr.realtick;
 		break;
 
 	case WTAP_FILE_NETXRAY_1_0:
-		timeunit = 1000.0;
+		timeunit = hdr.realtick;
 		break;
 
 	case WTAP_FILE_NETXRAY_1_1:
-		timeunit = 1000000.0;
+		timeunit = hdr.realtick;
 		break;
 
 	case WTAP_FILE_NETXRAY_2_00x:
-		/*
-		 * It appears that, at least for Ethernet
-		 * captures, if hdr.xxb[20] is 2, that indicates
-		 * that it's a gigabit Ethernet capture, possibly
-		 * from a special whizzo gigabit pod, and also
-		 * indicates that the time stamps have some
-		 * higher resolution than in other captures,
-		 * possibly thanks to a high-resolution
-		 * timer on the pod.
-		 *
-		 * It also appears that the time units might differ
-		 * for gigabit pod captures between version 002.001
-		 * and 002.002.
-		 */
-		if (network_type == 1 && hdr.xxb[20] == CAPTYPE_GIGPOD) {
-			if (version_minor == 1) {
-				/*
-				 * It appears that the time units for
-				 * these captures are nanoseconds, unless
-				 * hdr.timeunit is 2, in which case it's
-				 * 1/31250000.0 of a second.
-				 */
-				if (hdr.timeunit == 2)
-					timeunit = 31250000.0;
-				else
-					timeunit = 1e9;
-			} else {
-				/*
-				 * These just seem to be 1000 times
-				 * the regular time stamp units, unless
-				 * hdr.timeunit is 2, in which case it's
-				 * 1/1125000 of a second.
-				 *
-				 * In addition, the start timestamp
-				 * appears to be 0.
-				 */
-				if (hdr.timeunit == 2)
-					timeunit = 1250000.0;
-				else
-					timeunit = TpS[hdr.timeunit]*1000.0;
-				start_timestamp = 0.0;
-			}
-		} else {
-			if (hdr.timeunit > NUM_NETXRAY_TIMEUNITS) {
-				*err = WTAP_ERR_UNSUPPORTED;
-				*err_info = g_strdup_printf("netxray: Unknown timeunit %u",
-				    hdr.timeunit);
-				return -1;
-			}
-			timeunit = TpS[hdr.timeunit];
-		}
+		
+		timeunit = hdr.realtick;
+		if (hdr.timeunit != 2)
+			start_timestamp = 0.0;
 		break;
 
 	default:
@@ -404,12 +364,12 @@
 		 * frames (as a result, presumably, of having passed
 		 * through NDISWAN).
 		 *
-		 * In version 2, it looks as if there's stuff in the "xxb"
+		 * In version 2, it looks as if there's stuff in the "xxj"
 		 * words of the file header to specify what particular
 		 * type of WAN capture we have.
 		 */
 		if (version_major == 2) {
-			switch (hdr.xxb[20]) {
+			switch (hdr.xxj[4]) {
 
 			case CAPTYPE_PPP:
 				/*
@@ -430,7 +390,7 @@
 				/*
 				 * Various HDLC flavors?
 				 */
-				switch (hdr.xxb[28]) {
+				switch (hdr.xxj[12]) {
 
 				case 0:	/* LAPB/X.25 */
 					file_encap = WTAP_ENCAP_LAPB;
@@ -440,13 +400,13 @@
 				case 2:	/* T1 PRI */
 				case 3:	/* BRI */
 					file_encap = WTAP_ENCAP_ISDN;
-					isdn_type = hdr.xxb[28];
+					isdn_type = hdr.xxj[12];
 					break;
 
 				default:
 					*err = WTAP_ERR_UNSUPPORTED_ENCAP;
 					*err_info = g_strdup_printf("netxray: WAN HDLC capture subsubtype 0x%02x unknown or unsupported",
-					   hdr.xxb[28]);
+					   hdr.xxj[12]);
 					return -1;
 				}
 				break;
@@ -461,7 +421,7 @@
 			default:
 				*err = WTAP_ERR_UNSUPPORTED_ENCAP;
 				*err_info = g_strdup_printf("netxray: WAN capture subtype 0x%02x unknown or unsupported",
-				   hdr.xxb[20]);
+				   hdr.xxj[4]);
 				return -1;
 			}
 		} else
@@ -515,11 +475,17 @@
 		 *
 		 * XXX - is that actually a 2-byte little-endian 0x1234?
 		 * What does that field signify?
+                 *
+                 * XXX - The field appears to be the tick value for the capture
+                 * If it is 0xDE341200 then it is the older version and
+                 * appears to be hard coded.
 		 */
+
 		if (version_major == 2) {
-			if (hdr.xxb[13] == 0x34 && hdr.xxb[14] == 0x12)
+			if (hdr.realtick == 0xDE341200)
 				wth->capture.netxray->fcs_valid = TRUE;
 		}
+
 		break;
 	}
 
@@ -1412,24 +1378,24 @@
     switch (wdh->encap) {
 
     case WTAP_ENCAP_PPP_WITH_PHDR:
-	file_hdr.xxb[20] = CAPTYPE_PPP;
+	file_hdr.xxj[4] = CAPTYPE_PPP;
 	break;
 
     case WTAP_ENCAP_FRELAY_WITH_PHDR:
-	file_hdr.xxb[20] = CAPTYPE_FRELAY;
+	file_hdr.xxj[4] = CAPTYPE_FRELAY;
 	break;
 
     case WTAP_ENCAP_LAPB:
-	file_hdr.xxb[20] = CAPTYPE_HDLC;
-	file_hdr.xxb[28] = 0;
+	file_hdr.xxj[4] = CAPTYPE_HDLC;
+	file_hdr.xxj[12] = 0;
 	break;
 
     case WTAP_ENCAP_SDLC:
-	file_hdr.xxb[20] = CAPTYPE_SDLC;
+	file_hdr.xxj[4] = CAPTYPE_SDLC;
 	break;
 
     default:
-	file_hdr.xxb[20] = CAPTYPE_NDIS;
+	file_hdr.xxj[4] = CAPTYPE_NDIS;
 	break;
     }
 
@@ -1448,3 +1414,4 @@
 
     return TRUE;
 }
+