["Bob DeBolt" <bob.debolt@xxxxxxxxxxxxxx>]

Greets

To begin, forgive me if this has some rambling to it, a car crash a week
ago
has left my concentration rather off, but improving.
 
I have been having an issues for a number of months now relating 
to the ISP of a client. 

The current situation is this:

1) Head office
2) Primary satellite office
3) Secondary Satellite office

The satellite offices of the client installed VOIP and have
had a ton of grief with it, not an uncommon story I'm sure.
The VOIP system has dropped TOO many calls and is in the process of
being 
replaced with the original PSTN.

The head office and the primary satellite office are connected to the
ISP 
through several miles of wireless. 

The traffic to and from the gateway of each of the primary offices own 
gateways check good with ICMP, TCP and UDP tests using hping, pchar,
PingPlotter
and several others that I forget as several months have past since last
used.

Repeated tests to each others gateway and to each of the internal IP
addresses of the route to each other shows intermittent packet loss. 
 

The core router of the ISP is a Cisco 7200. The clients maximum
throughput has not 
been an issue at any time. There is a 3 to 10 second "hesitation" across
the network as the 
7200 rereads the routing tables, processor is at %100 during the period,
as told by the ISP.

The ISP has stated that Cisco says this delay is normal for the routing
table reload.


Connections between the satellite offices will drop connections usually
during the 
measured "hesitation", but not always. When the VOIP system was in place
and several times 
during this hesitation, with active Citrix sessions, I observed no
interruption in service 
or voice hesitation. 

During off hours the session drops have been as frequent as during main
business hours. 
I have often been dropped while doing remote service via SSH to either
office location
at any time of day or night, through the IPSec tunnel or not. It does
tend to die in 
early-mid afternoon more than any other time of day even though my
clients bandwidth has
lots of room left.

The bandwidth at the primary satellite office is a shared radio between
several other clients
which resulted in some very serious over billing charges being made to
my client. 

As the ISP has indicated to me and I have observed, the method employed
by the ISP to diagnose 
the packet drop / latency is the Cisco routers "ping".

The satellite offices are connected through OpenBSD 3.6 stable IPSec
tunnels. 
The VOIP was NOT running through the IPSec Tunnels at any time, in fact 
with the VOIP system turned off there is no difference in the packet
loss graphs as 
indicated by Ping Plotter ICMP mode. The Citrix sessions, whether or not
running through 
the tunnels, traffic shows the same packet loss, session dropping
characteristics.

With only one Citrix client connected to the head office late at night
via Internet 
there was packet loss whose timing matched the Citrix session.  

Each piece of equipment including the firewalls have been completely
swapped out
with the same results.

I do not have direct access to the ISP equipment but have run a number
of end to 
end tests with programs like pchar, hping etc., currently setting up a
smokeping 
box to give an executive view of problems.

Using pchar as the example, the tests, there has been a couple dozen of
them have all 
shown very similar results as shown below.

pchar to X.X.118.174 (X.X.118.174) using UDP/IPv4
Using raw socket input
Packet size increments from 32 to 1500 by 32
46 test(s) per repetition
32 repetition(s) per hop
 0: X.X.117.25 (X.X.117.25)
    Partial loss:      0 / 1472 (0%)
    Partial char:      rtt = 3.699518 ms, (b = 0.004700 ms/B), r2 =
0.999498
                       stddev rtt = 0.012836, stddev b = 0.000016
    Partial queueing:  avg = 0.001132 ms (240 bytes)
    Hop char:          rtt = 3.699518 ms, bw = 1701.956678 Kbps
    Hop queueing:      avg = 0.001132 ms (240 bytes)
 1: 172.18.8.94 (172.18.8.94)
    Partial loss:      2 / 1472 (0%)
    Partial char:      rtt = 8.557417 ms, (b = 0.006275 ms/B), r2 =
0.987962
                       stddev rtt = 0.084370, stddev b = 0.000104
    Partial queueing:  avg = 0.031362 ms (19442 bytes)
    Hop char:          rtt = 4.857899 ms, bw = 5081.703107 Kbps
    Hop queueing:      avg = 0.030230 ms (19202 bytes)
 2: 10.64.8.13 (10.64.8.13)
    Partial loss:      743 / 1472 (50%)
    Partial char:      rtt = 10.569593 ms, (b = 0.006782 ms/B), r2 =
0.982938
                       stddev rtt = 0.108841, stddev b = 0.000135
    Partial queueing:  avg = 0.003006 ms (19442 bytes)
    Hop char:          rtt = 2.012176 ms, bw = 15770.949211 Kbps
    Hop queueing:      avg = -0.028356 ms (0 bytes)
 3: 10.64.8.38 (10.64.8.38)
    Path length:       3 hops
    Path char:         rtt = 10.569593 ms r2 = 0.982938
    Path bottleneck:   1701.956678 Kbps
    Path pipe:         2248 bytes
    Path queueing:     average = 0.003006 ms (19442 bytes)
    Start time:        Thu Feb  3 08:41:37 2005
    End time:          Thu Feb  3 09:36:07 2005

I realize this is a large piece of pie to look at but my mind has gone
numb working on this.

I am open to suggestions as far as dealing with the ISP and a better
tools selection / useage. 
It is the only client network I work on that has this issue and each of
my clients use OpenBSD 
stable / IPSec. Coincidentally it is the only client using the ISP.

I am at a point where I need to refocus the effort, I'm sure many of you
know what I am 
talking about. Any suggestions / flames are welcome, I picked the
Ethereal list to send this to as
I have followed this list for sometime now and see by answers there is
considerable experience.
I am hoping that one of you can help me to kick start his process again.

There is likely some very important info I'm missing here, hopefully my
concentration will improve in
another week or two.


Bob