I was wondering whether there's a Network guru around who's brain I can pick.
I'm running a small ISP and I'm using custom build Linux routers to
get the job done but I've started noticing some oddities on the
network for the past week. We have around 650Mbp/s of total Internet
Capacity but usually averages between 450 to 550Mbp/s during the day
but I started seeing network spikes every 10 to 20 seconds pushing my
usage on my core firewall to over 1GB but only for a few seconds after
which it drops back down to normal.
The other strange part is , I'm only seeing the bandwidth spikes on my
core router between my inner (eth5) and outer (eth0) Interface. These
traffic spikes isn't visible on my edge router where my upstream
providers flow into. I'm also not seeing these spikes on any of my
MRTG graphs. Perhaps I'm missing something here but in my
understanding my core Firewalls traffic should match the traffic on my
edge firewall right ?
So in short , my outgoing flow on eth5 which is essentially incoming
traffic for my customers will spike to 1Gb but the increase traffic
isn't visible on my edge router .
My outer (eth0) Interface plugs into my bandwidth manager which in
turn plugs into my edge routers inner (eno2) where all our upstream
providers are flowing into (eno1) . I've done a few packet captures
with tcpdump and imported it into Wireshark but I'm not really 100%
what to look for.
Any ideas on what might be causing these spikes would be greatly appreciated. .
I used Nload on each interface to get some realtime statistics.
The below values were recorded on my Core firewall.
Eth0 (Outer Interface , goes to Bandwidth manager which goes to my
edge firewall (eno2)
Nload stats on interface Eth0
Curr: 301.95 MBit/s
Avg: 431.78 MBit/s
Min: 0.00 Bit/s
Max: 1.43 GBit/s
Ttl: 787808.69 GByte
Eth5 (Inner Interface , plugs into a switch which feeds my customers)
Nload stats on interface eth5
Curr: 385.77 MBit/s
Avg: 399.40 MBit/s
Min: 262.46 MBit/s
Max: 1.29 GBit/s
Ttl: 27408.75 GByte
And these values were recorded on my Inner interface (eno2) of my edge router.
eno1 WAN (This is my Wan Interface to the outside world)
Edge Firewall / router (Centos 7);
eno2 LAN (This interface plugs into my bandwidth manager which in turn
plugs into my Core firewalls outer interface(eth0)
Nload stats on interface eno2
Curr: 336.97 MBit/s
Avg: 381.66 MBit/s
Min: 309.01 MBit/s
Max: 548.93 MBit/s
Ttl: 504389.72 GByte