[John Powell <jrp999@xxxxxxxxx>]

Hi Everyone,

I am using DUMPCAP to capture packets in a high packet rate environment.

My operating system is: CENTOS 6.3

I am experience this problem on source compiled versions:  wireshark-1.6.12 and wireshark-1.8.4.

In order to allow DUMPCAP to be run as a NON-ROOT user I am using the following:

• setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /usr/local/bin/dumpcap -v
  

The issue is that I am experiencing packet loss to apparent disk contention when writing the packets to the disk - see attached file: packet-loss-atop.txt

To help alleviate the problem I have tried the following:

• Disabled SELINUX
• Disabled AUDIT
  
• RAID 0 (striped disks) to load share the writing out of the data
• ARRAY /dev/md2 level=raid0 num-devices=2
     devices=/dev/sda4,/dev/sdb4
• Turn off journals on ext4
• tune2fs -o journal_data_writeback /dev/md2
• tune2fs -O ^has_journal /dev/md2
• change fstab to:
• UUID=.. /data   ext4    defaults,data="" style>         0 0
• Use -B option on Dumpcap to buffer the data
• root      /usr/local/bin/dumpcap -B 16 -i 2 -f vlan and (not vrrp and not udp port 1985 and not ether host 01:00:0c:cc:cc:cc) -g -b filesize:250000 -b duration:900 -w /data/eth1.cap
  

These changes have increased the throughput but I still experience packet loss - see attached IO Graph: packet-loss-io-graph.jpg

The Vendor solutions we have looked at will not decode UNISTIM signalling properly which is requirement for this tool.

Any suggestions on how to better configure either the operating system or wireshark to increase packet capture throughput will be greatly appreciated.

Thanks in advance for your assistance.

-John


ATOP - stc0033281                                  2012/12/12  12:35:16                                  ------                                    1s elapsed
PRC | sys    0.23s | user   0.18s | #proc    216  | #trun      2 | #tslpi   315 | #tslpu     1 | #zombie    1 | clones     0  |              | #exit      0 |
CPU | sys      17% | user     13% | irq       1%  | idle    116% | wait     53% |              | steal     0% | guest     0%  | curf 2.58GHz | curscal  81% |
cpu | sys      14% | user     12% | irq       1%  | idle     69% | cpu001 w  4% |              | steal     0% | guest     0%  | curf 2.00GHz | curscal  63% |
cpu | sys       5% | user      1% | irq       0%  | idle     49% | cpu000 w 44% |              | steal     0% | guest     0%  | curf 3.17GHz | curscal 100% |
CPL | avg1    0.80 | avg5    0.80 |               | avg15   0.81 |              | csw    25366 | intr   16698 |               |              | numcpu     2 |
MEM | tot     3.6G | free  155.2M | cache   3.1G  | dirty  84.2M | buff    2.3M | slab  196.6M |              |               |              |              |
SWP | tot     5.9G | free    5.0G |               |              |              |              |              |               | vmcom   1.7G | vmlim   7.7G |
PAG | scan    3456 |              | stall      0  |              |              |              |              | swin       0  |              | swout      0 |
MDD |          md2 | busy      0% | read       1  | write  15442 | KiB/r      4 | KiB/w      4 | MBr/s   0.00 | MBw/s  60.32  | avq     0.00 | avio 0.00 ms |
DSK |          sda | busy    107% | read       1  | write    205 | KiB/r      4 | KiB/w    506 | MBr/s   0.00 | MBw/s 101.33  | avq    93.88 | avio 4.51 ms |
DSK |          sdb | busy     92% | read       0  | write    191 | KiB/r      0 | KiB/w    511 | MBr/s   0.00 | MBw/s  95.50  | avq    86.84 | avio 4.20 ms |
NET | transport    | tcpi       2 | tcpo       2  | udpi       0 | udpo       0 | tcpao      0 | tcppo      0 | tcprs      0  | tcpie      0 | udpip      0 |
NET | network      | ipi        3 | ipo        2  | ipfrw      0 | deliv      2 |              |              |               | icmpi      0 | icmpo      0 |
NET | eth1      7% | pcki   40678 | pcko       0  | si   71 Mbps | so    0 Kbps | coll       0 | erri       0 | erro       0  | drpi       0 | drpo       0 |
NET | eth0      0% | pcki       3 | pcko       2  | si    2 Kbps | so   14 Kbps | coll       0 | erri       0 | erro       0  | drpi       0 | drpo       0 |
  PID                        RDDSK                        WRDSK                       WCANCL                        DSK                       CMD         1/1
 1123                           0K                        9892K                           0K                       100%                       dumpcap
  991                           4K                          36K                           0K                         0%                       flush-9:2
   18                           0K                           4K                           0K                         0%                       sync_supers
27471                           0K                           0K                           0K                         0%                       nautilus
 2842                           0K                           0K                           0K                         0%                       nautilus
 1133                           0K                           0K                           0K                         0%                       atop
 1128                           0K                           0K                           0K                         0%                       dumpcap
 2960                           0K                           0K                           0K                         0%                       wireshark
   38                           0K                           0K                           0K                         0%                       kswapd0
 1870                           0K                           0K                           0K                         0%                       kondemand/0

Attachment: packet-loss-io-graph.JPG
Description: JPEG image