perfSONAR

Running measurement tools

BWCTL

BWCTL is a policy daemon that allows the user to invoke a series of other tools (iperf, iperf3, nuttcp, ping, owamp, traceroute, tracepath) via a set of commands.  The BWCTL daemon handles permissions to run the test, as well as serialized scheduling to prevent tests from running at the same time. 

To invoke BWCTL with the iperf3 tool, the following command can be used:

[zurawski@bnl-pt1 ~]$ bwctl -T iperf3 -f m -t 10 -i 1 -c llnl-pt1.es.net
bwctl: Using tool: iperf3
bwctl: 36 seconds until test results available

SENDER START
Connecting to host 198.129.254.106, port 5843
[ 18] local 198.124.238.38 port 41955 connected to 198.129.254.106 port 5843
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[ 18]   0.00-1.00   sec  4.98 MBytes  41.7 Mbits/sec    0    987 KBytes       
[ 18]   1.00-2.00   sec  71.7 MBytes   601 Mbits/sec    0   12.7 MBytes       
[ 18]   2.00-3.00   sec   441 MBytes  3701 Mbits/sec    0   42.8 MBytes       
[ 18]   3.00-4.00   sec   546 MBytes  4581 Mbits/sec    0   42.8 MBytes       
[ 18]   4.00-5.00   sec   532 MBytes  4467 Mbits/sec    2   43.2 MBytes       
[ 18]   5.00-6.00   sec   541 MBytes  4541 Mbits/sec    0   43.2 MBytes       
[ 18]   6.00-7.00   sec   539 MBytes  4519 Mbits/sec    0   43.2 MBytes       
[ 18]   7.00-8.00   sec   545 MBytes  4572 Mbits/sec    0   43.2 MBytes       
[ 18]   8.00-9.00   sec   545 MBytes  4571 Mbits/sec    0   43.2 MBytes       
[ 18]   9.00-10.00  sec   544 MBytes  4560 Mbits/sec    0   43.2 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[ 18]   0.00-10.00  sec  4.21 GBytes  3616 Mbits/sec    2             sender
[ 18]   0.00-10.00  sec  4.21 GBytes  3616 Mbits/sec                  receiver

iperf Done.

SENDER END

There are numerous BWCTL command line options, some popular ones are:

  1. The "-f m" flag sets the output to Mbps
  2. The "-T" flag sets the tool (iperf, iperf3, nuttcp).  Note that older versions may not support all tools
  3. The "-t" flag sets the length of the test.  Note that policy may prevent a longer test
  4. The "-i" flag sets the reporting interval of the results
  5. The "-x" flag shows the server and client side of the test
  6. The "-c" flag sets the host that will receive the data
  7. The "-s" flag sets the host that will send the data
  8. If both "-c" and "-s" are specified to remote hosts, you initiate what is called a "3rd party" test between them
  9. The "-w" flag can be used to set the window size
  10. The "-P" flag can be used to start parallel streams 

An example of a test using the historical iperf tool follows:

[zurawski@bnl-pt1 ~]$ bwctl -T iperf -f m -t 10 -i 1 -c llnl-pt1.es.net
bwctl: Using tool: iperf
bwctl: 36 seconds until test results available

RECEIVER START
------------------------------------------------------------
Server listening on TCP port 5111
Binding to local address 198.129.254.106
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ 14] local 198.129.254.106 port 5111 connected with 198.124.238.38 port 5111
[ ID] Interval       Transfer     Bandwidth
[ 14]  0.0- 1.0 sec  4.88 MBytes  40.9 Mbits/sec
[ 14]  1.0- 2.0 sec  58.7 MBytes   492 Mbits/sec
[ 14]  2.0- 3.0 sec   447 MBytes  3749 Mbits/sec
[ 14]  3.0- 4.0 sec   605 MBytes  5072 Mbits/sec
[ 14]  4.0- 5.0 sec   595 MBytes  4994 Mbits/sec
[ 14]  5.0- 6.0 sec   603 MBytes  5058 Mbits/sec
[ 14]  6.0- 7.0 sec   603 MBytes  5059 Mbits/sec
[ 14]  7.0- 8.0 sec   595 MBytes  4994 Mbits/sec
[ 14]  8.0- 9.0 sec   591 MBytes  4961 Mbits/sec
[ 14]  9.0-10.0 sec   606 MBytes  5081 Mbits/sec
[ 14]  0.0-10.0 sec  4714 MBytes  3948 Mbits/sec
[ 14] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

RECEIVER END

Another test using the nuttcp tool follows:

[zurawski@bnl-pt1 ~]$ bwctl -T nuttcp -f m -t 10 -i 1 -c hous-pt1.es.net
bwctl: Using tool: nuttcp
bwctl: 36 seconds until test results available

SENDER START
nuttcp-t: v7.1.6: socket
nuttcp-t: buflen=65536, nstream=1, port=5310 tcp -> 198.129.254.50
nuttcp-t: time limit = 10.00 seconds
nuttcp-t: connect to 198.129.254.50 with mss=8948, RTT=42.169 ms
nuttcp-t: send window size = 94800, receive window size = 87380
nuttcp-t: available send window = 71100, available receive window = 65535
nuttcp-r: v7.1.6: socket
nuttcp-r: buflen=65536, nstream=1, port=5310 tcp
nuttcp-r: interval reporting every 1.00 second
nuttcp-r: accept from 198.124.238.38
nuttcp-r: send window size = 98720, receive window size = 87380
nuttcp-r: available send window = 74040, available receive window = 65535
   32.2500 MB /   1.00 sec =  270.5242 Mbps     0 retrans
  734.5000 MB /   1.00 sec = 6161.3217 Mbps     3 retrans
 1035.1250 MB /   1.00 sec = 8683.4489 Mbps     0 retrans
 1050.7500 MB /   1.00 sec = 8814.3651 Mbps     0 retrans
 1086.2500 MB /   1.00 sec = 9112.1801 Mbps     0 retrans
 1084.1250 MB /   1.00 sec = 9093.9632 Mbps     0 retrans
 1077.8750 MB /   1.00 sec = 9041.9794 Mbps     0 retrans
 1065.8750 MB /   1.00 sec = 8941.3685 Mbps     0 retrans
 1049.6250 MB /   1.00 sec = 8804.8134 Mbps     0 retrans
 1039.9375 MB /   1.00 sec = 8723.7502 Mbps     0 retrans
nuttcp-t: 9257.0000 MB in 10.00 real seconds = 947911.21 KB/sec = 7765.2886 Mbps
nuttcp-t: 9257.0000 MB in 8.45 CPU seconds = 1121700.12 KB/cpu sec
nuttcp-t: retrans = 3
nuttcp-t: 148112 I/O calls, msec/call = 0.07, calls/sec = 14811.11
nuttcp-t: 0.0user 8.4sys 0:10real 84% 0i+0d 768maxrss 0+3pf 47+19csw

nuttcp-r: 9257.0000 MB in 10.08 real seconds = 940528.20 KB/sec = 7704.8070 Mbps
nuttcp-r: 9257.0000 MB in 6.92 CPU seconds = 1368843.20 KB/cpu sec
nuttcp-r: 189403 I/O calls, msec/call = 0.05, calls/sec = 18792.67
nuttcp-r: 0.0user 6.8sys 0:10real 68% 0i+0d 772maxrss 0+5pf 37781+36csw

SENDER END

There are extensions to BWCTL that allow it to operate on other tools.  The BWPING tool can run ping and owping, the first example shows this operating with ping:

[zurawski@aofa-pt1 ~]$ bwping -c nersc-pt1.es.net
bwping: Using tool: ping
bwping: 74 seconds until test results available

SENDER START
PING 198.129.254.22 (198.129.254.22) from 198.124.238.42 : 56(84) bytes of data.
64 bytes from 198.129.254.22: icmp_seq=1 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=2 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=3 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=4 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=5 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=6 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=7 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=8 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=9 ttl=56 time=68.9 ms
64 bytes from 198.129.254.22: icmp_seq=10 ttl=56 time=68.9 ms

--- 198.129.254.22 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9080ms
rtt min/avg/max/mdev = 68.958/68.970/68.993/0.262 ms

SENDER END

This allows the concept of a 'reverse' test when the "-s" flag is used.  Note that if the other end does not have BWCTL you can only run the test in one direction (and must specify the -E flag):

a[zurawski@aofa-pt1 ~]$ bwping -E -c www.psu.edu
bwping: Using tool: ping
bwping: 38 seconds until test results available

SENDER START
PING 2610:8:4000:50:146:186:157:6(2610:8:4000:50:146:186:157:6) from 2001:400:6001:1150::3 : 56 data bytes
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=1 ttl=57 time=17.6 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=2 ttl=57 time=15.2 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=3 ttl=57 time=15.2 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=4 ttl=57 time=14.9 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=5 ttl=57 time=15.2 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=6 ttl=57 time=15.1 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=7 ttl=57 time=15.0 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=8 ttl=57 time=15.0 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=9 ttl=57 time=15.4 ms
64 bytes from 2610:8:4000:50:146:186:157:6: icmp_seq=10 ttl=57 time=15.1 ms

--- 2610:8:4000:50:146:186:157:6 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9026ms
rtt min/avg/max/mdev = 14.988/15.417/17.616/0.747 ms

SENDER END

OWAMP can be invoked the same way, and by default it will only test the direction you ask:

[zurawski@aofa-pt1 ~]$ bwping -T owamp -s lhcmon.bnl.gov
bwping: Using tool: owamp
bwping: 32 seconds until test results available

SENDER START
Approximately 13.1 seconds until results available

--- owping statistics from [lhcmon.bnl.gov]:8955 to [198.124.238.42]:5168 ---
SID:	c67cee2ad78502dc46af825288cb7bce
first:	2014-07-31T11:01:02.047
last:	2014-07-31T11:01:15.420
10 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 1.29/1.3/1.33 ms, (err=0.167 ms)
one-way jitter = 0.1 ms (P95-P50)
Hops = 4 (consistently)
no reordering


SENDER END

A different extension, BWTRACEROUTE, will allow the use of the traceroute and tracepath tools.  The same caveats apply as with BWPING: if the endpoint is not running BWCTL the "-E" flag must be used and the "-s" direction allows for a reverse test:

[zurawski@wash-pt1 ~]$ bwtraceroute -s hous-pt1.es.net
bwtraceroute: Using tool: traceroute
bwtraceroute: 38 seconds until test results available

SENDER START
traceroute to 198.124.238.34 (198.124.238.34), 30 hops max, 60 byte packets
 1  houscr1-houspt1.es.net (198.129.254.49)  0.740 ms  1.168 ms  1.529 ms
 2  nashcr5-ip-a-houscr5.es.net (134.55.39.50)  16.578 ms  16.884 ms  17.133 ms
 3  atlacr5-ip-a-nashcr5.es.net (134.55.38.73)  22.773 ms  23.088 ms  23.435 ms
 4  washcr5-ip-a-atlacr5.es.net (134.55.42.70)  35.692 ms  36.000 ms  36.249 ms
 5  wash-pt1.es.net (198.124.238.34)  35.327 ms  35.404 ms  35.469 ms

SENDER END

Tracepath is also possible with this tool:

[zurawski@wash-pt1 ~]$ bwtraceroute -T tracepath -s hous-pt1.es.net
bwtraceroute: Using tool: tracepath
bwtraceroute: 38 seconds until test results available

SENDER START
 1:  hous-pt1.es.net (198.129.254.50)                       0.077ms pmtu 9000
 1:  houscr1-houspt1.es.net (198.129.254.49)                1.041ms 
 1:  houscr1-houspt1.es.net (198.129.254.49)                0.611ms 
 2:  nashcr5-ip-a-houscr5.es.net (134.55.39.50)            16.630ms 
 3:  atlacr5-ip-a-nashcr5.es.net (134.55.38.73)            22.816ms 
 4:  washcr5-ip-a-atlacr5.es.net (134.55.42.70)            35.732ms 
 5:  wash-pt1.es.net (198.124.238.34)                      35.668ms reached
     Resume: pmtu 9000 hops 5 back 60 

SENDER END

OWAMP

The OWPING tool is invoked from the command line to run an OWAMP test:

[zurawski@bnl-owamp ~]$ owping sacr-owamp.es.net
Approximately 13.3 seconds until results available

--- owping statistics from [2001:400:6401:1110::3]:8934 to [sacr-owamp.es.net]:9027 ---
SID:	00000003d784fc6306f66dbc17b49b7e
first:	2014-07-31T10:33:24.415
last:	2014-07-31T10:33:33.419
100 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 36.4/36.4/37.1 ms, (err=0.284 ms)
one-way jitter = 0.1 ms (P95-P50)
Hops = 7 (consistently)
no reordering


--- owping statistics from [sacr-owamp.es.net]:9080 to [2001:400:6401:1110::3]:8786 ---
SID:	00000003d784fc63237bce01ee6b57f5
first:	2014-07-31T10:33:24.452
last:	2014-07-31T10:33:35.199
100 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 34.4/34.5/34.6 ms, (err=0.284 ms)
one-way jitter = 0 ms (P95-P50)
Hops = 7 (consistently)
no reordering

There will be two sets of results (for each independent direction).  A report will be available that features the number of packets sent, lost, and duplicated as well as the delays and jitters calculated. 

Other options to consider are when running this test are:

  1. The "-c" option to increase the number of packets sent.  For example the default is to send 100 packets, "-c 1000" will send 1000 packets. 
  2. The "-i" option to increase (or decrease) the inter-arrival time of packets.  For example the default is to send packets spaced at a rate of 10 packets per second (e.g. "-i .1").  To send 100 packets per second, use "-i .01". 
  3. The "-s" option can be used to increase the size of the packets sent.  For example -s 8000 will add 8000 bytes in padding to the test packets. 

If the test does not complete for one (or both) directions, ensure that TCP port 861 (OWAMP control) is not blocked, along with the values that are set in /etc/owampd/owampd.conf for the "testports" variable; these are UDP test packets. 

Traceroute/Tracepath

Traceroute and Tracepath are used to determine the number of network layer (OSI Layer 3) hops between a source and a destination.  An example of traceroute operation follows:

[zurawski@bnl-owamp ~]$ traceroute sacr-owamp.es.net
traceroute to sacr-owamp.es.net (198.129.254.18), 30 hops max, 60 byte packets
 1  bnlmr2-bnlowamp.es.net (198.124.238.50)  0.218 ms  0.202 ms  0.185 ms
 2  newycr5-ip-a-bnlmr2.es.net (134.55.221.134)  3.303 ms  3.727 ms  4.064 ms
 3  starcr5-ip-a-newycr5.es.net (134.55.49.30)  28.365 ms  28.591 ms  28.827 ms
 4  chiccr5-ip-a-starcr5.es.net (134.55.42.41)  32.710 ms  32.987 ms  33.223 ms
 5  kanscr5-ip-a-chiccr5.es.net (134.55.43.82)  39.602 ms  39.863 ms  40.142 ms
 6  denvcr5-ip-a-kanscr5.es.net (134.55.49.57)  50.188 ms  50.160 ms  50.448 ms
 7  sacrcr5-ip-a-denvcr5.es.net (134.55.50.201)  71.197 ms  71.089 ms  71.374 ms
 8  sacr-owamp.es.net (198.129.254.18)  70.763 ms  70.757 ms  70.737 ms

Each "hop" is represented as an IP address, and DNS lookup if available.  Three probes (default behavior) will be present after each hop noting the recorded round trip latency.  It is important to note that some devices, such as routers and switches, treat ICMP measurement packets with a TTL of 0 (the default for tools like traceroute) very differently.  It is not uncommon to see this latency number appear as "high", if the network device delayed answering. 

An example of tracepath follows:

[zurawski@bnl-owamp ~]$ tracepath sacr-owamp.es.net
 1:  bnl-owamp.es.net (198.124.238.49)                      0.265ms pmtu 1500
 1:  bnlmr2-bnlowamp.es.net (198.124.238.50)                0.311ms asymm  2 
 1:  bnlmr2-bnlowamp.es.net (198.124.238.50)                0.326ms asymm  2 
 2:  newycr5-ip-a-bnlmr2.es.net (134.55.221.134)            2.883ms 
 3:  starcr5-ip-a-newycr5.es.net (134.55.49.30)            28.890ms 
 4:  chiccr5-ip-a-starcr5.es.net (134.55.42.41)            29.147ms 
 5:  kanscr5-ip-a-chiccr5.es.net (134.55.43.82)            39.760ms 
 6:  denvcr5-ip-a-kanscr5.es.net (134.55.49.57)            54.900ms 
 7:  sacrcr5-ip-a-denvcr5.es.net (134.55.50.201)           71.383ms 
 8:  sacr-owamp.es.net (198.129.254.18)                    71.076ms reached
     Resume: pmtu 1500 hops 8 back 57 

This tool is similar in behavior in that it represents each hop, but will also try to record the path MTU (and when this changes), along with a notion of when there is an asymmetry detected.  This may be a common message if a path features many LAG for the connectivity. 

If either tool does not succeed, it may be due to the probes being blocked by portions of the path.  It is common for networks to block ICMP packets (the default probe type for some measurement tools).  Consider using options to change the probe type to TCP or UDP if you notice that your tests are being blocked. 

Ping

Ping is a round trip measurement tool that uses ICMP packets (or UDP, or TCP if instructed to do so) to establish the length of time to send data between a source and destination.  An example of this tool follows:

a[zurawski@bnl-owamp ~]$ ping -c 5 sacr-owamp.es.net
PING sacr-owamp.es.net (198.129.254.18) 56(84) bytes of data.
64 bytes from sacr-owamp.es.net (198.129.254.18): icmp_seq=1 ttl=57 time=70.8 ms
64 bytes from sacr-owamp.es.net (198.129.254.18): icmp_seq=2 ttl=57 time=70.8 ms
64 bytes from sacr-owamp.es.net (198.129.254.18): icmp_seq=3 ttl=57 time=70.8 ms
64 bytes from sacr-owamp.es.net (198.129.254.18): icmp_seq=4 ttl=57 time=70.7 ms
64 bytes from sacr-owamp.es.net (198.129.254.18): icmp_seq=5 ttl=57 time=70.8 ms

--- sacr-owamp.es.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4069ms
rtt min/avg/max/mdev = 70.777/70.813/70.850/0.024 ms

Ping reports the output of each probe, which consists of a packet being sent that has a certain size and TTL value, and the length of time it took to complete the process. 

There are numerous options available for Ping to increase the number of packets sent, the size of packets (useful for debugging MTU problems), and if packets should be fragmented.  Please read the man pages for your particular variety. 

TCPDump/TCPTrace/XPlot

An overview on these tools is available at https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/tcpdump-tcptrace/.

Mininet

Mininet creates an instant virtual network on a single host.  A tutorial in using perfSONAR on miniet can be found here:

perfSONAR & Mininet Tutorial (798.0 KB)