|
|
|
Tenet test their 10 Gb/s Seacom bandwidth
|
|
Date: 2009/08/13 08:48
|
By: admin
|
Status: Admin
|
|
|
|
|
Admin
| Posts: 58 |  |
|
|
http://en.wikipedia.org/wiki/Iperf
This is for all those techies following this forum.
Tenet test Seacom's 10 Gb/s bandwidth using iperf
From: Andrew Alston [mailto:*******@tenet.ac.za]
Sent: 28 July 2009 7:30 PM
To: ren-news@lists.tenet.ac.za
Subject: [REN-news] Seacom - The first test results
Hi All,
As has already been stated elsewhere, the Seacom cable was brought live on Thursday of last week. At the same time, TENET succeeded in bringing our backhaul from the landing station back to Durban University of Technology live, and we have been testing the Seacom capacity since this for stability, throughput, latency and other factors. What follows is fairly technical, and below it I will include a less technical summary of the results so far.
While I will not address the time frames for distribution of Seacom bandwidth beyond our Durban point of presence in this email, I thought the initial test results that we have seen would be of interest to those of you on this list.
So, a breakdown of what is in place, what has been done and what we have seen so far.
On Wednesday morning, we started work with the DWDM Equipment from the Landing station to DUT, via our Optical Amplifer in Umhlanga. Connectivity all the way through was up and running by Thursday afternoon, and at 4:32pm on Thursday we saw the connectivity establish between our London router and the core router at DUT. I need to give a special mention to the Cisco team, DFA, I.S and Ashwin at DUT who all worked with us during this period to make this a success, their assistance was invaluable.
At approximately 8pm on Thursday we established the first routing over the Seacom cable, and a test IP Segment was announced from Durban to London.
Then, using a single server equipped with a single gigabit network card, tests began.
The latency indication on the cable from DUT to London is approximately 215ms. I plan to send out a fairly detailed email about latency and performance tuning at a later stage as well to explain how latency can impact TCP throughput and how systems can be tuned to handle latency.
The TCP throughput was tested, first with no tweaking involved in the IP Stack and the TCP Window sizes, and initial indication were that using standard configurations you would find on a default windows desktop, a standard single tcp session gave a throughput of approximately 8mbit. However, it was possible to run a LOT of combined sessions at this speed to get far far higher throughput. We also tested under Linux machines which by default have far better TCP tuning when dealing with latencies, and were seeing speeds of up to 100mbit/second per session. (I will talk a bit about latency and tcp stack tuning in a bit).
We then went ahead and started adjusting TCP settings to allow for very high speed single session throughput, and were able to run single TCP sessions in excess of 900mbit/second. On a 12 hour test using a single TCP session between machines with adjusted sessions, we succeeded in transferring a total of 3.6 terabytes from London to South Africa at an average speed of 84.2 Mbytes/second (673mbit/second)
On UDP sessions, we were running at approximately 700mbit/second with no packet reordering and no packet loss.
It should be noted that the limiting factor of all of these tests was the speed at which the server we were using for testing could run. Due to only having a single gigabit NIC that was NOT on a PCI-E Bus, we were hitting the limits of the machine long before we hit the limits of the bandwidth on the Seacom capacity itself.
In tests using the server and another two notebooks all running in parallel, we pushed the line to a little over 2gigabit, without any degradation of performance on the latency.
We also established there were no SDH errors on the line on either input or output packets, which is actually fairly abnormal for SDH capacity (Since SDH allows for a certain error rate by standard, and providing that the error rate stays at below 1:10^7 you are within normal operating limits)
The cable has also been 100% stable since Thursday night with no drops.
In Summary:
In testing the limiting factors were our test machines, the seacom bandwidth is simply to fast to saturate without an entire bank of servers or machines running tests.
We were able to completely saturate the capacity of a test server using a single tcp session after adjusting the stack.
Without adjusting the stack, using multiple sessions, the same saturation of the test server was possible.
I realize that this email is of a very technical nature, and those who do have questions about the content please do not hesitate to contact me.
Many Thanks
Andrew Alston
TENET – Chief Technology Officer
What follows are some test output samples:
With a 16meg TCP Window size, single session IPERF test:
------------------------------------------------------------
Client connecting to 196.32.210.12, TCP port 5001
TCP window size: 15.3 MByte (default)
------------------------------------------------------------
[ 3] local 155.232.9.11 port 45057 connected with 196.32.210.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-15.0 sec 372 MBytes 208 Mbits/sec
[ 3] 15.0-30.0 sec 518 MBytes 290 Mbits/sec
[ 3] 30.0-45.0 sec 608 MBytes 340 Mbits/sec
[ 3] 45.0-60.0 sec 785 MBytes 439 Mbits/sec
[ 3] 60.0-75.0 sec 1464 MBytes 819 Mbits/sec
[ 3] 75.0-90.0 sec 940 MBytes 526 Mbits/sec
[ 3] 90.0-105.0 sec 623 MBytes 348 Mbits/sec
[ 3] 105.0-120.0 sec 664 MBytes 372 Mbits/sec
[ 3] 120.0-135.0 sec 641 MBytes 358 Mbits/sec
[ 3] 135.0-150.0 sec 669 MBytes 374 Mbits/sec
[ 3] 150.0-165.0 sec 1017 MBytes 569 Mbits/sec
[ 3] 165.0-180.0 sec 1585 MBytes 886 Mbits/sec
[ 3] 180.0-195.0 sec 1600 MBytes 895 Mbits/sec
[ 3] 195.0-210.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 210.0-225.0 sec 1011 MBytes 565 Mbits/sec
[ 3] 225.0-240.0 sec 705 MBytes 394 Mbits/sec
[ 3] 240.0-255.0 sec 768 MBytes 430 Mbits/sec
[ 3] 255.0-270.0 sec 839 MBytes 469 Mbits/sec
[ 3] 270.0-285.0 sec 1350 MBytes 755 Mbits/sec
[ 3] 285.0-300.0 sec 1611 MBytes 901 Mbits/sec
[ 3] 300.0-315.0 sec 1591 MBytes 890 Mbits/sec
[ 3] 315.0-330.0 sec 1597 MBytes 893 Mbits/sec
[ 3] 330.0-345.0 sec 1603 MBytes 897 Mbits/sec
[ 3] 345.0-360.0 sec 1595 MBytes 892 Mbits/sec
[ 3] 360.0-375.0 sec 1601 MBytes 895 Mbits/sec
[ 3] 375.0-390.0 sec 1598 MBytes 893 Mbits/sec
[ 3] 390.0-405.0 sec 1607 MBytes 899 Mbits/sec
[ 3] 405.0-420.0 sec 1592 MBytes 890 Mbits/sec
[ 3] 420.0-435.0 sec 1603 MBytes 896 Mbits/sec
[ 3] 435.0-450.0 sec 1599 MBytes 894 Mbits/sec
[ 3] 450.0-465.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 465.0-480.0 sec 1600 MBytes 895 Mbits/sec
[ 3] 480.0-495.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 495.0-510.0 sec 1609 MBytes 900 Mbits/sec
[ 3] 510.0-525.0 sec 1591 MBytes 890 Mbits/sec
[ 3] 525.0-540.0 sec 1595 MBytes 892 Mbits/sec
[ 3] 540.0-555.0 sec 1610 MBytes 901 Mbits/sec
[ 3] 555.0-570.0 sec 1604 MBytes 897 Mbits/sec
[ 3] 570.0-585.0 sec 1591 MBytes 890 Mbits/sec
[ 3] 585.0-600.0 sec 1596 MBytes 893 Mbits/sec
[ 3] 600.0-615.0 sec 1608 MBytes 899 Mbits/sec
[ 3] 615.0-630.0 sec 1599 MBytes 894 Mbits/sec
[ 3] 630.0-645.0 sec 1587 MBytes 887 Mbits/sec
[ 3] 645.0-660.0 sec 1601 MBytes 896 Mbits/sec
[ 3] 660.0-675.0 sec 1609 MBytes 900 Mbits/sec
[ 3] 675.0-690.0 sec 1588 MBytes 888 Mbits/sec
[ 3] 690.0-705.0 sec 1608 MBytes 899 Mbits/sec
[ 3] 705.0-720.0 sec 1589 MBytes 889 Mbits/sec
[ 3] 720.0-735.0 sec 1612 MBytes 902 Mbits/sec
[ 3] 735.0-750.0 sec 1588 MBytes 888 Mbits/sec
[ 3] 750.0-765.0 sec 1610 MBytes 900 Mbits/sec
[ 3] 765.0-780.0 sec 1588 MBytes 888 Mbits/sec
[ 3] 780.0-795.0 sec 1607 MBytes 899 Mbits/sec
[ 3] 795.0-810.0 sec 1606 MBytes 898 Mbits/sec
[ 3] 810.0-825.0 sec 1591 MBytes 890 Mbits/sec
[ 3] 825.0-840.0 sec 1606 MBytes 898 Mbits/sec
[ 3] 840.0-855.0 sec 1597 MBytes 893 Mbits/sec
[ 3] 855.0-870.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 870.0-885.0 sec 1594 MBytes 891 Mbits/sec
[ 3] 885.0-900.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 900.0-915.0 sec 1606 MBytes 898 Mbits/sec
[ 3] 915.0-930.0 sec 1597 MBytes 893 Mbits/sec
[ 3] 930.0-945.0 sec 1597 MBytes 893 Mbits/sec
[ 3] 945.0-960.0 sec 1597 MBytes 893 Mbits/sec
[ 3] 960.0-975.0 sec 1604 MBytes 897 Mbits/sec
[ 3] 975.0-990.0 sec 1599 MBytes 894 Mbits/sec
[ 3] 990.0-1005.0 sec 1597 MBytes 893 Mbits/sec
[ 3] 1005.0-1020.0 sec 1604 MBytes 897 Mbits/sec
[ 3] 1020.0-1035.0 sec 1593 MBytes 891 Mbits/sec
[ 3] 1035.0-1050.0 sec 1612 MBytes 901 Mbits/sec
[ 3] 1050.0-1065.0 sec 1599 MBytes 894 Mbits/sec
[ 3] 1065.0-1080.0 sec 1591 MBytes 890 Mbits/sec
[ 3] 1080.0-1095.0 sec 1602 MBytes 896 Mbits/sec
[ 3] 1095.0-1110.0 sec 1599 MBytes 894 Mbits/sec
[ 3] 1110.0-1125.0 sec 1603 MBytes 896 Mbits/sec
[ 3] 1125.0-1140.0 sec 1601 MBytes 895 Mbits/sec
[ 3] 1140.0-1155.0 sec 1603 MBytes 896 Mbits/sec
[ 3] 1155.0-1170.0 sec 1603 MBytes 896 Mbits/sec
[ 3] 1170.0-1185.0 sec 1609 MBytes 900 Mbits/sec
[ 3] 1185.0-1200.0 sec 1595 MBytes 892 Mbits/sec
[ 3] 1200.0-1215.0 sec 1605 MBytes 898 Mbits/sec
[ 3] 1215.0-1230.0 sec 1591 MBytes 890 Mbits/sec
[ 3] 1230.0-1245.0 sec 1614 MBytes 903 Mbits/sec
[ 3] 1245.0-1260.0 sec 1590 MBytes 889 Mbits/sec
[ 3] 1260.0-1275.0 sec 1605 MBytes 898 Mbits/sec
[ 3] 1275.0-1290.0 sec 1609 MBytes 900 Mbits/sec
[ 3] 1290.0-1305.0 sec 1589 MBytes 889 Mbits/sec
[ 3] 1305.0-1320.0 sec 1607 MBytes 899 Mbits/sec
[ 3] 1320.0-1335.0 sec 1598 MBytes 893 Mbits/sec
[ 3] 1335.0-1350.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 1350.0-1365.0 sec 1612 MBytes 902 Mbits/sec
[ 3] 1365.0-1380.0 sec 1602 MBytes 896 Mbits/sec
[ 3] 1380.0-1395.0 sec 1602 MBytes 896 Mbits/sec
[ 3] 1395.0-1410.0 sec 1596 MBytes 892 Mbits/sec
[ 3] 1410.0-1425.0 sec 1607 MBytes 899 Mbits/sec
[ 3] 1425.0-1440.0 sec 1587 MBytes 888 Mbits/sec
[ 3] 1440.0-1455.0 sec 1610 MBytes 900 Mbits/sec
[ 3] 1455.0-1470.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 1470.0-1485.0 sec 1602 MBytes 896 Mbits/sec
[ 3] 1485.0-1500.0 sec 1602 MBytes 896 Mbits/sec
[ 3] 1500.0-1515.0 sec 1595 MBytes 892 Mbits/sec
[ 3] 1515.0-1530.0 sec 1606 MBytes 898 Mbits/sec
[ 3] 1530.0-1545.0 sec 1593 MBytes 891 Mbits/sec
[ 3] 1545.0-1560.0 sec 1607 MBytes 899 Mbits/sec
[ 3] 1560.0-1575.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 1575.0-1590.0 sec 1608 MBytes 899 Mbits/sec
[ 3] 1590.0-1605.0 sec 1606 MBytes 898 Mbits/sec
[ 3] 1605.0-1620.0 sec 1597 MBytes 893 Mbits/sec
[ 3] 1620.0-1635.0 sec 1598 MBytes 894 Mbits/sec
[ 3] 1635.0-1650.0 sec 1602 MBytes 896 Mbits/sec
[ 3] 1650.0-1665.0 sec 1599 MBytes 894 Mbits/sec
[ 3] 1665.0-1680.0 sec 1604 MBytes 897 Mbits/sec
[ 3] 1680.0-1695.0 sec 1605 MBytes 898 Mbits/sec
[ 3] 1695.0-1710.0 sec 1601 MBytes 895 Mbits/sec
[ 3] 1710.0-1725.0 sec 1599 MBytes 894 Mbits/sec
[ 3] 1725.0-1740.0 sec 1600 MBytes 895 Mbits/sec
[ 3] 1740.0-1755.0 sec 1602 MBytes 896 Mbits/sec
[ 3] 1755.0-1770.0 sec 1608 MBytes 899 Mbits/sec
[ 3] 1770.0-1785.0 sec 1602 MBytes 896 Mbits/sec
[ 3] 1785.0-1800.0 sec 1601 MBytes 895 Mbits/sec
[ 3] 0.0-1800.0 sec 179400 MBytes 836 Mbits/sec
IPerf 900 second UDP Test, set to 750mbit/second (Please note, the slight packet loss and reordering you are seeing on these tests is because of server saturation and is well within normal limits for UDP)
[ 3] local 155.232.9.11 port 5001 connected with 196.32.210.12 port 47330
[ 3] 0.0-15.0 sec 1.28 GBytes 732 Mbits/sec 0.010 ms 671/934133 (0.072%)
[ 3] 0.0-15.0 sec 1 datagrams received out-of-order
[ 3] 15.0-30.0 sec 1.28 GBytes 735 Mbits/sec 0.010 ms 179/937251 (0.019%)
[ 3] 30.0-45.0 sec 1.27 GBytes 727 Mbits/sec 0.010 ms 464/927987 (0.05%)
[ 3] 45.0-60.0 sec 1.27 GBytes 725 Mbits/sec 0.008 ms 1820/926208 (0.2%)
[ 3] 60.0-75.0 sec 1.25 GBytes 717 Mbits/sec 0.008 ms 4002/918175 (0.44%)
[ 3] 75.0-90.0 sec 1.28 GBytes 733 Mbits/sec 0.004 ms 1899/936517 (0.2%)
[ 3] 90.0-105.0 sec 1.27 GBytes 727 Mbits/sec 0.009 ms 3079/930147 (0.33%)
[ 3] 105.0-120.0 sec 1.26 GBytes 722 Mbits/sec 0.009 ms 4558/925298 (0.49%)
[ 3] 120.0-135.0 sec 1.26 GBytes 722 Mbits/sec 0.010 ms 3118/924208 (0.34%)
[ 3] 135.0-150.0 sec 1.27 GBytes 728 Mbits/sec 0.011 ms 1077/929785 (0.12%)
[ 3] 150.0-165.0 sec 1.28 GBytes 731 Mbits/sec 0.011 ms 1688/933738 (0.18%)
[ 3] 165.0-180.0 sec 1.02 GBytes 582 Mbits/sec 0.009 ms 187174/929868 (20%)
[ 3] 180.0-195.0 sec 1.22 GBytes 701 Mbits/sec 0.010 ms 37672/932202 (4%)
[ 3] 195.0-210.0 sec 1.28 GBytes 731 Mbits/sec 0.009 ms 529/932523 (0.057%)
[ 3] 210.0-225.0 sec 1.28 GBytes 732 Mbits/sec 0.009 ms 680/934856 (0.073%)
[ 3] 225.0-240.0 sec 1.28 GBytes 733 Mbits/sec 0.010 ms 210/935704 (0.022%)
[ 3] 240.0-255.0 sec 1.27 GBytes 728 Mbits/sec 0.008 ms 927/929025 (0.1%)
[ 3] 255.0-270.0 sec 1.28 GBytes 730 Mbits/sec 0.010 ms 1089/932780 (0.12%)
[ 3] 270.0-285.0 sec 1.27 GBytes 729 Mbits/sec 0.008 ms 918/931070 (0.099%)
[ 3] 285.0-300.0 sec 1.27 GBytes 730 Mbits/sec 0.009 ms 891/931857 (0.096%)
[ 3] 300.0-315.0 sec 1.26 GBytes 723 Mbits/sec 0.009 ms 3423/925518 (0.37%)
[ 3] 300.0-315.0 sec 1 datagrams received out-of-order
[ 3] 315.0-330.0 sec 1.27 GBytes 727 Mbits/sec 0.009 ms 2370/930183 (0.25%)
[ 3] 330.0-345.0 sec 1.28 GBytes 733 Mbits/sec 0.008 ms 99/934931 (0.011%)
[ 3] 345.0-360.0 sec 1.26 GBytes 723 Mbits/sec 0.009 ms 3180/925405 (0.34%)
[ 3] 360.0-375.0 sec 1.27 GBytes 725 Mbits/sec 0.010 ms 931/925972 (0.1%)
[ 3] 375.0-390.0 sec 1.28 GBytes 730 Mbits/sec 0.006 ms 0/931603 (0%)
[ 3] 390.0-405.0 sec 1.28 GBytes 733 Mbits/sec 0.010 ms 1092/935898 (0.12%)
[ 3] 405.0-420.0 sec 1.26 GBytes 723 Mbits/sec 0.013 ms 2571/924464 (0.28%)
[ 3] 420.0-435.0 sec 1.27 GBytes 726 Mbits/sec 0.010 ms 1396/927780 (0.15%)
[ 3] 435.0-450.0 sec 1.25 GBytes 714 Mbits/sec 0.009 ms 5528/916068 (0.6%)
[ 3] 450.0-465.0 sec 1.28 GBytes 732 Mbits/sec 0.009 ms 410/934124 (0.044%)
[ 3] 465.0-480.0 sec 1.27 GBytes 727 Mbits/sec 0.009 ms 7420/935224 (0.79%)
[ 3] 480.0-495.0 sec 1.28 GBytes 732 Mbits/sec 0.008 ms 1118/934212 (0.12%)
[ 3] 495.0-510.0 sec 1.27 GBytes 728 Mbits/sec 0.010 ms 1091/929787 (0.12%)
[ 3] 510.0-525.0 sec 1.26 GBytes 721 Mbits/sec 0.009 ms 9382/928684 (1%)
[ 3] 525.0-540.0 sec 1.28 GBytes 730 Mbits/sec 0.002 ms 461/931850 (0.049%)
[ 3] 540.0-555.0 sec 1.27 GBytes 730 Mbits/sec 0.003 ms 194/931042 (0.021%)
[ 3] 555.0-570.0 sec 1.28 GBytes 734 Mbits/sec 0.001 ms 337/935976 (0.036%)
[ 3] 570.0-585.0 sec 1.27 GBytes 729 Mbits/sec 0.005 ms 1149/930504 (0.12%)
[ 3] 585.0-600.0 sec 1.26 GBytes 720 Mbits/sec 0.002 ms 3498/921970 (0.38%)
[ 3] 600.0-615.0 sec 1.26 GBytes 724 Mbits/sec 0.001 ms 1313/924564 (0.14%)
[ 3] 615.0-630.0 sec 1.26 GBytes 721 Mbits/sec 0.001 ms 2492/922685 (0.27%)
[ 3] 630.0-645.0 sec 1.26 GBytes 721 Mbits/sec 0.001 ms 1113/920194 (0.12%)
[ 3] 645.0-660.0 sec 1.27 GBytes 726 Mbits/sec 0.002 ms 1196/927047 (0.13%)
[ 3] 660.0-675.0 sec 1.28 GBytes 731 Mbits/sec 0.005 ms 1175/933690 (0.13%)
[ 3] 675.0-690.0 sec 1.28 GBytes 731 Mbits/sec 0.006 ms 777/933003 (0.083%)
[ 3] 690.0-705.0 sec 1.27 GBytes 725 Mbits/sec 0.003 ms 1749/927058 (0.19%)
[ 3] 705.0-720.0 sec 1.27 GBytes 728 Mbits/sec 0.004 ms 1325/929716 (0.14%)
[ 3] 720.0-735.0 sec 1.28 GBytes 731 Mbits/sec 0.001 ms 899/933367 (0.096%)
[ 3] 735.0-750.0 sec 1.28 GBytes 734 Mbits/sec 0.004 ms 0/936488 (0%)
[ 3] 750.0-765.0 sec 1.27 GBytes 730 Mbits/sec 0.001 ms 1140/932250 (0.12%)
[ 3] 765.0-780.0 sec 1.02 GBytes 585 Mbits/sec 0.010 ms 184896/930982 (20%)
[ 3] 780.0-795.0 sec 1.22 GBytes 700 Mbits/sec 0.009 ms 41129/933455 (4.4%)
[ 3] 795.0-810.0 sec 1.27 GBytes 726 Mbits/sec 0.007 ms 4415/930808 (0.47%)
[ 3] 810.0-825.0 sec 1.28 GBytes 732 Mbits/sec 0.008 ms 697/934220 (0.075%)
[ 3] 825.0-840.0 sec 1.27 GBytes 728 Mbits/sec 0.011 ms 1805/930134 (0.19%)
[ 3] 840.0-855.0 sec 1.27 GBytes 729 Mbits/sec 0.008 ms 4300/934322 (0.46%)
[ 3] 855.0-870.0 sec 1.24 GBytes 709 Mbits/sec 0.009 ms 15972/919678 (1.7%)
[ 3] 870.0-885.0 sec 1.14 GBytes 650 Mbits/sec 0.002 ms 106256/935885 (11%)
[ 3] 0.0-899.9 sec 75.4 GBytes 720 Mbits/sec 0.006 ms 705679/55799972 (1.3%)
[ 3] 0.0-899.9 sec 2 datagrams received out-of-order
Test done from an off-net institution without them doing any abnormal adjustments on window sizes:
--2009-07-25 13:02:28-- http://noc.as5580.net/1000mb.bin
Resolving noc.as5580.net... 80.94.65.2
Connecting to noc.as5580.net|80.94.65.2|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M) [application/octet-stream]
Saving to: `/dev/null'
2009-07-25 13:03:03 (29.3 MB/s) - `/dev/null' saved [1048576000/1048576000]
|
|
|
The administrator has disabled public write access. |
|
|
