StarTrinity.comMeasuring quality |
Performance and Stress Testing of SIP Servers, Clients and IP Networks
- Archive of performance and stability reports
- FreeSWITCH 1.5.13 wav file audio playback, pass-through G.711
- Asterisk 11.7.0 on Ubuntu 14.04 LTS x64 - performance (5..110 channels), stability (1.5M calls)
- StarTrinity Softswitch - wav file audio playback, B2BUA with G.729
- RTP round-trip delay time (RTT, RTD) over 1GBit LAN, internet, 802.11g WLAN, 802.11n WLAN
- 802.11g, 802.11n wireless LAN VoIP call load capacity
- Android SIP Client (3CXPhone) with WiFi connection
- Elastix 2.4 (Asterisk 1.8.2) IP PBX running on a laptop
- Internet connection between SIPP at Raspberry Pi 2 and StarTrinity SIP Tester at windows server
- Penetration test suite - updates
- FreeSWITCH penetration tests (since 2018)
- 3CX penetration tests (since 2018)
Introduction
The emergence of Voice over IP (VoIP) technology offered numerous advantages for end users and providers, but simultaneously introduced security threats, vulnerabilities and attacks which hadn't previously encountered in networks with a closed architecture like the Public Switch Telephone Network (PSTN). If audio data is transferred across public internet, there is no guarantee in constant traffic flow or reliable transport. Also, publically accessible SIP endpoints are subject to denial of service (DoS) attacks which can cause severe degradation of voice quality and call setup success rate. Even in a closed ethernet environment VoIP audio quality is still vulnerable to delays in RTP stream processing caused by network, client or server. These delays can be caused by overloads of CPU and network stack or file system operations. That's why it is necessary to know minimum hardware system requirements to handle certain call load. An integrator must test IP network and SIP servers before launching and make sure that there are no overloads in the system.
On this page we present methodology and results of SIP performance evaluation (benchmarking) for IP networks, servers and clients using a software tool StarTrinity SIP Tester. Our results provide insight into resources' usage by SIP servers and clients. Our measurements show that there is no straightforward relationship between call quality and simulated call load. Moreover, we show that CPU load peaks and big RTP delays (which result in audio gaps) can be caused by some other (non-SIP) processes in device's operating system.
SIP call quality measurement
The process of VoIP system assessment implies making many simultaneous SIP calls to device(s) under test (DUT) and measurement of call quality parameters. Basic parameters are:
- Maximum RTP jitter per call
- Global maximum RTP jitter (for all calls)
- Call answer delay (time between INVITE and 200 OK)
- Global maximum call answer delay
- Average CPU load
Archive of performance and stability reports
Our company is continuously making research in field of SIP call quality for various software, hardware and networks. The research is made by running stressing tests with thousands and millions of VoIP calls and analysing call quality indicators. Some of our reports are published here.
2015-01-19 StarTrinity Softswitch x64 on Windows Server 2012 R2 - 800+800 G729 passthru channels, 2.22 million calls during 3 days
Report for UAC (SIP call generator)
StarTrinity Softswitch was configured as B2BUA: it accepted calls from UAC SIP Tester #1 (UAC) and connected them with SIP tester #2 (UAS):
2015-01-16 StarTrinity Softswitch x64 on Windows Server 2012 R2 - 150..1650 G711 channels StarTrinity Softswitch was configured as B2BUA: it accepted calls from UAC SIP Tester #1 (UAC) and connected them with SIP tester #2 (UAS):
<transfer value="sip:111@devpc2:5090"/>
<exit />
We simulated variable call load from 150 to 1750 G.711 channels from SIP Tester to server with StarTrinity Softswitch via Cisco router:
Windows Server 2012 R2, Intel core i7-2600 CPU @3.40GHz, 4 cores. Duration of each call was 90 seconds.
StarTrinity Softswitch was configured to accept calls and play audio file:
2014-08-26 Asterisk 11.7.0 on Ubuntu 14.04 LTS x64 - 100 G.711 channels, 6 days, 1.5M calls
<accept />
<playaudio value="music.wav" maxtime="90s" />
<exit >
EnableSignalDetector = 0
MediaThreadsCount = 64
We simulated variable call load from 5 to 110 G.711 channels from SIP Tester to server with Asterisk via Cisco router: Intel core i7-2600 CPU @3.40GHz, 4 cores. Duration of each call was random from 30 to 40 seconds. We installed Asterisk PBX using Ubuntu software center, and used default configuration. Extension #500 by default accepts calls and plays demo IVR message.
RTP jitter was low, stability was great. Asterisk passed 1.5M SIP calls without crash (see detailed report). There were few calls with lost packets though (see the chart below).
2014-08-18 Asterisk 11.7.0 on Ubuntu 14.04 LTS x64 - 5..110 G.711 channels
We simulated variable call load from 5 to 110 G.711 channels from SIP Tester to server with Asterisk via Cisco router: Intel core i7-2600 CPU @3.40GHz, 4 cores. Duration of each call was random from 30 to 40 seconds. We installed Asterisk PBX using Ubuntu software center, and used default configuration. Extension #500 by default accepts calls and plays demo IVR message.
RTP jitter was fairly low, but Asterisk crashed with 110 G.711 channels (see the chart). Stability of FreeSWITCH is much better.
2014-07-27 802.11n 72Mbps wireless LAN - 2..80 G.711 channels, distance = 3m, 1 wall
We built call capacity chart for 802.11n 72Mbps wireless LAN between a server connected to WiFi adapter (Upvel UR-314AN) and a laptop (Samsung NP305E7A - Atheros AWB9485WB-EG). RTP jitter increases with number of channels, but there is no straightforward linear dependency. The jitter greatly depends on some other variable. Max call load capacity is hard to estimate because of sporadic RTP jitter.
2014-07-25 802.11g 54Mbps wireless LAN - 2..24 G.711 channels, distance = 3m, 1 wall
We built call capacity chart for 802.11g 54Mbps wireless LAN between a server with USB wifi adapter (ASUSN53) and a laptop (Samsung NP305E7A - Atheros AWB9485WB-EG). RTP jitter increases with number of channels, but there is no straightforward linear dependency. The jitter greatly depends on some other variable, probably obstacles between adapters. Max call load capacity is 24 G.711 channels = 2.1Mbps, it is much less than theoretical 54MBps (3.9%)
2014-07-22 FreeSWITCH 1.5.13 x64 on Windows Server 2012 R2 - 150..2250 G711 channels
We simulated variable call load from 150 to 2250 G.711 channels from SIP Tester to server with FreeSWITCH via Cisco router: Windows Server 2012 R2, Intel core i7-2600 CPU @3.40GHz, 4 cores. Duration of each call was 90 seconds. FreeSWITCH was configured to accept calls and play audio file:
<action application="answer"/>
<action application="playback" data="ponce-preludio-in-e-major.wav"/>
<param name="max-sessions" value="10000" />
<param name="sessions-per-second" value="1000" />
FreeSWITCH showed good performance with high call load. RTP jitter was fairly low (see the chart). However it crashed in some other long-running tests with millions of calls. We are going to continue tests with FreeSWITCH installed on Linux, it can make a difference
2014-07-21 FreeSWITCH 1.5.13 x64 on Windows Server 2012 R2 - 250+250 G711 passthru channels, 2.89 million calls during 4 days
Report for UAC (SIP call generator)
FreeSWITCH was configured as B2BUA: it accepted calls from UAC SIP Tester #1 (UAC) and connected them with SIP tester #2 (UAS):
<action application="answer"/>
<action application="bridge" data="sofia/internal/freeswitch2siptester@192.168.4.4:5070" />
2014-07-27 RTP round-trip delay over 802.11n 72Mbps wireless LAN (distance 3m, 1 wall), single G.711 channel
We simulated 5713 SIP calls via 802.11n 72Mbps wireless LAN between a server connected to WiFi adapter (Upvel UR-314AN) and a laptop (Lenovo G700 - Broadcom). Test duration: 10 hours. RTT: avg=12.82ms, max=2122ms, 99.9-percentile=106ms. 0% RTP packets were lost. The results for 802.11n adapters are worse than previous results for 802.11g adapters.
Detailed report for UAC (SIP call generator)
2014-07-24 RTP round-trip delay over 802.11g 54Mbps wireless LAN (distance 3m, 1 wall), single G.711 channel
We simulated 8343 SIP calls via 802.11g 54Mbps wireless LAN between a server with USB wifi adapter (ASUSN53) and a laptop (Samsung NP305E7A - Atheros AWB9485WB-EG). Test duration: 14 hours. RTT: avg=12.25ms, max=73ms. Quality for 100% of calls was perfect, 0% RTP packets were lost. It looks strange because the results are better for larger distance (3m > 1m). It means that there are some unknown variables which affect quality of WLAN.
Detailed report for UAC (SIP call generator)
The charts show that round-trip delay varies with time, and it depends on obstacles between wifi adapters
2014-07-23 RTP round-trip delay over 802.11g 54Mbps wireless LAN (distance 1m), single G.711 channel
We simulated 12593 SIP calls via 802.11g 54Mbps wireless LAN between a server with USB wifi adapter (ASUSN53) and a laptop (Samsung NP305E7A - Atheros AWB9485WB-EG). VoIP RTP round-trip delay time (RTT, RTD) was calculated from difference of NIC timestamps in RX and TX RTP streams (loopback connection was used). Test duration: 21 hours. RTT: avg=12.34ms, max=393ms, 99.9-percentile=93ms. We observed few peaks of high RTT and packet loss (see the chart). Quality for 99.9% of calls was perfect.
Detailed report for UAC (SIP call generator)
2014-07-22 RTP round-trip delay over 1GBit LAN, single G.711 channel
We simulated 6684 SIP calls via 1GBit Cisco switch between 2 servers. VoIP RTP round-trip delay time (RTT, RTD) was calculated from difference of NIC timestamps in RX and TX RTP streams (loopback connection was used). RTT did not go over RTP packet time (20ms). No RTP packet losses were detected.
Detailed report for UAC (SIP call generator)
2014-07-21 RTP round-trip delay over internet, single G.711 channel
We simulated 12509 SIP calls via cable internet from office in Russia, Kazan to a VPS server in US, NJ, Secausus. VoIP RTP round-trip delay time (RTT, RTD) was calculated from difference of NIC timestamps in RX and TX RTP streams (loopback connection was used). We observed 4 peaks of RTT with maximum of 2779ms. Average RTT: 189ms. 99.9-percentile: 374ms.
Average RTP packet loss: 0.01%. Max RTP packet loss: 9.24%. 99.5-percentile RTP packet loss: 0.33%
Detailed report for UAC (SIP call generator)
2014-03-14 ASUS USB-N53 Wireless USB Adapter, 5 G711 channels
| Call generator (UAC) node | Laptop CPU Pentium 2020M 2.4GHz 2 cores | detailed report | jitter chart | |
| Call receiver (UAS) node | Laptop CPU AMD A8 3520M 1600MHz 4 cores | detailed report | jitter chart | |
| IP Network | Wireless 54MBit, ASUS USB-N53 USB Adapter at UAS | ||
| Number of G711 channels | 5 | Number of calls attempted | 7155 |
| RTP jitter at UAS node | avg: 80ms | 99-percentile: 577ms | max: 10635ms |
| RTP jitter at UAC node | avg: 53ms | 99-percentile: 88ms | max: 5696ms |
| Answer delay at UAC node (INVITE...200 OK) | avg: 25ms | 99-percentile: 51ms | max: 715ms |
| -24dB delay at UAC node (INVITE...RTP audio signal) | avg: 97ms | 99-percentile: 123ms | max: 783ms |
2014-03-12 ASUS USB-N53 Wireless USB Adapter, 5 G711 channels.
| Call generator (UAC) node | Laptop CPU Pentium Dual-Core T3100 1.9GHz 2 cores | view detailed report | |
| Call receiver (UAS) node | Laptop CPU AMD A8 3520M 1600MHz 4 cores | view detailed report | |
| IP Network | Wireless 54MBit, ASUS USB-N53 USB Adapter at UAS | ||
| Number of G711 channels | 5 | Number of calls attempted | 2235 |
| RTP jitter at UAS node | avg: 80ms | 99-percentile: 256ms | max: 738ms |
| Media threads delay at UAC node | avg: 41ms | 99-percentile: 155ms | max: 570ms |
| RTP jitter at UAC node | avg: 30ms | 99-percentile: 63ms | max: 255ms |
| Answer delay at UAC node | avg: 36ms | 99-percentile: 195ms | max: 337ms |
2014-03-11 ASUS USB-N53 Wireless USB Adapter, 5 G711 channels.
| Call generator (UAC) node | Laptop CPU AMD A8 3520M 1600MHz 4 cores | view detailed report | |
| Call receiver (UAS) node | Desktop PC CPU i7 3400MHz 8 cores | view detailed report | |
| IP Network | Wireless 54MBit, ASUS USB-N53 USB Adapter at UAS | ||
| Number of G711 channels | 5 | Number of calls attempted | 2178 |
| RTP delta at UAS node | avg: 162ms | 99-percentile: 3577ms | max: 4621ms |
| RTP delta at UAC node | avg: 38ms | 99-percentile: 162ms | max: 2018ms |
| Answer delay at UAC node | avg: 154ms | 99-percentile: 7507ms | max: 15528ms |
2014-03-09 ASUS USB-N53 Wireless USB Adapter, 20 G711 channels.
Report for UAC node (SIP call generator - laptop Samsung NP305E7A)
Report for UAS node (SIP call responder - desktop PC i7 3400MHz 8 cores)
The test was performed using laptop and desktop PC (UAC and UAS) connected via USB WiFi adapter (installed on UAS node). RTP audio quality was measured using winpcap RTP/SIP sniffer which is integrated into SIP Tester on both UAC and UAS nodes. For 20075 simulated G711a SIP calls max RTP jitter was 3106ms on UAC node, 11799ms on UAS node. For 99% of calls it was less than 619ms on UAC node, 1099ms on UAS node. RTP packet loss reached 56% for 1 minute long SIP call. Conclusion - tested hardware is not acceptable for 20 G711 channels. We are going to continue tests with less number of parallel calls.
2014-03-08 ASUS USB-N53 Wireless USB Adapter, 10 G711 channels.
Report for UAC node (SIP call generator - laptop Samsung NP305E7A)
Report for UAS node (SIP call responder - laptop Lenovo G700)
The test was performed using 2 laptops (UAC and UAS) connected via USB WiFi adapter (installed on UAS node). RTP audio quality was measured using winpcap RTP/SIP sniffer which is integrated into SIP Tester. For 8350 simulated G711a SIP calls max RTP jitter was 456ms on UAC node. For 99% of calls it was less than 135ms (UAC node). Conclusion - tested hardware is good for non-critical VoIP applications
2014-01-27 3CX Phone System 12.0.33463.0 with RTP audio going over 100Mbps ethernet LAN without router, 2 G711 channels
A test with minimal call load (2 channels). Max jitter is 154ms which is good, but max answer delay is not acceptable (7.1 seconds). In worst case the server ignores few INVITE messages. 4751 calls have been attempted. See comments in report file.
2014-01-24 3CX Phone System 12.0.33463.0 with RTP audio going over 100Mbps ethernet LAN without router, 4 G711 channels
Max jitter is 381ms, but for 99.99% of calls it is less than 106ms. 32989 calls have been attempted. See comments in report file.
2014-01-23 3CX Phone System 12.0.33463.0 with RTP audio going over 100Mbps ethernet LAN without router, 4 G711 channels
PESQ_MOS >= 1.25. 9928 calls have been attempted, part of them have successfully passed through PBX back to SIP Tester. See comments in report file.
2014-01-17 3CX Phone System 12.0.33463.0 with RTP audio going over 54Mbps wireless LAN without router, 4 G711 channels
Test is successful, quality of audio passing through 3CX PBX is good enough (PESQ_MOS >= 2.64). Wireless LAN is not a problem. 2188 calls have been attempted.
2014-01-16 3CX Phone System 12.0.33463.0 with RTP audio going over 100Mbps ethernet via router, 4 G711 channels | PESQ_MOS=3.03 PESQ_MOS=3.28
Test is successful, quality of audio passing through 3CX PBX is good enough. 2008 calls have been attempted.
2014-01-15 3CX Phone System 12.0.33463.0 with RTP audio going over 54Mbps wifi via router, 4 G711 channels |
PESQ_MOS=1.06 PESQ_MOS=1.79 PESQ_MOS=2.59 PESQ_MOS=4.54
To configure 3CX we have set Extension options -> 'pbx delivers audio' to 'true'.
NetGear router WGR614v9 is not suitable for VoIP traffic if wifi is used
Elastix 2.4 (Asterisk 1.8.2) IP PBX running on a laptop
The server (DUT) is configured to answer call (send 200 OK to INVITE), activate RTP media flow and make another call to SIP Tester (bridged redirection). Asterisk performance stress test
- SIP server software: Asterisk 1.8.20
- Operating system: CentOS 5.9
- Processor: Intel Celeron M 1.6GHz
- RAM: 512MB DDR2 400MHz
- Network: Ethernet 100Mbps
- Router: NetGear WGR614v9
- HDD: 120GB Samsung 4800rpm
- Call duration: 1 minute
- Audio codec: G.711
Test results
We have measured call quality parameters for different numbers of parallel calls. VoIP server call capacity is 20 calls if one needs to achieve mean jitter level below 100ms.
- Mean jitter and average CPU load increase with number of simultaneous calls
- Low average CPU load does not mean low jitter
- There are bursts in CPU load
- Max jitter is much higher than mean jitter
- Max jitter depends on number of calls observed
- There is no straightforward linear relationship between max jitter and number of simultaneous calls
- Delays in server are random and do not increase linearly with number of calls. They are absolutely random and we guess that they can be independent from RTP media processing.
- CPU load peaks occur even without any calls
- RTP processing is sensitive to execution of other tasks performed on Linux server
Example asterisk configuration files for SIP Tester
/etc/asterisk/sip_custom.conf
[1001]
secret=
dtmfmode=rfc2833
canreinvite=no
context=from-internal
host=dynamic
trustrpid=yes
sendrpid=no
type=friend
nat=no
port=5060
qualify=yes
qualifyfreq=60
transport=udp
encryption=no
callgroup=
pickupgroup=
dial=SIP/1001
mailbox=1001@device
permit=0.0.0.0/0.0.0.0
callerid=1001 <1001>
callcounter=yes
faxdetect=no
/etc/asterisk/extensions_custom.conf
[ext-local-custom]
exten => 1001,1,dial(SIP/1001)
exten => 1001,hint,SIP/1001
Android SIP Client (3CXPhone) with WiFi connection
- SIP client software: 3CXPhone 2.0.5
- Operating system: Anfroid 4.0.4
- Mobile device: ZTE V880E
- Network: WiFi 802.11bg
- Router: NetGear WGR614v9
- Call duration: 10 minutes
- Audio codec: G.711
We have made SIP calls from Android device to computer with SIP tester and measured call quality. We have found out that RTP jitter does not depend on WiFi signal strength if it is greater than -78 dBm. Big delays in RTP flow happen when one launches some other application during the test call.
- Max jitter without interference from other applications: 60..70ms
- Max jitter with interference from other applications: 110..130ms
Internet connection between SIPP at Raspberry Pi 2 and StarTrinity SIP Tester at windows server
We have installed SIPP (open-source SIP Tester) at Raspberry PI 2, configured it to make 1-channel test calls to our server. The SIPP client played an audio file to StarTrinity SIP Tester, the audio quality was verified with PESQ algorithm. view detailed reportResults
PESQ score drops below 4.5 for 0.05% of calls because of packet loss in internet, probability of the packet loss changes with time: March 24 .. 28 - there is packet loss, March 29 .. April 7 - no packet losses. This means that one needs to run tests during at least 2 weeks to check stability of internet connection. One more finding: packet losses are assymetric: download and upload packet drops occur at different timePESQ MOS chart
Called to caller packet loss chart
Caller to called packet loss chart
Caller to called RTP jitter chart
Leave a comment