Mobile Broadband: Wellington Carrier Rankings - Truenet

PO Box 11-053, Wellington Ph. 04 83 111 38

Mobile Broadband: Wellington Carrier Rankings TrueNet has launched its latest innovation - Mobile Broadband measurement tests which are being done throughout New Zealand. Initial results showing the performance of each carrier in Wellington for May 2011 reveal comparisons of mobile phone performance between Telecom, Vodafone and 2degrees, these have not been available until now. These results show the differences between carriers. The test locations are shown in the map of Drawing 1: Wellington Test Sites. Our measurements include speed tests for file downloads, web page downloads, file uploads, and national and international latency. This series of file download tests shows that all carriers are able to reach the same speeds, and the top speed within the Wellington region is just under 6 Mb/s. The colour of each spot on the map represents the best performing carrier for the test at that site, with grey indicating some carriers are equal best. Larger spots show faster downloads.

Drawing 1: Wellington Test Sites

Ofcom, the UK telecommunications regulator released a new report on Friday 27th May 2011;

TrueNet Ltd. PO Box 11-053, Wellington. Ph: (04) 83 111 38 Page 1


“Measuring Mobile Broadband in the UK: performance delivered to PCs via dongles/datacards”1 [Ofcom] TrueNet tests in NZ are similar to those performed for Ofcom in the UK, so we have made direct comparison throughout this document. We used the same or similar criteria to rank carriers in New Zealand.

The results show: Test

Best using Ofcom criteria

Ofcom Criteria (2)

File download speed

Telecom

>3Mb/s (5.14)

Web-page download time

2Degrees

2 seconds* (5.22) but we used 4 seconds

Best single sample Telecom, marginally from both Vodafone and 2Degrees Telecom, significantly

Vodafone, significantly National Vodafone, Telecom 200ms (7.17) Latency significantly International No Ofcom measure, Vodafone, Vodafone Latency we used 300ms significantly * Based on the text content of the webpage only, TrueNet tests the entire webpage. Upload speed

Telecom

0.4Mb/s (7.21)

Best performers • • •

Telecom is ahead of the rest when testing national download speeds for video or mail. 2Degrees is by far the best for web browsing such as downloading TradeMe. Vodafone is the best for uploading a file and international downloads.

Methodology 2degrees network covers all of Wellington, subject to downgrades to the slower 2G network where their 3G network does not reach. We performed all tests on 3G only which means they are without the impact of 2Degrees roaming or 2G services. Vodafone have a second 3G network on a lower frequency band they call “extended reach” which our units test automatically if it is available, otherwise they do not perform a test, even if a 2G network is available. 1 http://stakeholders.ofcom.org.uk/binaries/research/telecomsresearch/bbspeeds2010/Mobile_BB_performance.pdf 2 http://stakeholders.ofcom.org.uk/binaries/research/telecomsresearch/bbspeeds2010/Mobile_BB_performance.pdf TrueNet Ltd. PO Box 11-053, Wellington. Ph: (04) 83 111 38 Page 2


The TrueNet probe is installed in a vehicle and contains a GPS unit and three mobile network devices, one dedicated to each carrier. Dedicated units avoid the potential for interference by other software that may be active on a device such as a smartphone. The probe performs a run of three sets of tests, one for each carrier, every few hundred metres or within a time interval if the vehicle is stationary. Every test is concurrent, avoiding any timing issue, weather, or temporary obstruction affecting the results for any single carrier. Each set of tests consists of several consecutive tests; single file download, file upload, national and international latency, web page download etc. Our results include tests that Did Not Finish (DNF). When a test is performed successfully by at least one carrier but does not succeed for any other carrier that test is recorded as a DNF for that carrier. The number of DNF results for 2Degrees is higher because their network restricts data roaming to their “Zones” on the standard contract we use (Mobile Broadband 3GB Data Pack) and we do not do 2G testing. In Wellington, any test which is outside the 2Degrees 3G network will produce a DNF result. The tests for Vodafone and Telecom are also exclusively 3G.



Download Speeds Our results show that 62% of Telecom's download speeds exceed 3Mb/s. By contrast, for slow results of 2Mb/s or less, Vodafone had 43%, 2Degrees had 55%, and Telecom had 24%.

Source: TrueNet probes during May 2011. Data is based on 3G bearers, including "Extended reach". Maximum margin of error for this Wellington area is 4%. Higher speeds indicate better performance.

These measurements contrast with Ofcom's release where the best performer, O2, had 42% above 3Mb/s.3 The UK worst performances were similar to 2Degrees in TrueNet's Wellington results with a majority of tests below 2Mb/s, however 2Degrees 3G network does not fully cover Wellington so had many DNF results. Removing the DNF results, and reporting only the areas where their network covers, 2Degrees had just 23% below 2Mb/s.

3 Ofcom Mobile BB Performance report, sec 5.14, page 32 TrueNet Ltd. PO Box 11-053, Wellington. Ph: (04) 83 111 38 Page 4


Web Page Download Time This test downloads the complete front page of TradeMe including text and images. The Ofcom methodology, however, excludes images, leaving the majority of the page untested. A direct comparison between Ofcom and TrueNet for this particular test is therefore not possible. TrueNet decided an approximate comparison with Ofcom is the capability of carriers to download the test web page within 4 seconds. TrueNet tests show that 2Degrees achieved 54% with Telecom managing 48% and Vodafone 26%. Again, removing the DNF results, 2Degrees had 84% within 4 seconds, that means if there is a 3G signal with 2Degrees, the page will arrive quickly.

Source: TrueNet probes during May 2011. Data is based on 3G bearers, including "Extended reach". Maximum margin of error is 4%. Less time to download indicates better performance. Note: this graph has the scale reversed to show best at right



Upload Files Uploading is likely to be important when using mobile broadband for emailing large files or uploading photos. All three New Zealand carriers have slow upload performances. Strange peaks appear suggesting upload packet shaping, much as was noted by Ofcom. 2Degrees has many results at 0.2Mb/s, with all tests slower than 0.3Mb/s, while Vodafone peak at around 0.3Mb/s, with most tests less than 0.4Mb/s. Telecom results are better with many uploads greater than 0.5Mb/s, and many almost 1Mb/s. We recorded 87% of upload speeds in Wellington less than 0.4Mb/s, excluding DNF. Ofcom results in the UK for the same standard were better at 74%.

Source: TrueNet probes during May 2011. Data is based on 3G bearers, including "Extended reach". Maximum margin of error is 4%. Higher speeds indicate better performance.

Ofcom noted extreme differences in upload speeds. TrueNet also found wide variation. “There was a wide range of distribution of average upload speeds among the 436 probes which had a 3G/HSPA connection. Nearly 70% of probes had



average upload speeds of more than 0.6Mbit/s, indicating that consumers can experience HSUPA upload speeds in some situations” [Ofcom] TrueNet found that just 7% of upload tests had speeds greater than 0.6Mb/s. The highest speed was 0.86Mb/s from Vodafone. This impact can be seen in the upload chart, where Telecom tests peaked at 0.4Mb/s, then again at >0.7Mb/s. This may be due to some cellsites having HSUPA. HSUPA enables faster upload speeds, up to 5.7Mb/s for devices like the iPhone, but typically 3G mobile has been limited to 0.73Mb/s until recently.



National Latency TrueNet checks latency (delay in time for a signal to be returned from a remote site) for both National and International connections. High latency slows both file and webpage downloads, especially if the traffic includes a lot of small files, common with video and gaming.

Source: TrueNet probes during May 2011. Data is based on 3G bearers, including "Extended reach". Maximum margin of error is 4%. Less time to return a packet indicates better performance. Note: this graph has the scale reversed to show best at right

In Wellington, 66% of all tests are better than 160ms, and overall results are similar to Ofcom's UK results. Ofcom reported a third of tests over 200ms, and in Wellington, TrueNet measured 29% in this category. 2Degrees results were the worst at 51%, mostly due to the DNF results, without these they had only 22%. Telecom managed 20% below 100ms (para 7.17), although the best result was a 65ms response from Vodafone.



International Latency International Latency was not considered by Ofcom, but it is a critical factor in NZ Internet usage, because it slows international traffic dramatically due to the large distances to the rest of the world.

Source: TrueNet probes during May 2011. Data is based on 3G bearers, including "Extended reach". Tests are to Dallas, USA. Maximum margin of error is 4.1%. Less time to return a packet indicates better performance. Note: this graph has the scale reversed to show best at right

The Did Not Finish results from all carriers dominate the International latency statistics. Vodafone was the best with 40% less than 300ms, they also had the highest “worse than 400ms” count with 9%. Telecom had 32% below 300ms and 2Degrees was a distant 7%. For comparison, TrueNet fixed line average international latency measurements for each ISP were in the range 185ms to 231ms for May.
