In preparation for a new project in Spain, Systemics-PAB has verified the performance of 5GNR network services. The testing campaign was performed from 11 to 26 May 2021 over several main cities including Madrid, the capital and most populated city in Spain. Population of the measured cities represent 17.8% of the national population (8.45 million of inhabitants). The tests were performed for the Movistar, Orange and Vodafone networks. The MasMovil/Yoigo network was not tested due to the fact that it did not offer commercial 5G services in own network at the time of the test. Access to the 5GNR network was tested for data and video services.

Data service assessment was performed with SwissQual Smart Benchmarker system v20.3 equipped with Samsung Galaxy S21+ (SM-G996B) terminals with G996BXXS2AUBB firmware for each of measured operator.

Measured test cases consisted of:

  • 7s fixed duration transfer tests for downlink and uplink (capacity tests DL/UL), which were designed to stress test network in order to measure maximum throughput performance. Each test used 7 TCP sockets transferring in parallel 2GB datafiles to make sure that measured performance reflects network capabilities and is not impacted by protocol limitations
  • Fixed size small file transfers of 10MB for Downlink direction and 5MB for Uplink direction transferred over HTTP protocol with single TCP sockets. Based on this testcase user perception when transferring pictures or small multimedia files can be assessed (file transfers DL/UL)
  • Selection of 8 live pages as chosen based on popularity to assess the live browsing performance. In order to achieve results in line user perception metric of Time to Download 500kB of content was used. While it is common for today’s webpages to vary in size and in content, the user perception relies on how fast webpage render initial display content rather than download of full webpage (web browsing).
  • To simulate multimedia streaming services, Youtube livestream service was measured (video streaming). Assessed metrics were: Time to First Picture – duration between pressing play button and first frames being rendered on screen as well as Visual Quality measured by VMOS algorithm, which taken into account no only resolution of clip, but also metrics more impacting user perception like Bluriness, Frezings, Jerkiness of the video clip.

Countrywide results
Tests were performed on route with the total length of 1500 km drivetested in 78 hours. Almost 2500 tests of DL/UL capacity were performed in each network in order to access the best throughput performance. The similar number of file transfers DL/UL were executed to check the quality of the use case with sending attachments. The video performance was assessed with the test of live video streaming. Around 1180 tests of video streaming were executed in each tested network. Close to 10800 web browsing tests were done in every tested network.

During the tests, the highest usage of the 5GNR technology for data transmission was measured in the Orange network, where it exceeded 51%. The second place in this respect was taken by Vodafone with a result of over 37%, followed by Movistar reaching almost 35%. Compared to the 2020 measurements, this represents an increase of almost 17% for the Orange and Vodafone networks. Movistar was not tested last year.

01

At national level the tests show that Orange is the fastest network with the highest average download speed of 156.6Mb/s while the upload tests results of all the networks tested are very similar with a slight advantage for Vodafone 37.4Mb/s, while Orange achieved 34.0Mb/s and Movistar 31.6Mb/s.

02

The highest maximum transmission speeds, measured as the value exceeded by 10% of the fastest tests were obtained on the Orange network and exceeded 623Mb/s. In the Vodafone network result of 506Mb/s was achieved. Movistar obtained this indicator in vicinity of 305Mb/s.

03

In the case of Live web browsing testing was performed based on 8 live pages selected from the Alexa ranking of most popular sites in Spain. Vodafone and Orange achieved a similar efficiency of access to live browsing service of about 97%, while Movistar had this indicator at a level lower by 10pp, which most likely indicates temporary underperformance of network at the time of the test. The fastest time to page content (first 500kB of useful content) loaded on the Orange network.

For the video service tests, streaming content from YouTube was used. The highest efficiency was achieved by Vodafone with a score of almost 99%. All operators achieved almost identical results in the video quality test, within 4.05 on a five-point quality scale. A slightly shorter time to first picture was measured on Orange’s network and was at 1.05 seconds.

Focusing on Madrid
During tests in Madrid, the highest usage of the 5GNR technology for data transmission was measured in the Orange network, where it reached 48%. The second place was taken by Movistar with a result of over 37%, followed by Vodafone with less than 30%.

04

In Madrid, Orange is the fastest network with the highest average download speed with 156.2Mb/s while the upload tests results of all the networks tested are very similar with a slight advantage for Vodafone 39.2Mb/s, while Orange achieved 34.9Mb/s and Movistar 34.6Mb/s.

05

In the case of Live web browsing testing was performed based on 8 live pages selected from the Alexa ranking of most popular sites in Spain. Vodafone and Orange achieved a similar efficiency of access to live browsing service of exceeding 97%, while Movistar had this indicator at a level below 90%. The fastest time to page content (first 500kB of useful content) loaded was measured in Orange network and was 1 second.

The shortest time to access the video streaming service was achieved by Orange with the result below 1 second. All operators achieved almost identical results in the video quality test, within 4.05 on a five-point quality scale.

Conclusion
The study shows a significant development in the availability of 5G networks in tested Spanish cities compared to 2020, with 5G network transmissions now available on more than a third of the test route or as in the case of Orange network, on more than 50% of that route. The quality indicators may not yet be in line with high expectations for 5G mature networks but they clearly show the great lead forward in the quality of service due to implementation of the new technique and the evolutionary nature of the deployment. The results of tests show that Orange leads the 5G competition in Spain.