How to evaluate the quality of mobile networks
Assessing the quality of services offered in mobile networks and their performance has long been of interest to telecommunications market participants. Among them, especially network operators and regulators, as well as trade media and manufacturers of equipment for measuring and testing such as Rohde&Schwarz and partner company Systemics-PAB, an independent provider of testing and assessment services.
ETSI scoring method
Both companies cooperate in developing best measurement methods and algorithms for assessing the quality of service in mobile networks. They are also active in the ETSI organisation where technical reports are composed describing and recommending best practices in network quality testing, resolving the previous inability to consistently compare operator services and networks mainly due to the lack of unified measurement methods. The final report of this work, TR103559 “Best practices for robust network QoS benchmark testing and scoring”, was published in August 2019. Along with general principles for conducting test campaigns. It covers service baselines, measurements and KPIs and includes detailed descriptions of the individual values measured, as well as an algorithm for converting them to obtain an overall network performance score. The report is the result of a joint effort of network operators, infrastructure manufacturers, measurement equipment manufacturers, and the providers of measurement and evaluation services for the quality of telecommunications services. Representatives of Rohde&Schwarz and Systemics-PAB also participated in its development.
User perception as a rating scale
Mobile users now see smartphones as the primary means of accessing digital services and applications of all kinds. This has been contributed to by the rapid increase in mobile network capacity, where data-intensive services and applications are dominant. Usage profiles and requirements for satisfactory quality of service vary significantly from one application or service to another. Users have different quality expectations when uploading a photo to a social network than when watching a live football match. Despite high bandwidth, today’s networks are still not a perfect transport medium. Interference happens or the data throughput required by the application cannot be provided or cannot be provided continuously. Transmission times also lead to a time offset between sending and receiving data. Many applications adapt to the changing parameters of the transmission channel and try to minimise the negative impact of the network on the perceived quality, for example by temporarily changing the resolution of images or by buffering to compensate for transmission interruptions.
.
Quality of service assessment
Network operators have a vested interest in knowing how to assess the performance of their networks from a user perspective and to identify how they can improve it in the most efficient way possible. Technical parameters such as data rates, transmission delays, the loss of transmitted data packets and the need to retransmit them define the technical side of network performance but are less well understood by the wider public. To obtain practical information on overall network performance, the ETSI method proposes a weighted sum of quasi-perceptual quality measurements for typical use cases such as telephone calls, data send/receive, video transmission, web browsing performance or the ability to use social media efficiently.
To properly cover all quality aspects of the tested services, the behaviour of a typical user is simulated. For this purpose, automatic mobile test systems (in a car or backpack) based on commercially available user terminals (usually smartphones) establish connections in real networks and run a sequence of tests for each of the use cases described above. In this way, from a practical point of view, representative situations of network use are mapped. For each of the applications represented by the executed test, the availability of the service is determined, its continuity, the time of access to the service, the quality of the transmitted content (speech, image or data transmission speed). For each aspect, quality indicators are measured and compared with expectations defined by a minimum indicator level and a level beyond which the customer’s perception does not change.
The quality indicators obtained are aggregated to defined areas and, after applying an algorithm that weighs their contribution to the assumed network user experience, a numerical description of the evaluation of the quality of service provided is obtained.
Benefits for operators and network users
The results of nationwide benchmarking influence public opinion, especially in Europe, and are an important decision factor for many users. For example, periodic network assessments by consumer magazines are often described in popular news portals. The implementation of such measurements by different measurement providers using different evaluation methods did not give the recipients of the research results a clear situation, especially when the results of two different studies were seen to give different results. The ETSI-compliant method is used to assess the quality of services in projects carried out by Systemics-PAB. With a clearly defined scoring system, all changes in the network and their impact on the perceived quality can be directly checked and quantified. The ETSI technical report allows the use of modified weights and thresholds tailored to the market to better reflect its specific characteristics (demographics, service usage, maturity of telecommunications networks, available technologies). Rohde&Schwarz uses the method described by ETSI to calculate the network performance score as part of the analysis and reporting in its analytics tool, SmartAnalytics. The integrated view of the network performance score also shows potential improvement opportunities for the selected region or application and enables an effective focus on problematic situations in the network.
Jens Berger, Rohde & Schwarz i Jan Kondej, Systemics-PAB
