Assuring QoE in the OTT World: From the network side

This article talks about how network side monitoring can be used to measure QoE during ott testing and what you can do to ensure QoE from the Network Side. 

The consumer use of OTT services has been gaining more and more popularity, but still, the operators are not ready to let go of this important emerging market. Both fixed and mobile operators are aware of the revenue potential presented by OTT services. 

According to a survey by Streaming media, OTT services are expected to grow by 26% by the end of 2022. As the use of OTT services continues to grow, so do the concerns among the operators regarding QoS (Quality of Service)

Now, more than ever, QoE (Quality of Experience) is a problem that must be considered in the design, deployment, and functionality of OTT services. The application vendors of this multi-billion-dollar marketplace are increasingly competitive, user demands are increasingly high, and expectations are becoming less tolerant of service quality issues. 

The expected growth in OTT services that continue to consume more of the overall throughput has added to this complexity and requires more planning ahead if QoE is to be assured. Let’s get started! 

What is QoE? 

The quality of experience is an essential factor for OTT providers as it is directly related to customer satisfaction, impacting subscriber churn rates and revenue generation. QoE is a measure of the quality of experience a user has with your product or service. It is the perception of the user on how good the experience was when using a product or service and their satisfaction with it.

QoE can be measured differently, but it should be measurable and actionable. To assure QoE, OTT providers must be able to measure and control the quality from end to end and across multiple networks. This includes monitoring and managing both their infrastructure and that of their partners. 

The main benefit of measuring QoE is that it allows us to improve our services. It helps us understand what users want and need to provide them with better experiences. If we have good data on QoE, we can collect feedback on what users think about our product or service and make changes accordingly. For example, if we know that users are not happy with our mobile app because it crashes often, then we can make changes to improve this problem for future versions.

Network Side

In OTT, Quality of experience is vital in determining the service’s success. With QoE being one of the critical factors in monetizing an OTT service, it is essential to ensure that both the network and the client-side measurements are taken into account during testing.

When it comes to testing QoE, two main approaches can be taken:

• In client-side testing, end-users are asked to complete surveys or use apps to report on their user experience. Client-side testing allows for real-time feedback but does not give insight into what is happening at the network level or how different devices impact the quality of experience.

• Network side monitoring – from a network perspective, operators can monitor performance metrics such as bitrate distribution, packet loss, or latency between servers and clients.

Network side monitoring has become more critical as we see a paradigm shift from traditional telecommunication networks to next-generation networks (NGNs). NGNs have brought about many changes in terms of technology, operator business models, and regulations. 

Many new technologies are being implemented, like SDN/NFV, NFVI, and 5G, which require deep knowledge and understanding of what is happening at various layers inside the network. This has led to increased network design, deployment, and maintenance complexity, which requires automation and orchestration capabilities.

Additionally, many new services like SIP Trunking are being offered by operators today, which require additional monitoring capabilities on the network side.

Network side monitoring works by sending packets from the CDN endpoints to our servers and servers; we push those packets back to CDN endpoints. This allows us to monitor performance metrics, such as packet loss and latency on both sides (CDN and server). Once we have these metrics, we can analyze them and determine whether there are any issues.

A few parameters that can be monitored using a network monitoring system are:

1. Network latency – Latency is the time taken by a packet to travel from one point to another in the network. It is measured in milliseconds (ms).

2. Packet loss – Packet loss is defined as the number of packets that fail to reach their destination within a given period. Packets may fail for several reasons, such as congestion or transmission errors. Packet loss should be less than 1% for best results.

3. Jitter – Jitter refers to variations in delay between two consecutive packets transmitted over a network connection. Jitter should be less than 100 ms for best results.

4. Bandwidth variation: variation in bandwidth usage over time

Challenges in assuring QoE

As the OTT market continues to grow, it is increasingly important for service providers to have a solid testing strategy to ensure that their services meet their subscribers’ needs. The complexity of an OTT service makes it challenging to test the overall quality of experience (QoE) on the network side. This is especially true regarding VoD (video on demand) and lives streaming applications, which can be very challenging to test.

The following are some of the challenges associated with assuring QoE (quality of experience) in ott automation testing:

• Latency: In a VoD streaming application, latency can lead to poor playback quality or buffering. With live streaming, low latency is essential to prevent delays in audio/video synchronization, resulting in loss of viewers’ attention. 

• Bandwidth Stability: Bandwidth stability is critical for VoD and live streaming applications because it impacts content availability and overall QoE. For example, if a video stream requires 1 Mbps bandwidth but only 500 Kbps bandwidth is available, there will be buffering issues and likely viewer drop-off due to frustration.

• The complexity of an OTT service, which includes many different technologies from video encoding and delivery to content management, can be challenging to simulate.

• It’s hard to simulate real-world conditions in lab tests. For example, multiple users may access a single server for hosting media files or streaming live events.

• Due to limited resources and time constraints, testing all possible combinations of devices and use cases is difficult. For example, if your app supports only iOS or Android devices, then you’ll need to create separate test scenarios for each platform — which can be time-consuming and expensive if you have thousands of device models.

• Difficulty defining and maintaining SLAs (service level agreements) between providers (OTT vs. CDN). For example, if an operator provides CDN services for an OTT provider such as Netflix or YouTube and uses the dedicated infrastructure, it may not be easy to guarantee SLA levels due to their relationship with these parties. In addition, how does one monitor traffic flows across multiple networks when there is no single control point?

• Lack of visibility into traffic patterns across multiple networks makes it challenging to establish SLAs between 

Network testing can be divided into two categories – performance testing and functional testing. 

Performance testing ensures sufficient bandwidth and no congestion issues are in place. Functional testing ensures that an application can properly connect to a server and send/receive data packets without any packet loss or latency issues.

While functional testing is an essential aspect of network testing, there are some challenges associated with it as well:

• Functional tests cannot be automated; they require manual intervention and human input, which makes them time-consuming and expensive to run.

• Functional tests cannot be easily extended to cover all possible use cases; this means that gaps will always exist in your coverage that might not be discovered until after the launch.

The challenge lies in ensuring different aspects of the network are working together to provide a quality experience for consumers. As it stands, there is no way to test the network performance for all types of OTT applications.

Best practices to assure QoE

Best practices to assure QoE (quality of experience) in ott application testing from the network side are:

1. Perform end-to-end tests from source to sink. This includes all the components that provide an end-user with a seamless experience when watching live TV on their devices. This can consist of devices such as set-top boxes and digital TVs, as well as back-office systems such as billing and customer management systems.

1. Use Real-Time Monitoring Tools: Real-time monitoring tools collect real-time performance statistics related to parameters such as jitter, packet loss, and latency. This helps identify any issues early, which could otherwise lead to a poor user experience. For example, if there is a sudden increase in jitter or latency on one or more streams, it can be identified immediately with these real-time monitoring tools, and appropriate action can be taken. You can make adjustments when necessary to ensure ongoing improvement and optimization. Use monitoring tools such as RUM (accurate user monitoring) technology, which provides insight into how users use your OTT services.

1. Ensure the best possible performance by using next-generation networks such as 5G that provide greater bandwidth, lower latency, and higher reliability than previous generations of wireless technology.

1. Understand the network traffic generated by the application and the bandwidth requirements of each piece of content.

1. Check that the network can handle the scale of traffic and bandwidth required by the application.

1. Ensure sufficient capacity exists to provide service during peak load times (e.g., during high-demand periods such as peak viewing hours).

1. Ensure that your streaming service has sufficient redundancy and failover mechanisms in place so that if a server goes down, another is ready to take over without any interruption in service.

1. Check with your ISP (Internet Service Provider) or cable company to make sure they have enough capacity in place to support your OTT streaming service needs, both now and in the future, as you grow your subscriber base and increase your bandwidth requirements over time (e.g., if you are planning on streaming 4K video content someday soon).

1. Evaluate different delivery methods for video content, such as adaptive bitrate streaming vs. HTTP/RTSP/RTP, as well as other protocols like HLS vs. DASH, to determine which type of stream will work

1. Use a network emulator such as PacketZoom’s Emulator or Wireshark to measure packet loss and latency at all levels of the network stack before and during your ott testing platform.

1. Use Video Stream Analysis tools to identify problems with video quality, such as buffering during playback or poor resolution at certain times of the day.

1. Check that the streaming server has enough capacity to handle all traffic without any congestion issues by running stress tests on it before your live launch.

1. Use Network Performance Monitoring Tools: Network performance monitoring tools provide detailed information about various parameters such as throughput, latency, and jitter, which are required for analyzing QoE from a network perspective. These tools also help identify problems at an early stage so that they can be resolved before they cause any harm to your OTT services

1. Make sure the test is run in a controlled environment.

1. Do not use any artificial traffic generators during the test.

1. Use multiple clients and data collection points to collect network data.

1. Use multiple CDNs and content delivery networks (CDNs) in your media testing strategy to ensure no single point of failure for your OTT strategy.

1. Test with various devices and operating systems (OSs) as well as on different networks and geographical locations so that you can get a fair idea of how well your content will perform with various devices and OSs.

Conclusion

The revolution in media consumption and the over-the-top market is here to stay. Nobody can ignore the advantages of ott testing platforms as they are redefining, reshaping, and transforming the way we watch television and movies. 

However, to ensure a better OTT experience, there is a need for more consideration of the relationship between content owners, service providers, and the users in the distribution chain. Network testing needs to mirror real-world use cases using an appropriate mix of commercial traffic, real-user tests, and synthetic emulation.