Demystifying and Tackling Live Streaming Latency
One of the most notable market trends of late in the media and entertainment industry is the growth of live streaming. The global live streaming market rose from $1.24 billion in 2022 to $1.49 billion in 2023 at a compound annual growth rate (CAGR) of 20.6%. According to a Research and Markets report, it is expected to grow to $3.21 billion by 2027 at a CAGR of 21.2%, suggesting the market’s momentum is set to continue.
A key driver of this trend is the evolution of streaming consumer behaviour. Deloitte’s 17th annual Digital Media Trends report found that consumers, especially younger ones, have shifted from watching TV shows and movies at home in favour of user-generated content (UGC) on social media and a desire for social connection, often through the vehicle of video games (Deloitte said almost half of Gen Z and millennial gamers say they socialize more in video games than in the physical world). Some of the most popular UGC today is from social media users streaming live video directly to their followers, while video games with strong social dynamics like Fortnite rank among top contenders on live streaming platforms.
Pandemic restrictions in 2020, combined with monetization opportunities, have acted as secondary drivers of live streaming from a business perspective. Live streaming has emerged as a lucrative revenue stream for content creators, influencers and businesses. Many across the media and entertainment space have leveraged live streaming technology to engage broader audiences or to better support advertisers for live sports events, fitness classes, virtual learning, music concerts and art auctions. Thus, for instance, in June 2020 Sotheby’s threw the first live streamed global art auction event that attracted buyers across three continents: New York, London and Hong Kong. The second Sotheby’s global live streamed evening sales event that year garnered an online audience of 150,000. Christie’s entered the live streaming arena the same year, drawing approximately 100,000 viewers to its first live streamed auction, highlighting the growing prominence of both digital platforms in the art auction business and live streaming in the world of entertainment.
Given the dramatic rise and expected future trajectory of live streaming owing to these social and economic drivers, it is critical that live streaming platforms and solutions deliver smooth, reliable, high-quality experiences to ever larger and more dispersed audiences. One of the keys to achieving this objective is tackling the live streaming latency problem.
Defining Latency, and Why it Matters
Latency can be simply defined as the time between the initialisation of a data transfer and the actual start of that data transfer. In terms of live video streaming, it can be interpreted as a delay between the moment a video frame is captured and the moment it is displayed to the user — it is also fair to characterize latency as a delay between the video stream clock and the real-time clock.
For live streamed events like video game tournaments, sports contests or art auctions, latency is a crucial aspect. Any streaming lags, even as much as a 10–30-second delay, can lead to disrupted streams and a less reliable user experience, which is unacceptable in a setting where real-time information means everything and each second counts.
Given the rise in recent years in both viewers and streamers, tackling the live streaming latency problem today is more essential than ever. In the latest Bitmovin survey, 25% of participants — comprised of video developers and industry experts across 42 countries — cited low latency among their biggest challenges. It also took the third highest spot for where respondents see the most opportunity for innovation in their services.
If achieving low latency is such an important challenge, and is seen as a prime focus for innovation, what are the root causes of the live streaming latency problem, and how can they best be addressed?
The Causes of Latency and How to Address Them
One important aspect that affects latency in live streaming is video encoding. During its streaming journey, a video needs to be encoded, transported and decoded. Encoding high-quality videos absorbs a lot of computing power, which inevitably adds latency to the process of encoding live streams. The higher the quality of the input and output streams, the larger the latency impact can be.
Another cause of latency is rooted in the very nature of the pseudo-streaming protocols used by most OTT streaming technologies, like HLS or MPEG-DASH. These protocols slice video stream content into small segments or “chunks”, which are then downloaded sequentially by the receiving device. The larger the size of these data chunks, the longer it can take for each of them to download to reach the viewer. Conversely, smaller, or shorter video segments contribute to faster downloads and a smoother viewing experience.
Latency can be seriously affected by network configuration issues and congestion, and there are several ways to mitigate them. In most cases, an adaptive bitrate technique helps in resolving network configuration issues and issues related to lower bandwidth. The Content Delivery Network (CDN) should be optimized for latency by, for example, leveraging HTTP/2 Push and HTTP/3 protocols for faster data transfer, and by using multiplexing and header compression. Ideally, a CDN provider should have a direct connection to the major ISP networks in the region to facilitate latency optimization. And finally, the number of CDN servers should be adequate to ensure that the physical distance between the point-of-presence and the end user does not affect latency. When a CDN server is physically closer to the user, the route between the user and the service is much more predictable, resulting in less variability in latency.
HLS and MPEG-DASH utilize established CDN infrastructure for video delivery, ensuring cost-effectiveness and scalability. However, there is typically a 30-second delay between real-time events and end-user screen delivery, which is unsuitable for live sports and similar applications. How can we achieve minimal delay while leveraging existing CDN infrastructure? Low-latency pseudo-streaming protocols like Low-Latency HTTP Live Streaming (LL-HLS) and Low Latency Dynamic Adaptive Streaming over HTTP (LL-DASH) offer improved versions of HLS and MPEG-DASH standards, addressing latency concerns.
The main idea behind these superior pseudo-streaming protocols is to shorten video chunks as much as possible, to around 300–500 milliseconds (ms) each in size. At this level they can be delivered much faster with less delay, thus decreasing the latency in rendering the first video frame. This capability has been made possible by the evolution of CDNs, which are now equipped to manage more frequent requests for smaller video segments.
However, even latency of 300 ms can be too high for such applications as online games with live voice and video chat features. When super low latency is required, real-time streaming protocols must be used. In real-time streaming, a connection is established between communicating parties, and data is transferred as soon as it is generated, frame-by-frame. Whenever a new chunk of data is ready, it will be pushed to the existing connection.
There are two well-established solutions used for bidirectional, real-time communication: RTMP and WebRTC.
RTMP is a classic client-server protocol that has been around since 1996. It was designed by Macromedia and later acquired by Adobe. RTMP has been instrumental in the success of numerous projects and video applications, including ChatRoulette and Periscope. However, nowadays it is considered insecure to create a plugin that can connect to other servers directly over TCP or UDP. Therefore, RTMP playback has not been supported in web browsers since 2016. Nonetheless, RTMP is still used in numerous applications, such as pushing video streams to YouTube or Facebook. Additionally, multiple drone vendors primarily use RTMP to deliver their feeds to a remote server.
In 2011, Google started the WebRTC project, which provided the means for real-time communication without directly allowing access to TCP/UDP for plugins and JavaScript. This project became the standard for real-time communication on the Web, including video chats and file transfer with sub-second latency.
Strategies for Tackling Latency: Case Studies and Solutions by DataArt
DataArt has considerable expertise and experience in combatting live streaming latency issues, ensuring real-time processing, filtering, and transformation of live video streams, and implementing a variety of protocols for video content delivery and real-time communication.
For instance, DataArt has built and configured a video CDN with hundreds of servers for one of their client’s events, involving video live streaming from hundreds of cameras.
In another case for a startup offering a multi-camera live streaming platform for bloggers, influencers, and content creators, DataArt used the WebRTC protocol to ensure a smooth transition between live feeds from multiple devices and with minimal delays. The main problem was significant delays in WebRTC real-time communication with mobile device users. In fact, due to the Network Address Translation (NAT) configuration, most of the traffic was being relayed to external TURN servers, which were physically remote from one segment of the target audience. This latency challenge was solved by introducing additional Traversal Using Relays around NAT (TURN) servers. The solution required strategically placing additional servers in the affected region and adjusting DNS to resolve TURN servers based on latency and geo proximity. The result was real-time video stream delivery to the web-browser and backend server with minimal, sub-second delays.
DataArt has also implemented low-latency solutions around pseudo-streaming protocols, such as delivering live video feeds from wearable cameras through the RTSP protocol, and subsequently delivering this content to viewers using the LL-HLS protocol. In this scenario, a two-second delay was acceptable, and the retooled infrastructure was much cheaper to maintain than the previous one involving WebRTC.
Concluding Thoughts
As the live streaming market continues to thrive and grow further into the future, tackling the related latency challenges remains a crucial and ongoing endeavour. It is fascinating to see how low-latency pseudo-streaming protocols are continuing to evolve to keep pace with these challenges. If you are looking for ways to expand your business with live streaming audio or video services, or to improve your current platform’s live streaming capabilities, learn more from DataArt’s Media & Entertainment experts.
Author: Doron Fagelson,
Vice President of Media and Entertainment Practice at DataArt
Originally published on https://www.dataart.com/blog/
