In what can easily be described as a nightmare for any broadcaster, Google’s YouTube TV service went down for around an hour during the FIFA World Cup 2018 semi-final between England and Croatia earlier this week. Yes, that means people in the US who paid $40 per month for the service ended up missing out on an entire half of one of the biggest games in football. Had a similar scenario played out in India, most people would probably not be very surprised.
However, the exact opposite happened during a recently concluded cricket tournament — Indian Premier League (IPL) 2018. Indian streaming service Hotstar successfully served 10.3 million concurrent viewers during the final minutes of the match.
So how did Hotstar do what YouTube TV could not, that too in a country known for some of the world’s slowest Internet speeds? The answer lies in months of preparation and optimisation, which is where Hotstar’s content delivery partner Akamai steps in.
Rishi Varma, director of product management at Akamai Technologies, tells Gadgets 360 that for a big tournament such as IPL 2018 the preparations begins far in advance. “It doesn’t matter which part of the world you are in, if you really have to deliver events this size, the biggest thing to that is preparation,” he says. "We make very educated calculated estimates of what we should be thinking of. This goes down granular details of how many viewers we expect, what’s the kind devices we expect them to connect from, what is the throughput you want to deliver to those devices, etc.”
Hotstar CEO Ajit Mohan tells Gadgets 360 that a team of around 50 engineers worked on streaming IPL 2018. “In terms of readiness, a lot of work is just around demand modelling,” he explains. "We now have done enough cricket for the last 3.5 years to know the demand… We are geared to know what the peak is before even the peak happens. You are also making sure that you are going in with a plan, and respond on the fly.” Mohan says.
The journey for Hotstar, which claims it has 150 million monthly active users, has not been smooth at all. The service experienced outages when it tried to air Game of Thrones “minutes after America”. While Hotstar didn’t reveal the number of people trying to watch Game of Thrones on a Monday morning in India, it’s safe to say that it was far less than the number of people trying to follow a cricket match live. Last year English Premier League live streams were unavailable on Hotstar’s Apple TV app for a few months due to “technical issues”.
“I’ll give you one more, third example… days after we launched,” says Mohan. “If you remember the India-Pakistan opener in the 2015 World Cup, we faced the same experience that some of the platforms are experiencing for FIFA now. We went down right off the bat within a few minutes."
"Because we have a large platform with surges happening with live sports, I think it has allowed us to design some of these choices that I have described including graceful degradation,” he continues.
Akamai’s Rishi Varma echoed these thoughts. “Whether it is a regular VOD stream like people watching a movie or a live event, the problems that can occur are anywhere in the chain,” he says. "For live events it’s just a lot more costlier because you only have a limited window to fix it. You can’t take an hour and a half to fix a football game, by then the game is over.”
From Akamai’s end one of the failsafes is having multiple backup streams running at the same time. “Usually we have multiple backups running so even if the primary falls there is technology that takes care of gracefully fading over to a backup,” says Varma. "It’s so transparent that people don’t even realise that they are looking at a backup stream.”
While no one can predict where a problem might arise, having failsafes and what Hotstar terms “graceful degradation” has been the key to their recent success. Having multiple streams running at the same time and serving the secondary or tertiary stream if the primary goes down is one way to avoid an outage. The other is to ensure that even if one part fails, the entire house doesn’t come crumbling down.
“The surge in demand usually happens within milliseconds or microseconds. You can move from 1 million to 5 million in milliseconds,” says Mohan. "Your platform and technology has to be able to gear with surge that won’t happen on any other content.”
Given the mobile-first nature of Hotstar’s target audience, Akamai optimised its network to serve video to smartphones. Over 90 percent of Hotstar's traffic for IPL 2018 was from mobile devices, with over 90 percent of them being Android phones, according to Akamai.
“If you understand the audience you need to deliver to, you can make sure that you have the right infrastructure to deliver,” says Varma. "There are some specific parts of our product portfolio that help address the scale issue and help protect or insulate the broadcaster’s infrastructure and let us as the delivery network absorb the entire force of what’s coming in.”
In spite of all of those impressive achievements, the stark reality is that online broadcasts of live sports still lag behind their TV counterparts. Depending on the device you use, the “live” stream online could be as much as 30 to 45 seconds behind TV. If you think about it in terms of major events such as the cricket World Cup, you’d hear the celebrations of people in the neighbourhood before seeing the big moment on your “live” stream.
Hotstar’s Ajit Mohan tells Gadgets 360 that the company’s been hard at work trying to get the stream as close to TV as possible. Right now on Apple TV the latency on Hotstar is around 10-13 seconds, he says, adding that the number is 30 seconds on Android phones, and seven to eight seconds on the Web.
Akamai, on the other hand, is working on tech that will push online live streams ahead of TV. “Now we are at two to four seconds behind the TV broadcast,” says Varma. "We are working on technology that will possibly take us ahead of TV broadcast… I would argue that we are at the cusp of that happening already.”