Event ticket seller Ticketmaster experiences the traffic equivalent of a huge DDoS attack every time a major gig or show goes on sale, necessitating a steady migration to the cloud.
That’s according to Simon Tarry, director of engineering strategy at the firm, speaking to V3 ahead of the V3 Cloud and Infrastructure Live event on 20 and 21 April 2015.
“Ticketmaster’s been handling web traffic for almost two decades now, so we’ve built up our own infrastructure, and part of the problem with that, as a US company, is we’ve grown through acquisition and bought up a lot of ticketing businesses round the world – as well as all the infrastructure that comes with it,” Tarry explained.
With ticketing platforms scaling, and a growing audience increasingly consisting not just of fans “but automated bots as well”, Tarry said Ticketmaster’s existing infrastructure was reaching critical mass.
Ticketmaster currently handles around 1,300 users per second at peak times.
“We basically suffer huge DDoS attacks from a large on-sale, so we try to separate our human traffic from bot traffic,” Tarry told V3.
Having already been using a private cloud for the past three years, Ticketmaster is now going through a “strategic push” to AWS.
“We’re assessing at the moment that kind of journey – how to move what we have into an AWS architecture. So a lot of planning and training is going on right now,” Tarry said.
The initial migration to private cloud was an e-commerce stack for a one-off event.
“We tried that first, as a short project, as it was limited in scope to a certain degree,” said Tarry.
“But we made a lot of assumptions about the infrastructure that weren’t true,” he added.
“So we had to challenge a lot of our thinking about the infrastructure and how it would perform. The key criteria for us is to handle a very large on-sale on any platform.”
Keeping the lights on and maintaining the ability to cope with a punishing level of traffic was achieved by “strong tooling”, said Tarry, including load testing products from SOASTA. Mechanisms to interrogate the traffic in order to block out traffic Ticketmaster doesn’t want also help in this.
“Part of our DevOps culture is a kind of ‘swat team’ of guys who play ‘hunt the bottleneck’, spending time diagnosing, testing, and finding the next problem,” Tarry said.
“Ultimately, when we’re cloud based we want that capacity on-tap – it’s not something you can just do,” he said.
“You need to configure your systems to use that capability.”