Within the world of internet betting, another ... mechanisms and tools that can be used to ensure .... load testing is n
HOW TO ENSURE THE
STABILITY OF AN ECOMMERCE
PLATFORM AT TIMES OF PEAK DEMAND Written by Jon Bramley
Players • Performance • Partnership
TABLE OF CONTENTS 03 INTRODUCTION 04 WHAT IS A PEAK? 05 SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORM 07 HOW DO WE MANAGE PEAK LOADS? 08 EXPECTED PEAKS 10 POSTMORTEM 11 AN OMNI-CHANNEL PLATFORM 12 SUMMARY 13 WHO ARE OPENBET?
HOW TO ENSURE THE
STABILITY OF AN ECOMMERCE
PLATFORM ” “
Within the world of internet betting, another bookmaker is just a click away so performance and stability are paramount to a successful day.
AT TIMES OF PEAK DEMAND
INTRODUCTION The world of sportsbetting goes in cycles driven by the major sporting events of the year. Bookmakers want to use these events to maximise both their revenues and customer acquisition opportunities.
This means being prepared for both new levels of load on the platform and ensuring consistent high performance and stability throughout new peaks and alongside new, potentially disruptive, functionality.
This means releasing new features, running new promotions, launching dedicated microsites, etc.
In this paper we will explore different mechanisms and tools that can be used to ensure success and stability across these high profile, peak days.
So not only is the platform under increased customer demand due to the natural draw of the event but also due to the effects of the increased marketing.
3
WHAT IS A PEAK? Any system can have periods or days of unusual traffic. In the sportsbetting world we are used to ‘peaky traffic’ due to the pattern of betting naturally being weighted towards specific events and increasing just before the start as everyone gets their last-minute bets in. Saturday afternoon just before the Premiership matches all kick off is traditionally a heavy betting period, for example.
This national institution sees millions of otherwise non-betting people having their annual punt on the horse race and puts a load on the system of up to 400% of a regular Saturday.
vs
49,000
49,000
49,000 15.3 Million ONLINE BETS PER MINUTE
ONLINE BETS PER MINUTE
ONLINE BETS
However, to bring some real perspective to the challenge of peak loads, the most demanding day in the UK bookmakers’ calendar is the UK Grand National, held each April.
5,160
5,160
ITEMS SOLD PER MINUTE
7.4 Million
ITEMS SOLD PER MINUTE ITEMS SOLD
ONLINE BETS PER MINUTE
15.3 Million
Grand National 2015 for OpenBet
ONLINE BETS
7.4 Million Black Friday 2015 for Amazon
ITEMS SOLD
15.3 Million ONLINE BETS
4
SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORM Although a betting website could be thought of a just a specialised e-commerce platform, it actually has a number of characteristics that differentiate it and increases complexity.
Shifting product catalogue A large scale bookmaker will have around 4 million active betting opportunities live at any one time - but these opportunities do not live long and the entire catalogue will be replaced with another set incrementally over a period of just a few weeks. Within that time the price of each item will change many times. Over the course of a 90 minute football match, the 4,000 betting opportunities available will have over 35,000 price changes applied. And that’s just for one match - we will have over 200 events in-play simultaneously and many thousands more available for pre-match betting.
Complicated purchase process The purchase process in sports betting involves: • Validating the player’s account and their balance • Validating that the player has not exceeded their betting limits • Validating that the betting opportunities selected have not exceeded their risk limits • Validating that the betting opportunities can be combined in the chosen way • Validating that the opportunities and requested prices are still valid • Recalculating liability for the bookmaker once the bet is placed
5
SPORTSBETTING IS NOT A REGULAR ECOMMERCE PLATFORM Settlement and payouts The biggest difference between betting and a regular ecommerce platform is, of course, what the player is buying: a bet. At some later point, the bet needs to be settled against the bet result and potentially pay the player their winnings.
The main consideration here is that the settlement process itself is computationally intensive and also needs to happen as soon as possible after the result is available, both give the player a good experience, but also to maximise the recycling of winnings into another bet.
160000 140000 120000 100000 80000 60000 40000
Per Event
Football
Tennis
Cricket
Number of markets e.g. First goal scorer
1008
830
3868
142577
43688 30313
20000 0 FOOTBALL
TENNIS
CRICKET
Total updates
Number of market updates e.g. activation/suspension
8163
6693
39551
Number of selections e.g. Arsenal to win 2-0
4167
2353
11980
Number of selection updates e.g. odds change suspension
35520
23602
102810
Total updates
43688
30313
142577
6
HOW DO WE MANAGE PEAK LOADS? Due to the cyclical and peaky nature of sportsbetting our platform is already built to manage these types of load. Constant development of the system means that performance needs to be constantly monitored. Each system is looked after by a dedicated lead who understands that system’s ‘normal’ and works closely with operators to manage network and hardware growth alongside the system.
Total bets per hour
• We make heavy use of caching even for durations of only a couple of seconds to smooth out the large number of data requests and changing prices.
1
2
3
4
5
6
7
8
• We use push technology to publish price and status updates directly to connected users. • We cache HTML snippets to help build dynamic pages. • Our in-house Application Server is built to scale both horizontally and vertically and can dynamically use resources as needed. • Functionality can be split across servers to balance load and a server can be brought online in seconds and use our caching layer to manage unexpected peaks. • No player state is held at the Application Server level so traffic can be managed by any server. • Active monitoring of software, hardware and database usage catches issues before they affect players.
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
7
EXPECTED PEAKS
Beforehand
For those periods when we do expect very high peaks - like the Grand National - then we are able to plan and manage these days differently.
01
During the final few weeks, load testing continues and bookmaker marketing efforts are put into action.
04
It all starts months before, where a working group of technical experts review the previous year’s performance and lessons learned. Milestones are agreed alongside a readiness checklist that has evolved over the years
1/2 MILLION NEW ACCOUNTS
CREATED IN
24 HOURS
8
NEW ACCO
CREATE
24 HOU BE
MO
THE GRAND NATIONAL 2014 IN NUMBERS: 2014 IN63% NUMBERS: 63%
8M
1/2 MILLION
1/2 MILLION
1/2 MILLION NEW ACCOUNTS
CREATED IN
24 HOURS
02
ACCOUNTING FOR
8M
ACCOUNTING FOR
8M 63% 8M 63% 11 Billion 1 Billio 1 11 Billion About a month before Grand National NEW ACCOUNTS
NEW ACCOUNTS ACCOUNTING FOR
1/2 MILLION CREATED IN
CREATED OF REMOTE BETS ON IN
ACCOUNTING FOR NEW ACCOUNTS
24 HOURS CREATED IN
OF REMOTE BETS24 ONHOURS
03
MOBI LE
BETS PLACED M OHOURS BILE 24
BETS ON MOBI LE
ON SATURDAY
OF REMOTE
BETS ON MOBILE
OF REMOTE BETS PLACED ON SATURDAY
BETS PLACED ON SATURDAY
BETS PLACED
ACCOUNT TRANSACTIONS ACCOUNT TRANSACTIONS
ON SATURDAY
SPORTS BETS
another racing festival - Cheltenham - is Closer to the event, a feature freeze is implemented that a useful yardstick for the preparedness of allows the team to focus on performance and stability. BETTING OPPORTUNTIES the system andCONCURRENT allows further performance BE Enhanced load testing is started to spot weak areas. This2014 IN NUMBERS: ANY TIME ON THE PLATFORM CONCURRENT BETTINGAT OPPORTUNTIES 2014 IN NUMBERS: AT ANY TIME ON THE PLATFORM and stability changes to be made if the BETS PER MINUTE AT O load testing is not trivial to prepare as it must predict the levels of activity across the whole system: from 2014 IN NUMBERS: post-event analysis shows it is necessary. 1/2 MILLION
4 Million 4 Million
32,003
2014 IN NUMBERS: 8M 11 Billion 11 Billion 1 Billion 1 Billion ACCOUNTING FOR
registrations, to payments, casino activity, NEW ACCOUNTS 63% system 1/2 MILLION CREATED IN OF REMOTE NEW ACCOUNTS ON ACCOUNT TRANSACTIONS transfers, bet placement andBETS bet settlement. All of which ACCOUNT TRANSACTIONS 63% O B IL E CREATED24 IN HOURS BETS ON M OF REMOTE place unique stresses on the system and, in particular, the 24 HOURS M O B IL E centralised player account. ACCOUNT TRANSACTIONS
8M
ACCOUNTING FOR
1 Billion 11 Billion11 Billion 1 Billion 4 Million 4 Million 32,00032,000 BETS PLACED
SPORTS BETS
SPORTS BETS
ON SATURDAY
BETS PLACED ON SATURDAY
ACCOUNT TRANSACTIONS
SPORTS BETS
SPORTS BETS
CONCURRENT BETTING OPPORTUNTIES CONCURRENT BETTING OPPORTUNTIES BETS PER MINUTE AT OUR BETS PEAKPER MINUTE AT OUR PEAK AT ANY TIME ON THE PLATFORM AT ANY TIME ON THE PLATFORM
4 Million4 Million32,000 32,000
CONCURRENT BETTING OPPORTUNTIES BETS PER MINUTE AT OUR PEAK AT ANY TIME ON THE PLATFORM CONCURRENT BETTING OPPORTUNTIES BETS PER MINUTE AT OUR PEAK
2014 IN NUMBERS: AT ANY TIME ON THE PLATFORM 2014 IN NUMBERS:
11 Billion 11 Billion ACCOUNT TRANSACTIONS
ACCOUNT TRANSACTIONS
1/2 MILL
1 Billion 1 Billion SPORTS BETS
SPORTS BETS
8
EXPECTED PEAKS On the day itself, communication is vital. Each operator team is present and locate themselves in a single area of the office to make coordination easier. Slack is used for instant messaging with two dedicated channels: one for general communication across teams and another for the escalation of incidents. Account managers and the executive team are subscribed to this second channel. In the case of issues, communication to all other teams is key so they can be prepared for additional traffic (players moving to a site that is available) and to look out for similar issues. Experts jump in to analyse the problem and get it fixed as quickly as possible and identify the cause in case of knock-on effects to other operators. Throughout the day activity and load is constantly recorded for later analysis and to help plan for the following year.
On The Day
• Registration, payments and bet placement are the key focus areas for stability early in the day but this shifts to bet settlement after the events leading up to the big races to ensure players have their winnings returned to them for further bets they’d like to place. Settlement can be throttled by the amount of system resources dedicated to it but also can have prioritisation applied based on specific events, markets or even betting selection and by prioritising winning bets over non-winnings ones. The winning customer experience is key and postponing the settlement of non-winning bets to a quieter period later in the day saves vital system resources for continued bet placement. • Potential Bottlenecks caused by the wide number of third party systems our platform integrates with (payments, verifications, system transfers) are monitored carefully with timeouts kept as low as possible to prevent execution threads getting locked up. • As pressure builds, there is a list of non-essential activity and functionality that can be disabled to ensure the key functions of registration, login, and bet stay functional.
15.3 Million ONLINE SPORTS BETS
49,000 ONLINE BETS PER MINUTE
60 Million ONLINE ACCOUNT TRANSACTIONS
74% Bets PLACED VIA MOBILE CHANNEL
• Active monitoring of the load on each part of the system is used to balance functionality across servers more evenly and increase capacity where the models didn’t predict load accurately.
9
After the event
• Analyse any major issues, their route cause, immediate resolution and if any longer term solution is required • Analyse performance versus load for bottlenecks and hot points, and determine solutions. Add these to the backlog with priority based on the severity • Analyse the load patterns to feed into updated load test profiles • Share findings across teams and add shared learnings to the backlog • Carry out a retrospective for the day to take forward suggestions into next year
65 Account Transactions (millions)
POSTMORTEM
60.5
55 52.9
45 43.3
35 32.6
25
2013
2014
2015
2016
Year
10
AN OMNI-CHANNEL PLATFORM Sportsbetting has increasingly and rapidly become a mobile-first activity. The advent of the smartphone has materially changed the platform usage profile of online betting such that it is now the majority of online bets that are taken via mobile. For some operators over 80% of bets are taken via mobile channels, but even the annual average across all operators is over 70%. OpenBet’s omni-channel platform separates the functionality of the sportsbook from the player front-end and so it does not matter where the activity is taking place - the same robust back-end processes all bets and the same functionality is available to all channels. It is the responsibility of the front-end application to deliver the right player experience for the channel being used.
Grand National Mobile Usage
