What if Azure had helped me sign up for Free Mobile?
Archive media: Some illustrations are no longer available at their original addresses. Their original links are preserved below.
Free’s arrival in the French mobile market was… explosive. So was its signup website. It was effectively unavailable on the first day. The first registrations only became possible the following day.
The point here is not to judge the prices of the new plans, the quality of the service, or whether you get reception in the remotest corner of the Gers — although the Gers Liberation Front obliges me to say that the Gers is great. As a developer, I asked myself: If someone came to me tomorrow and asked me to design a website capable of handling one million requests per minute, what would I say?
Now you know what this post is about :).
This post is just an extract from some thinking I did during a four-hour car journey on the Wednesday after the Free Mobile announcement. It mixes ordinary observations with points that may seem obvious to many developers. Theories such as “Free deliberately restricted its service,” or “they hadn’t thought of all this” versus “they had thought everything through,” are not what interests me here. I want to share the thought process and some points that are probably unfamiliar to far too many web developers.
While writing, I quickly realized this would be a very long post, so I decided to turn it into a series. If you have comments, please use the comment section :).
The facts and requirements
We are launching a website that will receive very heavy traffic for a more or less sustained period: perhaps 72 hours of madness, followed by two or three weeks of very high demand. The site will let visitors:
- View a page presenting the offers.
- Download files such as price lists and contractual terms.
- Sign in if they are already subscribers.
- Subscribe in four steps: choose a plan, enter personal details, provide banking details, and choose a phone number or transfer an existing one.
The essential question is: what volume should we expect? One million requests per minute, according to Xavier Niel on Le Grand Journal on January 11, 2012. Let’s get to work!
Let’s pause on that number: one million hits per minute, 60 million per hour, 1.8 billion per month. France’s most visited website, leboncoin.fr, gets 5.8 billion a month, if we equate “request,” “hit,” and “page view.” The second French site gets 0.6 billion. In just a few hours, we have gone from zero traffic — setting aside the buzz phase, which had only a basic HTML file — to the second most visited site in France, ahead of L’Équipe, Le Monde, Le Figaro, seloger.com, and others. Not bad, right? :)
Ping me if you can
At this volume, even the smallest component becomes critical. We need to go through everything that happens when someone visits the site, step by step, and optimize each operation.
We start by opening a browser, typing http://mobile.free.fr, and pressing Enter. As the site’s operator, there is a whole part of the chain we cannot control: the visitor’s Wi-Fi, the quality of their ADSL line, and so on. The first link we can influence is DNS. But DNS… who cares, right? We have it with our registrar, or even on our web server. Isn’t that enough?
In reality, DNS is one of the most underestimated parts of website hosting. Between hosting and configuration, there is already plenty to think about.
Where is my DNS?
Your site’s DNS servers are usually provided by the company where you registered the domain. That is not compulsory. Many companies offer to manage DNS for you. You can also host it on your own servers, although that last option is not recommended in most situations.
How do you choose between your registrar and a third-party service? First, look at the registrar’s DNS service. Some limit the number of records; others limit record types, excluding SRV, TXT, or AAAA records for IPv6. One factor that might make you switch to a dedicated service is the SLA. What guarantees does your registrar offer? Installing DNS servers in your own cloud infrastructure is not necessarily the best answer either. DNS services are often among the most fragile points of Internet service architecture, including cloud architecture.
Besides mapping free.fr to x.x.x.x, what can DNS do?
If we focus only on web traffic, which needs to translate a domain name into an IP address, DNS is one of the cheapest load-balancing systems available. With round-robin DNS, the server returns different answers to successive requests. This lets you associate a list of IP addresses with one DNS record. Be careful: ALL those servers must be able to answer requests in the same way, running the same application version and accessing the same data. We will return to that later. A variation, weighted round robin, assigns weights to possible answers, so that the IP addresses of more powerful servers are returned more frequently.
DNS and application updates: Cloud infrastructures often duplicate all or part of an environment, or at least make doing so easier, aside from cost. DNS makes it straightforward to promote a staging or preproduction environment to production: change the IP address in DNS. Rolling back is just as simple. Windows Azure instances are a good example of simplicity here, although Azure operates one level below DNS by changing virtual IP addresses rather than DNS records. Azure VIP swaps are an even more effective solution for high availability and load management than DNS-based distribution.
Here, then, is our first tool for handling that traffic: DNS-based load balancing.
Are you hiding something from me?
Me? No. But your DNS servers routinely cache things. DNS records have many layers of caching: the DNS server’s TTL, or time to live; your ISP’s DNS servers; your router’s DNS; corporate DNS servers; the operating system’s DNS cache; and the browser’s cache — one minute in Firefox 3, fifteen minutes in IE 7.
Caching is helpful for our current problem because it reduces traffic to our DNS servers. But if you have to change server IP addresses in the middle of the rush, it can cause problems.
That brings us to one of the least considered DNS settings: TTL. It tells components caching a DNS response: “You may store this, but after [the TTL], ask me again, because my answer may have changed.” Set it to one day, and much of your traffic — everyone who visited before the change and whose cached answer remains valid — will still go to the old server. If that server has failed, those visitors get no service… for a day.
Here are the TTLs of a few major sites:
| Site | TTL in seconds |
|---|---|
| amazon.com | 60 |
| microsoft.com | 3600 (1 hour) |
| facebook.com | 3600 (1 hour) |
| Google.com | 300 (5 minutes) |
| youtube.com | 600 (10 minutes) |
| mobile.free.fr | 6400 (1 hour, 46 minutes, 40 seconds) |
The values are relatively short. A short TTL generates slightly more DNS traffic, but it also allows changes without an excessive propagation delay. If you need an emergency hosting change, a one-day TTL, which is quite common, can leave the site unavailable to some visitors for up to a day.
An effective DNS solution?
Finding an effective, affordable, secure DNS solution that can handle substantial load is not easy. One of the most interesting options, in my opinion, is Amazon Web Services Route 53. For an affordable price — 0.50 per million requests — it provides all the services mentioned above, with a 100% SLA and compensation for unavailability.
What goes behind DNS?
We know very little about the technology behind mobile.free.fr: an nginx proxy, JavaServer Faces, and some jQuery. We also know nothing about the overall infrastructure. This part will therefore rely first on assumptions, and second on a choice of Microsoft technologies. Surprised? :)
The challenges
From a user’s perspective, the signup site provides these functions:
- Display information pages.
- Download documents such as PDFs of contractual terms.
- Sign in with existing Free credentials.
- Complete a registration form.
- Transfer an existing number or choose a phone number.
- Receive access to the customer area by email.
The main contention points are in bold. Signing in with Free credentials requires access to an authentication service; we will assume that copying every username and password is not the chosen solution. Two people must not select the same phone number simultaneously.
One of the first things we can do is set up a CDN. Many providers exist, starting with Windows Azure. A CDN replicates content across many servers around the world that are specifically configured to deliver static files. One consideration here, as with global cloud services in general, is that they do not necessarily have a node in France, and certainly not several French nodes. A CDN is still useful because it can deliver static content — PDFs, CSS, and images — much faster than the web server. If we served only one country for a longer period than this traffic spike, such as a national broadcaster’s replay website, our strategy might be slightly different.
Displaying information is not the hardest part. We might think simple HTML pages are the best answer, but that is not enough. Even a basic HTML page involves performance considerations: HTML size, the number of referenced CSS and JavaScript files, minification, and so on. Looking closely at Free Mobile’s site, some of these rules are not followed. I use Yahoo’s indispensable Y!Slow plugin to check this. It assesses a website against the major performance criteria defined by Yahoo’s teams.
Moving to Windows Azure
Windows Azure is available, so why not plan to deploy our infrastructure there?
Actually, we already started with the CDN. And since it is in the cloud, load is no longer a problem, right? Wrong. Hosting all or part of an infrastructure in the cloud does not magically remove the need to consider the impact of heavy load, even for a service like a CDN!
We will assume that Azure Blob Storage will not lose our files while the signup site is online. However, Azure’s capacity to answer requests has limits. Globally, there is plenty of capacity :). But for an individual account or file, we need to examine the scalability targets. Limits exist at several levels. For example, one storage account can support “only” 5,000 downloads per second. Given our target of one million per second and three blobs per page — one JavaScript file, one CSS file, and an image used as a CSS sprite — one Azure storage account can support only 1,666 users per second, and we would need 600 storage accounts to handle the load. Each Azure account is limited to five storage accounts, a limit that can be increased by contacting support. This is a rough estimate: caching means not every client requests every file on every page load, or in the same second.
These numbers apply to storage accounts, not to the CDN. CDNs sit between users and storage accounts, caching Blob Storage data locally for a configurable duration. Their scalability target will therefore be different. We can estimate it from the number of CDN locations — determining whether data is fetched once per edge location or once per server within it — the number of files, and cache duration.
I have not found capacity metrics for the CDN, but they are clearly higher. Our visitors’ geography is also important. A CDN is especially valuable when users are spread around the world, and less so when they are all in one country. However, if the Paris node becomes overloaded, some traffic is automatically redistributed to other locations, such as Zurich, London, or Amsterdam.
Fly me to the cloud
We have done a good part of the work. Now we need to address a central component: the signup website itself. For this article, we will choose ASP.NET MVC. We need to build the site to withstand the load, knowing it will be hosted in Windows Azure. We develop it quickly and test it on our development machine.
Once deployed to Azure, a simple configuration-file change can expand the infrastructure hosting it. We have two levers:
- Scale up: increase the power of each server hosting the site — processors, RAM, and so on. In Windows Azure, this means increasing the instance’s “size.”
- Scale out: increase the number of servers hosting the site, or the number of Azure instances.
We might think our job is finished. We have considered DNS, reduced traffic volume, hosted static files on a CDN, and provided powerful servers to answer requests.
Unfortunately, that is not enough. In the next installment, we will see that even for a “simple” site like this, merely hosting it in Azure does not immediately make it capable of absorbing such a load.
The next installment will look at an architecture using a set of Windows Azure services to support this kind of high-traffic website. Stay tuned.