http://calpaterson.com/latency.html Cal Paterson | Home About Where's the fastest place to put my server? How much does it matter? February 2021 Using my own web server accesslogs and public latency data to get a quantitative answer and why roundtrips are such a pain. ... As network latencies grow, strange things can happen: "fat" sites can become fast (especially if served completely from CDN) and "thin" sites that use APIs can become slow. A typical latency for a desktop/ laptop user is 200ms, for a 4G mobile user, 300-400ms. I've assumed 40 megabit bandwidth, TLS, latency to CDN of 40ms and no existing connections. "Origin" here means the primary webserver (as opposed to "edge" CDN caches). What's the fastest place to put my server? Beyond the time taken for servers to respond to requests it takes time just to traverse the internet, just to get a packet from A to B. To estimate what the theoretical best physical place to put my own server is I've combined publicly publicly available data on latencies with my own web server accesslogs. I'm aiming to get a rough, quantitative, answer that's based on a real data set. Why location matters Time taken to traverse the internet is added to the time taken to respond to a request. Even if your API can respond to a request in 1ms, if the user is in London and your API server is in California the user still has to wait ~130 milliseconds for the response. It's a bit worse than just 130 milliseconds. Depending on what a user is doing they may end up making a number of those roundtrips. To download a web page usually requires five full roundtrips: one to resolve the domain name via DNS, one to establish the TCP connection, two more to set up an encrypted session with TLS and one, finally, for the page you wanted in the first place. Subsequent requests can (but don't always) reuse the DNS, TCP and TLS setup but a new roundtrip is still needed each time the server is consulted, for example for an API call or a new page. 130ms sounded fast at first, but the rigmarole of just getting a page and then making an couple of API calls can easily end up taking most of a second just in terms of time waiting for the network. All the other time required: for the server to decide what response to send to your request, time downloading the thing and then rendering whatever it is in your browser - that is all extra. The two kinds of "fast" for networks One of the confusing things about networking is the inspecific way in which people talk of getting "faster" networking: "faster" residental broadband for example, or "fast ethernet" (100 megabits per second, no longer impressive). This kind of "faster" is not in fact talking about speed. Greater speed would be reduced latency - so faster roundtrips. Instead "faster" networking is really about greater bandwidth: more bytes per second. APIs or CDNs One thing that does make things faster: a Content Distribution Network (or CDN). Instead of going all the way to California perhaps you can retrieve some of the web page from a cache in central London. Doing this saves time - perhaps taking just 50 milliseconds, a saving of 60%. Caches work great for CSS files, images and javascript - stuff that doesn't change for each user. It doesn't work as well for the responses to API calls, for which the responses are different for each user, and sometimes, each time. A quantitative approach A happy few can serve everything from their CDN. News sites, for example, show the exact same thing to everyone. Others are less lucky and can make only limited, or no, use of caching. These poor people have to pick a location for their main server to help them get their bytes to the users who want them as fast as possible. If they want to make that choice with the sole aim of reducing latency, where should they pick? Here's what I did: 1. I took my own accesslogs for a two week period in September just after I'd published something new. I got about a million requests during this period from 143k unique IPs. I excluded obvious robots (which was ~10% of requests). 2. I used Maxmind's GeoIP database to geocode each IP address in those accesslogs to geographic co-ordinates. 3. I then used WonderNetwork's published latency data for internet latencies between ~240 world cities. 4. I mapped those cities (semi-manually, which was pretty painful) from their names to Geonames ids - which gave me co-ordinates for the cities. 5. Then I loaded all of the above into a Postgres database with the PostGIS extension installed so I could do geographical queries. 6. I queried to estimate how long, by percentile, requests would have taken if I'd had my server in each of the 200 cities. The results In the table below I've recorded the outcome: how long users would take to complete a single roundtrip to my server if it were in each city. I've done this by percentiles so you have: * the average ("p50") * for three quarters of requests ("p75") * and for 99% of requests ("p99") All numbers are in milliseconds. See full results as a table (click to expand) I've included a bit of Javascript in this page, so you can click on the headings to sort. City p50 p75 p99 Manhattan 74 97 238 Detroit 89 115 245 Secaucus 71 96 246 Piscataway 75 98 251 Washington 82 105 253 Chicago 90 121 253 Kansas City 98 130 254 Indianapolis 96 125 254 St Louis 96 127 256 Cincinnati 92 121 257 Houston 104 134 257 Syracuse 77 102 257 Scranton 78 103 258 Quebec City 83 113 259 South Bend 92 118 259 Montreal 83 104 259 Charlotte 91 110 259 Salem 74 98 259 Buffalo 80 111 259 Albany 75 100 260 Monticello 94 123 260 Baltimore 80 105 260 Asheville 95 118 260 New York 77 103 261 Berkeley Springs 84 112 261 Minneapolis 102 133 261 Barcelona 102 148 261 Dallas 112 140 262 Des Moines 104 131 262 San Jose 139 165 263 Brunswick 77 101 264 Atlanta 88 113 264 San Francisco 136 168 264 Halifax 80 102 265 Philadelphia 77 100 266 Basel 97 146 267 Green Bay 103 131 267 Pittsburgh 88 117 267 Bern 99 147 267 Denver 112 141 267 Miami 103 129 267 Raleigh 88 111 268 Knoxville 114 135 268 Boston 77 105 268 Valencia 108 148 268 Jackson 105 132 268 Memphis 101 131 268 Jacksonville 95 122 268 Madrid 95 138 268 London 76 130 268 San Diego 138 162 269 San Antonio 112 138 269 Salt Lake City 120 151 269 Toronto 87 111 269 Cleveland 97 122 269 Austin 113 141 270 Colorado Springs 110 136 270 Orlando 103 126 270 Antwerp 93 137 271 Oklahoma City 114 147 271 Saskatoon 115 140 272 Lansing 98 127 272 Seattle 141 164 272 Columbus 92 120 273 Bristol 76 129 274 Tampa 104 130 274 Lausanne 95 139 274 Ottawa 85 111 274 Falkenstein 91 137 275 Maidstone 76 129 275 Paris 80 129 275 Toledo 102 129 275 Savannah 117 146 276 The Hague 82 138 276 Liege 87 136 277 Lincoln 100 124 277 New Orleans 115 142 278 Amsterdam 82 140 278 Las Vegas 136 163 279 Vienna 102 149 279 Coventry 80 132 279 Cromwell 80 106 280 Arezzo 109 160 280 Cheltenham 79 131 280 Sacramento 137 167 280 Alblasserdam 82 137 281 Vancouver 142 165 281 Fremont 131 157 283 Gosport 76 137 284 Frankfurt 93 136 284 Carlow 88 136 285 Phoenix 128 153 285 Portland 132 159 285 Cardiff 78 131 285 Luxembourg 87 137 285 Bruges 83 135 285 Eindhoven 85 133 285 Groningen 87 139 286 Manchester 80 137 286 Brussels 90 139 287 Brno 106 148 287 Edinburgh 84 136 287 Nuremberg 89 136 288 Albuquerque 125 159 289 Los Angeles 141 164 289 Ljubljana 110 152 289 Lugano 97 147 290 Zurich 103 146 290 Dronten 84 133 290 Newcastle 87 147 290 Rome 96 147 291 Dusseldorf 90 140 291 Munich 98 144 291 Venice 106 156 292 Edmonton 139 165 292 Copenhagen 96 145 292 St Petersburg 113 163 293 Dublin 85 143 293 Redding 142 178 293 Vilnius 110 162 293 Belfast 79 125 294 Nis 113 158 294 Douglas 87 143 294 Rotterdam 82 139 295 Bergen 107 157 295 Strasbourg 89 141 295 Roseburg 148 172 296 Graz 104 147 296 San Juan 117 141 298 Warsaw 108 161 299 Frosinone 105 153 299 Riyadh 159 206 300 Prague 103 152 301 Ktis 102 158 302 Mexico 139 164 302 Belgrade 113 160 302 Guadalajara 128 155 303 Milan 96 146 305 Bratislava 102 154 306 Osaka 181 240 307 Zagreb 103 150 308 Tallinn 108 162 308 Helsinki 105 156 308 Hamburg 127 166 309 Oslo 98 153 311 Bucharest 120 162 311 Riga 113 159 312 Panama 150 177 313 Tokyo 188 238 313 Kiev 119 168 313 Stockholm 102 153 314 Budapest 110 162 314 Kharkiv 128 169 315 Gothenburg 115 167 316 Pristina 122 167 316 Tirana 128 184 316 Geneva 96 142 316 Siauliai 113 163 317 Cairo 133 182 318 Sapporo 196 255 318 Bogota 170 188 319 Palermo 119 183 320 Gdansk 107 152 320 Caracas 149 176 320 Sofia 114 161 321 Westpoort 79 134 321 Honolulu 173 196 321 Roubaix 102 157 321 Kazan 138 190 322 Winnipeg 169 190 322 Varna 120 173 322 Tel Aviv 138 194 322 Lisbon 115 166 324 Jerusalem 145 198 324 Ankara 139 195 327 Heredia 164 188 327 Athens 128 183 329 Reykjavik 127 180 329 Paramaribo 166 194 330 Algiers 120 173 332 Chisinau 127 180 333 Bursa 135 188 334 Thessaloniki 134 187 336 Limassol 141 186 337 Lyon 95 145 340 Mumbai 204 248 340 Medellin 163 186 344 Valletta 120 176 345 Baku 160 205 346 Melbourne 227 269 346 Fez 149 198 348 Tunis 124 180 348 Koto 217 254 348 Dubai 192 243 350 Tbilisi 153 208 351 Malaysia 195 235 352 Hyderabad 214 260 354 Bangalore 212 252 355 Izmir 137 187 357 Adelaide 241 272 359 Chennai 221 248 359 Moscow 127 172 359 Lahore 217 270 361 Novosibirsk 163 206 362 Sydney 237 272 363 Karaganda 180 231 363 Vladivostok 223 264 364 Taipei 265 293 364 Lima 169 199 364 Istanbul 135 182 366 Hong Kong 199 223 366 Auckland 244 291 367 Jakarta 207 245 368 Seoul 231 277 371 Beirut 136 195 372 Accra 168 216 373 Singapore 190 246 374 Sao Paulo 193 213 375 Joao Pessoa 182 220 378 Perth 243 267 379 Ho Chi Minh City 253 287 380 Wellington 251 295 383 Brasilia 226 249 384 Manila 251 281 385 Pune 202 251 386 Dhaka 231 268 386 Phnom Penh 243 267 386 Santiago 202 230 390 Lagos 191 233 391 Quito 162 188 392 New Delhi 230 264 395 Johannesburg 237 283 398 Bangkok 222 254 401 Canberra 262 295 402 Dar es Salaam 214 267 407 Dagupan 239 268 408 Christchurch 257 309 409 Hanoi 235 264 415 Cape Town 216 262 417 Buenos Aires 232 253 417 Guatemala 217 249 418 Brisbane 261 288 422 Indore 304 352 457 Zhangjiakou 236 264 457 Nairobi 233 277 468 Kampala 244 287 480 Hangzhou 239 267 517 Shenzhen 242 275 523 Shanghai 300 367 551 Montevideo 738 775 902 You can also download the full results as a csv, if that's easier. The result: east coast of North America good, right on the Atlantic better The best places are all in North America, which is probably not a total surprise given that it's a pretty dense cluster of English speakers with another cluster not all that far away (in latency terms) in the UK/ROI and then a lot of English-as-a-second-language speakers in Europe. Being right on the Atlantic is best of all: New Jersey and New York state have many of the best places for p99 and it doesn't vary too much, at the top, between p50 and p99. If you're wondering why small New Jersey towns like Secaucus and Piscataway are so well connected - they have big data centres used by America's financial sector. As it stands, my server is currently in Helsinki. That's because, unusually for Finland, it was the cheapest option. I only pay about three quid a month for this server. If I moved it to somewhere in New Jersey, and spent more, users would definitely save time in aggregate: half of roundtrips would be completed in 75ms rather than 105ms, a saving of 30%. Over several roundtrips that would probably mount up to around a sixth of a second off the average of first-time page loads, which is not too bad. In case you can't tell, this website isn't hugely taxing for web browsers to render so cuts in the network wait time would make it considerably quicker. Since I don't dynamically generate anything on this site, the truth is that I'd be best off with a CDN. That would really save a lot of time for everyone: it's nearly twice as good to be served from a CDN (~40ms) than to be in the fastest place (71ms). How this might change over time Latencies aren't fixed and they might improve over time. Here's a table of roundtrip latencies from London to other world cities with more than 5 million people, comparing against the theoretical maximum speed, the speed of light: City name Distance (km) Real latency Theoretical Slowdown max factor New York 5,585 71 37 1.9 Lima 10,160 162 68 2.4 Jakarta 11,719 194 78 2.5 Cairo 3,513 60 23 2.6 St Petersburg 2,105 38 14 2.7 Bangalore 8,041 144 54 2.7 Bogota 8,500 160 57 2.8 Buenos Aires 11,103 220 74 3.0 Lagos 5,006 99 33 3.0 Moscow 2,508 51 17 3.0 Sao Paulo 9,473 193 63 3.1 Bangkok 9,543 213 64 3.3 Hong Kong 9,644 221 64 3.4 Istanbul 2,504 60 17 3.6 Lahore 6,298 151 42 3.6 Tokyo 9,582 239 64 3.7 Hangzhou 9,237 232 62 3.8 Shanghai 9,217 241 61 3.9 Mumbai 7,200 190 48 4.0 Taipei 9,800 268 65 4.1 Dhaka 8,017 229 53 4.3 Seoul 8,880 269 59 4.5 (Please note, a correction: the above table previously compared real roundtrips with theoretical straight line journeys - this has now been corrected, for more details see these two comments for discussion and more details - like how part of this is due to the nature of fibre optic cables and submarine cable curvature.) As you can see, New York's latency is within a factor of 2 of the speed of light but routes to other places like Dhaka and Seoul are much slower: being 4 times the speed of light. There are probably understandable reasons why the London to New York route has been so well optimised though I doubt it hurts that it's mostly ocean between them, so that undersea cables can run directly. Getting to Seoul or Dhaka will be a more circuitous route. I should probably mention that new protocols promise to reduce the number of round trips. TLS 1.3 can create an encrypted session with one round trip rather than two and HTTP3 can club together the HTTP round trip with the TLS one, meaning you now only need three: one for DNS, one single roundtrip for both a connecton and an encrypted session and then finally a third for the subject of your request. One false hope some people seem to have is that new protocols like HTTP3 do away with the need for Javascript/CSS bundling. That is based on a misunderstanding: while HTTP/3 will remove some initial roundtrips it does not remove subsequent roundtrips for extra Javascript or CSS. So bundling is sadly here to stay. Data weaknesses While I think this is an interesting exercise - and hopefully indicative - I should be honest and say that the quality of the data I'm using is solidly in the "medium-to-poor" category. Firstly, the GeoIP database's ability to predict the location of an IP address is mixed. Stated (ie: probably optimistic) accuracy ranges up to about 1000 kilometers in some cases, though for my dataset it thinks the average accuracy is 132km with a standard deviation of 276km - so not that accurate but I think still useful. My source of latency data, WonderNetwork, are really reporting point-in-time latency from when I got it (30th November 2020) as opposed to long term data. Sometimes the internet does go on the fritz in certain places. WonderNetwork have a lot of stations but their coverage isn't perfect. In the West it's excellent - in the UK even secondary towns (like Coventry) are represented. Their coverage worldwide is still good but more mixed. They don't have a lot of locations in Africa or South America and some of the latencies in South East Asia seem odd: Hong Kong and Shenzhen are 140ms away from each other when they're only 50km apart - that's a slowdown factor compared to the speed of light of more than a thousand times. Other mainland China pings are also strangely bad, though not on that scale. Perhaps the communists are inspecting each ICMP packet by hand? The other problem with the latency data is that I don't have the true co-ordinates for the datacentres that the servers are in - I had to geocode that myself with some scripting and a lot of manual data entry in Excel (I've published that sheet on github to save anyone from having to redo it). I've tried hard to check these but there still might be mistakes. By far the biggest weakness, though, is that I'm assuming that everyone is starting right from the centre of their closest city. This isn't true in practice and bias this adds can vary. Here in the UK, residental internet access is a total hack based on sending high frequency signals over copper telephone lines. My own latency to other hosts in London is about 9ms - which sounds bad for such a short distance but is still 31ms better than average. Many consumer level routers are not very good and add a lot of latency. The notorious bufferbloat problem is also a common source of latency, particularly affecting things that need a consistent latency level to work well - like videoconferencing and multiplayer computer games. Using a mobile phone network doesn't help either. 4G networks add circa 100ms of lag in good conditions but of course are much worse when the signal is poor and there are a lot of link-level retransmissions. I did try assuming the global average latency per kilometer (about 0.03ms) to compensate for distance from the closest city but I found this just added a bunch of noise to my results as for many IPs in my dataset this is an unrealistic detour: the closest city I have for them isn't that close at all. Generality It's fair to wonder to what extent my results would change for a different site. It's hard to say but I suspect that the results would be approximately the same for other sites which are in English and don't have any special geographical component to them. This is because I reckon that people reading this blog are probably pretty uniformly distributed over the English speaking population of the world. If I was writing in Russian or Italian the geographic base of readers would be pretty different and so the relative merits of different cities from a latency point of view would change. It wasn't too hard for me to run this test and I've released all the little bits of code I wrote (mostly data loading and querying snippets) so you could easily rerun this on your own accesslogs without too much effort. Please write to me if you do that, I'd love to know what results you get. Gratuitous roundtrips Picking a good spot for your server only goes so far. Even in good cases you will still have nearly a hundred milliseconds of latency for each roundtrip. As I said above there can be as many as five roundtrips when you visit a page. Having any unnecessary roundtrips will really slow things down. A single extra roundtrip would negate a fair chunk of the gains from putting your server in a fast place. It's easy to add roundtrips accidentally. A particularly surprising source of roundtrips are cross-origin (CORS) preflight requests. For security reasons to do with preventing cross-site scripting attacks, browsers will "check" certain HTTP requests made from Javascript. This is done by sending a request to the same url beforehand with the special OPTIONS verb. The response to this will decide whether the original request is allowed or not. The rules for when exactly preflighting is done are complicated but a surprising number of requests are caught up in the net: notably including JSON POSTs to subdomains (such as api.foo.com when you're on foo.com) and third party webfonts. CORS preflighting checks use a different set of caching headers to the rest of HTTP caching which are rarely set correctly and anyway are only applicable for subsequent requests. A lot of sites these days are written as "single page apps", where you load some static bundle of Javascript (hopefully from a CDN) and which then makes a (hopefully low) number of API requests inside your browser to decide what to show on the page. The hope is that this is faster after the first request as you don't have to redraw the whole screen when a user asks for a second page load. Usually, it doesn't end up helping much because a single HTML page tends to get replaced with multiple chained API calls. A couple of chained API calls to an origin server is almost always slower than redrawing the whole screen - particularly over a mobile network. I always think it's a bit rubbish when I get a loading bar on a web page - you already sent me a page, why didn't you just send the page I wanted! One of the great ironies of the web is that while Google don't do a good job of crawling these single page apps they certainly produce a lot of them. The "search console" (the website formerly known as "webmaster tools") is particularly diabolical. I suppose Google don't need to worry overly about SEO. Bandwidth improves quickly but latency improves slowly Internet bandwidth just gets better and better. You can shove a lot more bytes down the line per second than you could even a few years ago. Latency improvements, however, are pretty rare and as we get closer to the speed of light the improvement will drop off completely. 100 megawhats per second is less compelling when you still have to wait the same half a second for each page to load. Contact/etc Please do feel free to send me an email about this article, especially if you disagreed with it. If you liked it, you might like other things I've written. You can get notified when I write something new by email alert or by RSS feed. If you have enjoyed this article and as a result are feeling charitable towards me, please test out my site project, Quarchive, a FOSS social bookmarking style site, and email me your feedback! See also Last year APNIC analysed CDN performance across the world, and concluded that 40ms is typical. I wish they'd included percentile data in this post but I can still get the vague impression that CDNs perform best in the West and less well in South American, China and Africa which is a problem given that most servers are based in the West. While I was writing this post a number there was an outbreak of page-weight-based "clubs", like the "1MB club" and the, presumably more elite, 512K Club. I suppose I approve of the sentiment (and it's all in the name of fun I'm sure) I think they're over-emphasising the size of the stuff being transferred. If you're in London, asking for a dynamically generated page from California, it will still take a most of a second (130ms times 5 round trips) regardless of how big the thing is. The submarine cable map is always fun to look at. If you want to see a sign of the varying importance of different places: the Channel Islands (population 170 thousand) have 8 submarine cables, including two that simply connect Guernsey and Jersey. Madagascar (population 26 million) has just four. I also think it's funny that even though Alaska and Russia are pretty close there isn't a single cable between them. If you want to reproduce my results I've published my code and data on Github. I'm afraid that does not include my accesslogs which I can't make public for privacy reasons. Please don't expect me to have produced a repeatable build process for you: that takes a lot more time and effort so it's provided on a "some assembly required" basis. :)