How Space City Weather weathered Hurricane Laura

Howdy, folks—my name is Lee, and I’m the SCW server admin. I don’t post often (or really ever!), but with Eric and Matt off for the day to recover from their marathon forecasting job, I wanted to take the opportunity to talk to y’all a bit about how Space City Weather works, and how the site deals with the deluge of traffic that we get during significant weather events. This isn’t a forecasting type of post—I’m just an old angry IT guy, and I leave weather to the experts!—but a ton of folks have asked about the topic in feedback and in comments, so if you’re curious about what makes SCW tick, this post is for you.

On the other hand, if the idea of reading a post on servers sounds boring, then fear not—SCW will be back to regular forecasts on Monday!

(I’m going to keep this high-level and accessible, so if there are any hard-core geeks reading here who are jonesing for a deep-dive on how SCW is hosted, please see my Ars Technica article on the subject from a couple of years ago. The SCW hosting setup is still more or less identical to what it was when I wrote that piece just after Hurricane Harvey.)

How much traffic just blew through?

So, fun stuff first. On a normal day, Space City Weather does something like 10,000-20,000 page views to something like 5,000-10,000 visitors. This traffic is generated from folks who specifically visit spacecityweather.com, so those numbers don’t include people who just read the SCW e-mail or just peep at posts on the SCW Facebook page. (“Page views” are a good metric for how much work your server has to do, with each “page view” representing one page of content that the SCW server has to send to you—a page like this article that you’re reading right now, for example. “Visitors,” on the other hand, is the absolute number of people who stopped by. Some visitors look at more than one article, so you’re typically going to have more page views than visitors.)

Our peak traffic day with Hurricane Laura was Tuesday, August 25, when things looked like they might go very badly for Houston. On that day, we clocked a smidge under 1.6 million views to about 520,000 visitors:

That’s about 160x our usual daily traffic load, and even higher than the 1.1 million views we hit during the peak of Hurricane Harvey two years ago. And, unlike Harvey’s traffic, we ended up with not one but a pair of million-view days—August 25’s 1.6 million views were followed with another 1.3 million on August 26.

To put that load in terms of pages-per-second, SCW was serving up between 20-50 pages every second for hours at a time on both Tuesday and Wednesday. And the site did it without breaking a sweat.

How? Well, there’s the trick—we cheat a bit.

The stack

But first, before I explain how we cheat, I’ll talk just for a moment about the hardware and software that power the site. Space City Weather is hosted on a dedicated physical server, rather than on any kind of shared hosting. That means we lease an actual physical computer from a hosting company named Liquid Web; the server lives up in Liquid Web’s Michigan data center.

As Eric noted on Twitter the other day, SCW’s web server is named “Garak,” after the Cardassian tinker/tailor/soldier/spy from Star Trek: Deep Space Nine. Garak is an 8-core Xeon with 16GB of RAM and fast dual solid state disks, and he’s a workhorse.

Garak runs the application stack that makes Space City Weather work. Broadly, that stack includes WordPress (the actual blog application SCW runs—you’re reading a WordPress-generated page right now, and I’m writing this post in WordPress). Garak also runs Nginx (pronounced “engine-ex”), the actual web server application that sends you pages and pictures. There’s also a database to hold WordPress posts and configuration data, a heavy duty caching layer (using Varnish), and an SSL termination layer (using HAProxy) for keeping things secure while still taking advantage of cache.

Why do we use physical hosting (which is admittedly more expensive) rather than cloud-based hosting at AWS or another cloud provider? The answer is simple: because physical hosting is easy. I don’t have to know or care how to have multiple cloud servers of varying capacities standing by to come online in the event of a traffic storm—we just always have the big gun (that is, the physical server) in place. Liquid Web’s rates are reasonable and it’s a cost-effective solution that trades time for money, and I’m always going to spend money to gain time if that’s an option.

The cheat

But here’s the thing—like I said, we kind of cheat at big traffic events a bit, and the way we cheat is by using Cloudflare.

In the simplest terms, Cloudflare is a service that absorbs your network load for you. Space City Weather is the kind of site that lends itself very well to caching—that is, most of Space City Weather’s pages don’t change very often, so you can take advantage of that and distribute copies of those un-changing pages out to Cloudflare’s army of servers. When the traffic storm arrives, Cloudflare’s monster servers send the unchanging parts of the site to visitors, and Garak only has to be bothered when something does change—when Eric or Matt create or update a post, for example.

Spreading the load out to a paid service like Cloudflare is shockingly effective. How effective? This chart from Space City Weather’s Cloudflare dashboard tells the tale in a single picture:

Over the last week, Space City Weather served a bit under 12 terabytes of data to visitors. That is a stupidly ridiculously large amount of traffic. But out of that monster chunk of data, Garak only had to serve up about 110 gigabytes. A terabyte is 1000 gigabytes (or 1024 gigabytes, depending on what you’re measuring and who you’re talking to, but I’m trying to keep things simple), so Cloudflare handled about 99% of the Hurricane Laura traffic. This left Garak with only about 1% of the total work.

Considering that Garak’s monthly bandwidth allocation from Liquid Web is only 5 terabytes and every gigabyte over that comes with a penalty fee, we’d be looking at expensive overage charges without Cloudflare in place to handle the load. (And that’s not even addressing the fact that we’d probably need to spread things out between more than one physical server!)

Cloudflare more or less makes Space City Weather possible.

Funding, and the future

Fortunately, Cloudflare is also very kind to us on pricing. Because our monster traffic events are so rare and so peak-y (as opposed to constant and sustained), we get by on Cloudflare’s lowest-cost paid plan. That, along with all of our hosting and infrastructure costs, are handled via Reliant’s ongoing yearly sponsorship of Space City Weather. This enables us to keep the site very lean and focused on speed above all else, without intrusive ads, autoplaying garbage video, pop-ups, or any of that other annoying crap other sites have to festoon themselves with to earn money.

We’re nothing without our audience, and SCW readers are the best. As a life-long Houstonian born and raised, it brings me a lot of pride to see SCW thriving; this site is definitely a home-grown labor of love. And we hear your requests to help out, too—for new readers who perhaps just found SCW because of Hurricane Laura, we do have a fundraiser where folks can purchase SCW t-shirts, umbrellas, and a few other items. If you’re looking to score some sweet SCW merch, fundraiser time is the time to do it—that will be in November.

Thanks for taking the time to read this technical update, if you’ve made it this far, and thanks for stopping by the site. Eric and Matt will be back to forecasting on Monday, and we hope you’ll stick around!