GovCMS Case Study: managing increased website demand during COVID-19

The exceptional circumstances faced in 2020 have placed an unprecedented additional load on any websites that provide information on factors relating to, reporting on or assisting with the COVID-19 responses across the world.

As the hosting provider for GovCMS, the Australian Government whole-of-government Drupal hosting platform, this has led to a large load on a number of websites hosted on the GovCMS platform, and on the amazee.io, Salsa Digital, GovCMS and individual agency web teams.

Whilst amazee.io designed Lagoon to be fully scalable, the volume of COVID-19 related traffic experienced in March/April 2020 was an order of magnitude larger than anticipated. A number of factors drove customers to the sites hosted on GovCMS – from increased messaging and awareness, regular public briefings to a general increased need for information about crucial Government programs.

During this period, amazee.io and Salsa Digital worked together with GovCMS to implement a number of significant enhancements to the Lagoon platform and underlying AWS infrastructure. We introduced additional in-memory caching to alleviate database bottlenecks and implemented a new set of parameters to rapidly scale web infrastructure to handle massive influxes of traffic. In order to maximise performance for users, our team worked extensively with GovCMS to fine-tune caching and offload performance. We analysed our infrastructure performance data and made adjustments to some of our tuning parameters, set better defaults, and created more time-critical alarms on key infrastructure metrics.

Critical to the ongoing success of the platform we also adopted a much closer partnership model with the key agencies hosting their high-profile sites on GovCMS to ensure that we worked in close collaboration to anticipate their needs, meet their demands, and maintain their confidence in our platform.

Working as a fully-remote team already, amazee.io has had an advantage heading into this disruptive period, but now that online collaboration is becoming more prevalent across the globe, being able to stand up and participate in shared communication channels has given us unprecedented access to our customers and clients (and them to us!). This has meant that turnaround times can be minimised, decisions can be made with input from the correct people, and keeping broader teams appraised of rapidly changing situations is much easier!

During the April/May 2020 two-month period:

• Traffic was up to 168TB/month, highest single day was 8.4TB
• Hits were up to 1.96 billion hits/month, highest single day was 103 million
• Peak concurrent users on just one single site hit 157,000
• Peak pageviews in a single hour was 478,000
• Peak pageviews in a single minute was 89,000
• We maintained a 99.9% SLA for the high-profile sites we were managing
  

When we built and architected the Lagoon solution for GovCMS, we fully accounted for fault tolerance and autoscaling needs, ensuring that GovCMS (and their customers) could not only serve the current and predicted sites, but manage spikes in traffic without needing to overprovision capacity. Throughout the last few months, we’ve only needed to make minor modifications to the underlying infrastructure (mostly in the pursuit of extreme performance) used to serve these sites, which is a welcome change from the expensive hardware oversize and upsize paradigm of traditional hosting.

Taking our learning from this experience and implementing them into our default configurations will ensure that we are adequately positioned for any future high-traffic events.

GovCMS reference case studies:
https://www.govcms.gov.au/news-events/news/case-study-managing-website-forefront-pandemic

https://www.govcms.gov.au/news-events/news/case-study-services-australia-frontline-during-covid-19