Beating the Black Hats in time: Mitigating Drupal SA-CORE-2018–002 on an infrastructure level.

Security is one of our highest priorities. This can be seen in many different places within amazee.io including: authenticating developers with public/private keys instead of passwords, conducting penetration tests of our platforms together with security researchers, and using OpenShift which creates a virtual network for each Kubernetes Namespace.

Some issues can only be mitigated by one factor: speed. When a new security patch comes out, everyone gets the information at the same time — people with both good and with bad intentions. It is crucial for those of us wanting to protect the websites to fix the code as fast as possible. This is especially true for parts of our infrastructure like application code that we don’t have direct control over, such as the Drupal code.

Yesterday’s Drupal security release showed a highly critical security patch that needs to be implemented on every Drupal site in the world as quickly as possible. Even with fully automated continuous integration systems this would not be an easy task for our clients. As a hosting company we have a key advantage: we can roll out a piece of code that applies to allapplications hosted on our platform and by doing so protect all Drupal sites at once without the need to touch each of them individually.

In the spirit of open source we would like to share how accomplished this and the key role that OpenShift/Kubernetes plays in this solution.

The short version is:

We adapted the request sanitizer from the Drupal code base so it can run fully independent of Drupal Code and created a separated PHP file from it: https://github.com/amazeeio/SA-CORE-2018-002/blob/master/prepend.php

We created a bash script (https://github.com/amazeeio/SA-CORE-2018-002/blob/master/apply.sh) that checks each project inside OpenShift to see if it has an nginx pod with a PHP container, if it finds one:

• It creates a new ConfigMap based on the request sanitizer PHP file in each project.

• It adds this ConfigMap to each PHP container as a volume.

• It tells PHP to auto_prepend this request sanitizer PHP file file via an environment variable

• The bash script also checks if the request sanitizer file is correctly included and informs if not.

We then run the script against all projects in OpenShift which automatically get updated. A rolling deployment within OpenShift makes sure that there is no downtime of any sites. Plus as the PHP fix code is injected on an infrastructure level it persists even through future deployments.

All requests that trigger the mitigation code are logged by the internal ELK Stack, allowing us to monitor and report any exploitation attempts. This can all be done in real-time which allow us to immediately react to any potential issues that might develop after the patch has been reverse engineered.

We believe in Open Source and the higher security this brings. We will continue to open source as much as we can!