Audience Finder, UK, is the free national audience data and development tool that enables cultural organisations to understand, compare and apply audience insight. Audience Finder brings together information on all UK households with data from over 1000 cultural organisations: over 243 million tickets, 86 million transactions, approx. 410,000 surveys and web analytics from all the UK's major arts and cultural organisations. Currently it handles data for about 1000 arts organisations from across the UK, ranging from small, local theatres to large multi-venue organisations. Computing resources are used for two requirements - on the one hand the data caching process to gather the needed data, on the other hand the query of data by users that are creating high peaks for the server.

The demand for the service is increasing daily, though most users exclusively use Audience Finder in a certain period of the week, e.g. from Monday to Friday between 9am and 4pm. This causes peaks in CPU use and more CPU is consequently deployed to provide a seamless service. The data caching process is outsourced to a certain point in time, where the demand by users is low - e.g. around midnight - and raises a need for a large instance, due to the requirements of the caching data process. The given requirements need to be met to ensure a seamless usage of the application.

Technical Insights

Audience Finder is based on PHP and was deployed on a highly customized WordPress software platform and hosted on a large AWS EC2 instance to meet the requirements of the Audience Finder application. The platform enables the application to respond to high peaks without losing service quality. Beside high peaks, the instance is barely used and the under-utilised resources generate sunk costs.

MiCADO vs. Traditional Hosting

MiCADO

Scalable to meet demand

Scaling policies to use spare resources

Good use of resources

Simple deployment

Multi-cloud support

Traditional Hosting

Not scalable

No Scaling policies

Poor use of resources

Complicated deployment

No multi-cloud support

MiCADO reduces sunk costs

In order to shift from Capital Expenditures to Operational Expenditures as well as to minimise sunk costs, there is the need for an automatic scalable resource framework that is sensitive to the demand of the application.

MiCADO enables instances to up- and down-scale resources and additional worker instances (slaves) to address the needs in a flexible cost-efficient way. By supporting horizontal scaling, the minimum server size for the Audience Finder application can be reduced so demand can be met by scaling out, rather than paying for underutilized resources. Additionally, by allowing to use spare resources during off-peak hours the caching process can be run without impacting website usage.

Dockerising WordPress

WordPress is an established CMS whose initial release was in 2003. In that time the way that build websites has changed, and Docker is built to work using these modern best-practices:

Configuration in environment.
In WordPress configuration is in the file, but in modern apps configuration is better set in the environment, allowing the same app code to be deployed to different environments and configured for each.

Session stored in filesystem.
This is a PHP consideration, as the default location for sessions (i.e storing whether a visitor has logged into the site) happens at the filesystem. This makes it impossible to scale a PHP app horizontally as a user’s session data would cannot be communicated between them.

To dockerise Audience Finder, there is the need to dockerise WordPress.
In order to change how WordPress was configured Bedrock is used, which is an alternative installation of WordPress that allows the use of Environment variables to configure the application. To change how PHP stores sessions, so that a dockerised WordPress app could be horizontally scaled, Redis was configured to act as storage for PHP’s session files.

Dockerising Audience Finder

Previous Docker environment

Nginx and PHP-FPM in same container - both processes running at the same time

Database and other services outside of docker

New design for docker environment

Nginx and PHP-FPM in separate containers (one process per container) - communication between containers over port

database and other services outside the docker environment

Benefits

Containers can be scaled separately to meet demand. With proper caching in place, there is the need to increase the number of NGiNX containers to respond to requests, but not for a corresponding number of PHP containers

Docker images are easier to manage - official docker images can be used for the PHP-FPM container and no need to install NGiNX simplifying maintenance

Easier to debug a broken container as it is only running one service and logs - separated for the two services (PHP and NGiNX)

Ansible is used to deploy MICADO to an AWS EC2 t2.small instance - the ansible scripts are available at GitHub  - it creates the necessary AWS resources for our project so that new MICADO instances can be build quickly as possible.

 

"Our purpose is to enable cultural organisations to use our national data to increase their relevance, reach and resilience. We believe arts and culture have the power to improve the quality of people’s lives, to create meaning and community, to promote empathy and learning. The more citizens are involved in shaping culture and taking part, the greater the power. Our mission is to give the public a voice in shaping a vibrant, relevant culture."

The software developers behind the curtains: Outlandish

We work on technology that makes the world a fairer, better place: from web apps, data dashboards and monitoring tools that aid the discovery of new insights from complex data, to mass-impact campaigns and websites that promote your causes and mobilise your key audiences.

We place users at the heart of our projects, and use agile methodologies to focus on delivering iterative, releasable products with our clients.

We are committed to our vision: we’re a worker co-operative, we’re owned by our members, and the profits we generate are invested in prototyping and developing new projects for social change. We’re also co-founders of CoTech, the network of UK technology co-operatives that is increasing the depth and breadth of services available to clients striving for positive change.