Pregenerating Static Web Pages for Better Performance

Development

In my recent Tuning NGINX article, I talked about how it’s important to tune based on the specific needs of an application and its environment. In today’s article, we’re going to put that in practice. Last time, we tuned our environment by adjusting parameters within NGINX. Now, we’re going to explore a sometimes-overlooked aspect of tuning: making adjustments to how our application works.

The application we will be tuning is a generic WordPress site that is built on a LEMP (Linux + NGINX + MySQL + PHP) stack. I selected WordPress because by default it dynamically generates every page, whether it’s the front page of a blog that may not change often or specific articles that may receive comments daily.

For each request made to a WordPress site, that site is served by NGINX, a PHP application, and MySQL. This means that every request is serviced by three different service layers.

Pregenerating Semi-dynamic Pages

In order to increase our web application’s performance, we’ll be pregenerating the results of our “dynamic” pages and saving those results into a file cache. We will then configure NGINX to use the file cache for HTTP requests rather than our dynamic web application.

The end result will be that every HTTP request will be serviced by only one service layer, which will have quite an impact on performance.

Establishing a Baseline

Before making any changes, the first thing we should do is establish a baseline metric for application performance. In the previous article, I used ApacheBench to measure the number of requests per second that NGINX could service. We can use this same metric for our testing today as well.

Let’s see what happens if we run ab against an existing WordPress installation I have already set up.

$ ab -c 40 -n 1000 http://example.com/
Concurrency Level:      40
Time taken for tests:   141.269 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      64731000 bytes
HTML transferred:       64483000 bytes
Requests per second:    7.08 [#/sec] (mean)
Time per request:       5650.762 [ms] (mean)
Time per request:       141.269 [ms] (mean, across all concurrent requests)

Based on the output above, it appears that our baseline is 7.08 requests per second. At the moment, this web application is a vanilla WordPress installation without any performance plugins or tuning of the LEMP stack.

Let’s see how much of an improvement we’ll get by pregenerating our front page.

Creating a Cache

Even within WordPress, there are several ways to generate a file cache of dynamic content. For this article, I will be taking a simple, noninvasive approach. We’ll simply request the page we wish to cache and save the results into a file.

Create a store for our cache

Since we’ll be creating a file-based cache, let’s go ahead and create a directory within our application’s root directory to retain cached files.

# mkdir /var/www/example.com/htdocs/cached

Within the htdocs directory of our WordPress installation, we created a cached directory. Within this directory, we are going to create our cache files. To generate those files, we will be using the curl command.

# curl -H 'host: example.com' http://192.168.33.10 > /var/www/example.com/htdocs/cached/index.html

The curl command is a very powerful utility for working with web applications from the command line. Chances are, if you’ve worked from the command line of a Linux/Unix system, you have used curl at some point.

In the example above, the curl command will be performing an HTTP GET request to http://192.168.33.10. This address is the IP address of our example WordPress installation. In addition to the HTTP address in the command above, we are also using the -H flag to set the host header to a value of example.com. This header is used by NGINX to determine how our request is routed and which content to display. By setting this to example.com, we are ensuring that we are routed to our example site.

At the end of the command above, you can see a > used to redirect the output of the curl command to /var/www/example.com/htdocs/cached/index.html. This redirect will write the output of our HTTP request to the index.html file.

The output of our HTTP request is of course, the HTML generated when curl requested our WordPress site’s front page.

Whether it’s with curl or a web browser such as Chrome, the contents of the index page for this WordPress site are always the same. At least, until a new article is posted. By saving the generated HTML to a file, we can tell NGINX to serve requests from our cached HTML file rather than sending the request to the PHP application on the backend.

# chown -R www-data:www-data /var/www/example.com/htdocs/cached

Before we move on to configuring NGINX, we first need to reset the ownership of our newly created cache file. So far, the commands we have run have been executed as the root user. Which means that the file and directory we created are owned by the root user. By resetting the owner to www-data, we are ensuring that NGINX can read our cached file.

Configure NGINX

Now that we have our cached file created, we need to tell NGINX to serve that cached file instead of the WordPress application. To do this, we’ll be editing the /etc/nginx/sites-enabled/example.com file. This file is the example.com specific configuration file that defines how the example.com site is served by NGINX.

Let’s take a look at the current contents.

server {

    server_name example.com   www.example.com;

    access_log /var/log/nginx/example.com.access.log rt_cache;
    error_log /var/log/nginx/example.com.error.log;

    root /var/www/example.com/htdocs;

    index index.php index.html index.htm;

    include common/php.conf;
    include common/wpcommon.conf;
    include common/locations.conf;
    include /var/www/example.com/conf/nginx/*.conf;
}

The above is fairly standard for a site running WordPress. To leverage our file cache, we will need to add to add a few items to the above configuration.

location ~ ^/$ {
  try_files /cached/index.html /index.php;
}

While at first our additions may look a bit complex, they’re actually pretty simple once you understand what they’re doing.

The first part, location, is an NGINX directive that identifies the location of the HTTP request. If that request matches the ^/$ regular expression, NGINX will apply the directives within the curly brackets. The ^/$ regular expression will match any HTTP request that is targeting a location that begins and ends with /. Essentially this is the address for the WordPress front page (http://example.com/).

The second part, try_files, is an NGINX directive that tells NGINX to look for the specified file (/cached/index.html) and return the contents of that file rather than its normal processing. If for whatever reason it doesn’t find that file, NGINX will route the request to the defined URI (/index.php).

The fact that NGINX will redirect requests to /index.php if no cache file is found is very useful. If for some reason NGINX cannot find our cached file, the WordPress application would still serve content to visitors. Now that our changes have been added, let’s take a look at the entire configuration again.

server {

    server_name example.com   www.example.com;

    access_log /var/log/nginx/example.com.access.log rt_cache;
    error_log /var/log/nginx/example.com.error.log;

    root /var/www/example.com/htdocs;

    location ~ ^/$ {
      try_files /cached/index.html /index.php;
    }

    index index.php index.html index.htm;

    include common/php.conf;
    include common/wpcommon.conf;
    include common/locations.conf;
    include /var/www/example.com/conf/nginx/*.conf;
}

The location of these lines are somewhat arbitrary, however they should be added after the root definition and before the includes; the configurations being included in this example may have some processing rules that would override our directive.

With our changes made, we can apply them by issuing a reload of NGINX. We can do this with the service command.

# service nginx reload

With the reload complete, our configuration changes have now taken effect.

Measuring the Results

With our modifications made, let’s see how our performance has changed by rerunning the same test using the ab command.

$ ab -c 40 -n 1000 http://example.com/
Concurrency Level:      40
Time taken for tests:   1.524 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      64783000 bytes
HTML transferred:       64499000 bytes
Requests per second:    656.10 [#/sec] (mean)
Time per request:       60.966 [ms] (mean)
Time per request:       1.524 [ms] (mean, across all concurrent requests)

In the baseline test, our site was able to service 7.08 requests per second (mean). With the changes above, our site is now able to service 656.10 requests per second (mean). That is a performance increase of over 9000%, and the only thing we did was pregenerate the content of our front page.

Understanding the results

The resulting improvement may seem quite high; to better understand why the performance increased so much, let’s break down how our HTTP requests were serviced before and how they are serviced now.

With the vanilla WordPress installation, every HTTP request to / would be forwarded from NGINX to the php-fpm the application service used to serve PHP applications. Specifically, WordPress in this case. The WordPress application itself would then make several queries to the MySQL database. This transaction flow would occur for each and every HTTP request.

With the above configuration changes added, now NGINX will first search for the /cached/index.html file when an HTTP request is made to /. If it finds that file, it opens that file and returns the contents to the HTTP client.

By having NGINX serve the content from cache, we save time and system resources with each request by eliminating the need to call php-fpm and MySQL. We are also playing to one of NGINX’s strengths, as we saw in the previous NGINX tuning article; the NGINX service is very efficient at serving static content.

In Conclusion

In this article, we set up our web server (NGINX) to look for and use a cached HTML file to answer HTTP requests. This resulted in a huge performance increase in our web application’s response time. This improvement in performance does however come with some negatives.

One of the negatives is that we have now removed the dynamic nature of our front page. For this site, it’s okay because our front page is rarely updated. However when updates do happen, they will not automatically be seen by our visitors, which leads us to another negative. In order to ensure that visitors are seeing new content as soon as it’s published, we must update the cache every time new content is published.

With WordPress, this process is made easy thanks to caching plugins such as WP Total Cache and WP Super Cache. For non-WordPress web applications, this may require some customization, whether that customization is external to the web application like our curl example or an internal function.

The key to using this methodology is to understand just how dynamic each page needs to be and cache pages that can be cached, while selectively not caching pages that require dynamic content. I personally have found that after implementing a file-based cache within my web applications, I will often change the way I design my web applications to use caching more efficiently.

One way I update design for cache efficiency is to try to use client-side dynamic content rather than server-side. An example of this is using the Disqus comments plugin rather than WordPress’ native comment system. This allows me to serve dynamic content while also leveraging the speed of file caches.

Have a tip that makes it easier to create and manage caches? Share it by adding a comment to this article.

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles.
Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.



We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

  • Great article! I didn’t realise Nginx could be used for this. I have 2 questions:

    1. How can the cache be regenerated on a WP site without using those caching plugins?

    2. How can this process be rolled out to all pages on a site without manually doing it for each page?

    Thanks!

    • madflojo

      Hey @vatultd:disqus ,

      1. You could extend the curl into a shell script or rather write something that crawls your site and pregenerates pages based on what makes sense for your site.

      2. If it was me I’d write something pretty quick to do the above and use regular expressions to include/exclude pages.

  • Robert Rawlins

    This is great. I’ve been a fan of static site generators like Jekyll and Octopress for a long while now, and syncing the sites up to something like AWS S3 is another way of completely removing the weight from your servers.

    Doing this without the help of a generator/cms/framework can be a challenge though, especially if you’re dealing with larger amounts of content, and you need to update the template on a few hundred HTML files or something.

  • Actually, this is pretty easy to achieve with WP Super Cache. Set it to generate URLs using mod_rewrite (although Apache does not exist, it will skip the “index.php/” in generated URLs) and in Nginx’s location / {…} block, update the “try files…” line to:


    location / {
    try_files /wp-content/cache/supercache/$http_host/$request_uri/index.html $uri $uri/ /index.php?$args;
    }

    The entire block is more optimal too, as it does not need a regex in the location definition and it also covers static files, not just dynamic files. BTW, the folder structure you see above is just the default WP Super Cache folder structure.

  • Bart

    Wouldn’t it be more optimal to use a HTTP based caching solution (like Varnish or a CDN)? If you have the cache headers setup correctly it should work with dynamic pages.

    • madflojo

      It’s all about preference and what works for your situation. One of the reasons I did this over Varnish in my environment is that Varnish doesn’t handle TLS. So you have to put something like NGINX in front of Varnish. So it made more sense to me to just use the file cache.

  • Tilemahos Manolatos

    Awesome. You can run the curl from a cronjob – say per minute or per hour. Then the backend server will not be accessed but once in the cronjob execution interval. So yes, this simple approach removes much load from the backend server. The tradeoff is between the frequency of cronjob execution and the frequency of 1st page content change.

  • Emil Vikström

    Or you can configure Nginx to use microcaching. Set up a FastCGI cache that only caches pages for 5-10 seconds. That way you get all the benefits of dynamic pages while handling peaks with ease. We have our microcache on a RAM disk to speed it up further.

    We have a pretty cheap VPS. Because we run Nginx < 1.11.5 we can't use the max_conns setting: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns
    This means that I can't even run a pre-cache ApacheBench test with your settings. I had to decrease my concurrency limit to 5 and only run 100 requests, otherwise I got 503 errors. Anyway, on to the results!

    Without microcaching. 5 concurrent requests, 100 requests total:

    Concurrency Level: 5
    Time taken for tests: 3.992 seconds
    Complete requests: 100
    Failed requests: 0
    Non-2xx responses: 100
    Total transferred: 286100 bytes
    HTML transferred: 242900 bytes
    Requests per second: 25.05 [#/sec] (mean)
    Time per request: 199.598 [ms] (mean)
    Time per request: 39.920 [ms] (mean, across all concurrent requests)

    With microcaching. 40 concurrent requests, 1000 requests total:

    Concurrency Level: 40
    Time taken for tests: 0.097 seconds
    Complete requests: 1000
    Failed requests: 0
    Total transferred: 72045000 bytes
    HTML transferred: 71718000 bytes
    Requests per second: 10304.18 [#/sec] (mean)
    Time per request: 3.882 [ms] (mean)
    Time per request: 0.097 [ms] (mean, across all concurrent requests)

    Let's boost this up a notch! With microcaching. 1000 concurrent requests, 100000 requests in total:

    Concurrency Level: 1000
    Time taken for tests: 14.791 seconds
    Complete requests: 100000
    Failed requests: 63275
    (Connect: 0, Receive: 0, Length: 63275, Exceptions: 0)
    Total transferred: 7206189350 bytes
    HTML transferred: 7173489350 bytes
    Requests per second: 6760.86 [#/sec] (mean)
    Time per request: 147.910 [ms] (mean)
    Time per request: 0.148 [ms] (mean, across all concurrent requests)

    This also applies to all pages, not just the start page!

  • Pingback: The Week That Was – Week Ending 22nd Oct 16 | Practical Performance Analyst()