Add Post-Deploy Smoke Tests to Any Codeship Pipeline


Reading Time: 5 minutes

One of the biggest goals of a continuous development workflow is to find and fix bugs earlier in the development cycle. Automatically smoke testing every deployment helps developers quickly validate new changes and catch bugs before they make it to production.

Codeship’s deployment pipelines and Assertible’s web app testing tools make it easy to add an extra layer of automated testing after every build.

In this post, I’ll walk through setting up automated post-deployment smoke tests in a Codeship pipeline using Assertible. I’ve put together an example Ruby app that is deployed to staging and production environments on Heroku and validated with Assertible.

Continuous Delivery with Codeship

Codeship is a continuous integration and delivery service that allows you to build, test, and deploy code after every commit. Codeship makes it easy to get started by removing the need for yaml configurations and complex scripts. This, combined with the Assertible deployments API, gives developers an easy way to set up test automation for any web app.

Let’s start by outlining the steps of the pipeline:

  • Push code and build it with Codeship
  • Deploy the application to Heroku
  • Smoke test the app with Assertible

The primary goal of this process is to create a completely automated build, deployment, and smoke-testing pipeline that will catch application errors during development by testing the app on a live staging environment.

Connect a repository

The first step is to connect Codeship to a repository on GitHub. To do this, create a new project in your Codeship account and enter the Git clone URL of your repo:

Now every time code is pushed to the connected repository, the changes will be pulled and built by Codeship. Learn more about connecting repos in the docs.

Create a deployment pipeline

Now that the repo is connected to Codeship, the next step is to deploy the application. For deploying to Heroku, Codeship’s deployment pipelines are a perfect fit.

A deployment pipeline is specific to a branch in your repository, like master. Multiple pipelines can be set up to handle deployments from different branches; for example, the Ruby app is deployed to production from the master branch and to staging from any wip-* branch. The deployment pipeline is initiated only after all of the build and test steps have completed.

The example Ruby application is set up with the following pipelines:

  1. For commits on the master branch, deploy the app to production.

  2. For builds on a branch that start with wip-*, deploy the app to a staging environment.

    In Codeship, this is called Wildcard Branch Matching and is extremely useful for handling deployment of the same app to multiple environments.

Testing a web app on staging environments is an effective first line of defense to find bugs early in the development cycle, before they make it to production. The process outlined above provides automatic deployments to an isolated staging environment where the release can quickly be checked and validated.

Learn more about Managing Multiple Environments for an App in the Heroku docs.

!Sign up for a free Codeship Account

Smoke Testing with Assertible

With a deployment pipeline in place, you can set up Assertible to smoke test the application immediately after it’s deployed. This approach has several benefits:

  • Quickly validate potential releases
  • Catch bugs earlier in the development cycle
  • Tighten the feedback loop between QA and developers
  • Prevent wasting time on testing broken applications

The Assertible Deployments API can be integrated with the deployment pipelines to run an automated test suite against both the staging and production environments.

Define and test a web service

The first step is to define your web service in Assertible. Log in to your Assertible Dashboard and enter the URL of your web service or import your Swagger Spec. A test will automatically be created for the URL, and you can add more tests for other critical endpoints.

For the Ruby API example, I’ve created a test for each endpoint that will validate the core functionality of the app and fail if anything is critically wrong.

Ideally, the smoke tests are fast and check core behavior to identify a faulty deployment. The assertions made against the Ruby API include:

If any of these assertions fail, you will be immediately notified of a bad release.

Run tests after a deployment

Now that the web service is defined, we can go back to the Codeship deployment pipelines and add a second step to run the smoke tests. The Assertible deployments API makes this possible by tracking releases of an application and initiating tests against new deployments.

Add a Custom Script step to the deployment pipeline that makes a POST request to Assertible:

This step is called after the first step has successfully deployed the application to Heroku. You can use a simple cURL command to send the details of the release to Assertible, which will then initiate the tests for that service.

Environment variables

To execute the custom script, you need to set a few environment variables in your Codeship environment. They can be retrieved from the Assertible dashboard:

  • CI_COMMIT_ID (provided by Codeship)

Testing deployments on staging

Assertible supports the concept of environments for a web service. An environment is a separate URL where the same application may be deployed, like a staging or testing environment.

The example Ruby API has two environments:

The example application is deployed to both staging and production. Tests can be run against a specific environment by specifying the environmentName in the POST body.

When Assertible receives deployment events, it will look up the environment and run tests against that URL. Learn more about POSTing deployments to Assertible and how environments work here.

A Complete Deployment Pipeline

Everything is now set up for Codeship to build the code, deploy to Heroku, and run smoke tests against the application. As code is pushed to the repository, you will hopefully see passing tests in Assertible and a green build in Codeship:

Bonus: Status Badges

Once your continuous delivery and post-deployment testing pipeline is set up, you’ll want to show off how all your builds and tests are passing.

Codeship and Assertible both offer status badges you can use in a README, documentation, or anywhere else to let your team and users know the status of your web services.

Additional Resources

There are many great resources to learn more about setting up and extending this pipeline. Check out some of the articles below:

Do you have any questions, comments, or corrections? Hit me up on Twitter. I’d love to hear about your deployment and testing workflow. To get started building and testing your code, sign up for a free Assertible account and Codeship account.

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles.
Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.

We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.