How to Write Smoke Tests for an Ember Rails Stack

Development

Reading Time: 10 minutes

The following story shows the importance of smoke tests when writing an app using the Ember+Rails stack. It covers the most basic example, and even then things go awry. In addition, it shows an example of an end-to-end test and some of the difficulty in setting up an environment for smoke tests. Finally, it mentions exhaust, a library that can ease this pain.

Imagine, you’re ready to start writing your next great application. It’s like TodoMVC meets Twitter meets eBay: Users will write down their To Do lists and auction off tasks to other users in 140 characters or less. Investors are very excited. You’re very excited. High-fives all around. You’re going to change the world, and you’re going to do that using all the right technologies: Ember for the client side and Rails for the backend!

Here is your first user story:

cucumber
Given I am a user
When I go to the todos page
Then I should see a list of all todos

Getting Started with Ember Rails

Piece of cake! You start by installing the latest Ember:

> npm install -g ember bower 
> ember new todo-frontend 
> cd todo-frontend

“I don’t care what @dhh says,” you think out loud. “I’m a rockstar developer, so I’m gonna test-drive it like I stole it!” You don’t have an API to test yet, so you start by mocking it. You install pretender and generate your first acceptance test:

> ember install ember-cli-pretender 
> ember generate acceptance-test todos
// tests/acceptance/todos.js 
import Ember from 'ember'; 
import { module, test } from 'qunit'; 
import startApp from 'emberbook/tests/helpers/start-app'; 
import Pretender from 'pretender';

let TODOS = [ 
  {id: 1, title: "write a blog post"}, 
  {id: 2, title: "let people read it"}, 
  {id: 3, title: "... profit"} 
];

module('Acceptance | todos', { 
  beforeEach: function() { 
    this.application = startApp();

    this.server = new Pretender(function() {
      this.get('/todos', function(){
        var json = {
          todos: TODOS
        };
    
        return [200, {}, JSON.stringify(json)];
      });
    });
  },

  afterEach: function() { 
    Ember.run(this.application, 'destroy'); 
    this.server.shutdown(); 
  } 
});

test('visiting /todos', function(assert) { 
  visit('/todos');

  andThen(function() { 
    var title = find('h1'); 
    assert.equal(title.text(), 'Todo List');

    var todos = find('.todo-item');
    assert.equal(todos.length, TODOS.length);
    
    assert.equal(currentURL(), '/todos');
  }); 
});

Time to fire up the test runner and watch it all burn:

ember test --serve

Yep, it’s red like the devil. First error, no route.

// app/router.js 
import Ember from 'ember'; 
import config from './config/environment';

var Router = Ember.Router.extend({ 
  location: config.locationType 
});

Router.map(function() { 
  this.route("todos"); 
});

export default Router;

Now the test can’t find the text “Todo List.” You fix this as well:

{{!- app/templates/todos.hbs }}

# Todo List

{{#each model as |todo|}}
  <p class='todo-item'>
    {{todo.title}}
  </p>
{{/each}}

Now the test breaks because there are no To Do items on the page. You move forward:

// app/routes/todos.js 
import Ember from 'ember';

export default Ember.Route.extend({ 
  model() { 
    return this.store.findAll("todo"); 
  } 
});

Can’t find the “todo” model. One more fix:

// app/models/todo.js 
import DS from 'ember-data';

export default DS.Model.extend({ 
  title: DS.attr(), 
  complete: DS.attr("boolean") 
});

Whoa, it’s green! You pat yourself on the back. But you can’t actually deliver the story until there’s an API to back it up. Time to grab the latest version of Rails.

Testing an Ember Rails Application

This is going to be an API rather than a full-fledged application, so you only need a few gems. You spend several minutes pondering a post you just read about building Rails API apps and deciding whether to use the Rails 5 --api flag. However, you want this product to hit the market “yesterday,” so you just roll with what’s stable:

> gem install rails 
> rails new todo_api --skip-test-unit --skip-bundle 
> cd todo_api

You also skipped test-unit because you’d rather use Rspec’s eloquent DSL for this project’s tests.

Time to boil the Gemfile down to the few gems you really need. This API only needs to serve JSON, so out with anything related to the asset pipeline:

# Gemfile
source 'https://rubygems.org'

gem 'rails', '4.2.3' 
gem 'sqlite3' 
gem 'active_model_serializers'

group :development, :test do 
  gem 'rspec-rails' 
end

Bundle it and get ready to write some specs!

> bundle install 
> rails generate rspec:install

Time to bang out the first request spec:

# spec/requests/todos_spec.rb
require "rails_helper"

RSpec.describe "Todos", :type => :request do

  # letBANG because `before {expected_todos}` looks funny 
  let!(:expected_todos) do 
    3.times.map do |i| 
      Todo.create(title: "Todo #{i+1}") 
    end 
  end

  it "lists all todos" do 
    get "/todos" 
    todos = extract_key_from_response("todos", response)   
    expect(todos.count).to eq expected_todos.count 
  end

  def extract_key_from_response(key, response) 
    json_response = JSON.parse(response.body) 
    expect(json_response).to have_key key 
    json_response[key] 
  end

end

You run the test. Watch it go red! First gripe: no model. You fix that:

class Todo < ActiveRecord::Base
end

It’s migration time!

<code class="language-bash rails generate migration CreateTodos title:string</code>
# db/migrate/#{timestamp}\_create\_todos.rb
class CreateTodos < ActiveRecord::Migration 
  def change 
    create_table :todos do |t| 
      t.string :title 
      t.timestamps null: false 
    end 
  end 
end
> rake db:migrate

And of course, it needs a route.

# config/routes.rb
Rails.application.routes.draw do 
  resources :todos, only: :index 
end

The test is still red, because there is no controller. Next fix:

# controllers/todos_controller.rb
class TodosController < ApplicationController 
  def index 
    render json: todos 
  end

  private

  def todos 
    Todo.all 
  end 
end

All green! Time to boot up the app(s) and bask in your own creative genius. You run the following commands in different terminals:

> rails server 
> ember server

You can’t wait to see your baby running. You open “http://local:4200/todos”… and you get a blank screen. Now that was disappointing. Both test suites are green. What could have gone wrong?

After some inspection into the problem, you notice a GET http://localhost:4200/todos 404 (Not Found) in your logs. Well, crap. Ember doesn’t know the API is on a different domain. Well, that’s an easy fix. You just need to specify a host in the application adapter. And since you are attractive and intelligent, you know better than to hardcode the host. After all, it will change for staging and production. So you open config/environment.js in your Ember app, and you add the following:

// config/environment.js                                                                                             if (environment === 'test') {
  ENV.API_HOST = ''
} else {
  ENV.API_HOST = (process.env.API_HOST || 'http://localhost:3000')
}

You set the test environment’s API_HOST to an empty string, so that the current tests keep working. All other environments use the Rails’ default http://localhost:3000 unless an API_HOST environment variable is set. Now you can create an application adapter and fix the bug:

// app/adapters/application.js 
import DS from 'ember-data'; 
import config from '../config/environment';

export default DS.RESTAdapter.extend({ 
  host: config.API_HOST 
});

W00T! Fixed that problem. Time to run the tests. Once again, you start the Rails and Ember apps in two different terminals.

> rails server 
> ember server

Dang it. The app still doesn’t work. You didn’t set up CORS so that modern browsers can talk to the Rails API. You add rack-cors to the Gemfile, bundle it, and add the simplest policy you can think of to get things working. Allow everything!

# Gemfile
gem 'rack-cors', :require => 'rack/cors'
bundle install
# config/application.rb
config.middleware.insert_before 0, "Rack::Cors", :debug => true, :logger => (-> { Rails.logger }) do 
  allow do 
    origins '*' 
    resource '*', 
      :headers => :any, 
      :methods => [:get, :post, :delete, :put, :options, :head], 
      :max_age => 0 
  end 
end

You cross your fingers, run the tests, and start the servers. When you visit the todos route, you see your app is working. Phew! Even though your app is working now, you aren’t 100 percent happy. You spent the last hour test driving an application that didn’t work, even though all tests were green. And you didn’t even focus unit tests — all of your tests were integration tests. Shouldn’t integration tests prevent things like this?

Sign up for a free Codeship Account

Writing Smoke Tests for an Ember Rails App

At this point, you remember a post on the Hashrocket blog about test driving elixir apps with cucumber. You decide to try these techniques in practice and write some smoke tests. The type of integration tests that were just written are great for testing application(s) in isolation. These isolated integration tests run very fast. And because they run fast, the majority of your tests should be either request specs or Ember integration. There is no reason to have full coverage (testing every possibility) with smoke tests.

However, in order to ensure the app actually works, there should be at least one smoke test for every API endpoint. Knowing that cucumber-rails already loads your test environment, you think of the simplest solution you can to try writing a smoke test. Simply add cucumber-rails, manually start Ember and Rails, then overwrite how capybara works. This is the solution you come up with. You add cucumber to the Gemfile and install it.

# Gemfile
group :development, :test do 
  gem 'cucumber-rails', require: false 
  gem 'database_cleaner' 
  gem 'capybara-webkit' # ... 
end
> bundle install 
> rails generate cucumber:install

Then you overwrite capybara’s behavior.

# features/support/env.rb
Capybara.configure do |config| 
  # Don't start rails 
  config.run_server = false 
  # Set capybara's driver. Use your own favorite 
  config.default_driver = :webkit 
  # Make all requests to the Ember app 
  config.app_host = 'http://localhost:4200' 
end

And you write the first feature and step definitions.

# features/todo.feature
Feature: Todos 
  When a user visits "/todos", they should see all todos

  Scenario: User views todos 
    Given 4 todos 
    When I visit "/todos" 
    Then I should see "Todo List" 
    And I see 4 todos
# features/steps/todo.rb
Given(/^(\d+) todos$/) do |num| 
  num = num.to_i 
  num.times do |i| 
    Todo.create(title: "Title #{i}") 
  end 
end

When(/^I visit "(.*?)"$/) do |path| 
  visit path 
end

Then(/^I should see "(.*?)"$/) do |text| 
  expect(page).to have_text(text) 
end

Then(/^I see (\d+) todos$/) do |num| 
  expect(all(".todo-item").length).to eq num.to_i 
end

Now in separate terminals you run both Rails and Ember. This time Rails is started in the test environment so that Ember requests hit the Rails test database.

> rails server --environment test 
> ember server

You run the test suite with rake, and voilà! You see that the test suite passes. You know that both Ember and Rails are communicating end-to-end. And even though you are super proud of yourself, you’re not happy. “The servers should start automatically. Why can’t I just type rake?” you think to yourself. Here is the first solution you come up with.

# features/support/env.rb
ember_pid = fork do 
  puts "Starting Ember App" 
  Dir.chdir("/your/user/directory/todo-frontend") do 
    exec({"API_HOST" => "http://localhost:3001"}, "ember server --port 4201 --live-reload false") 
  end 
end 

rails_pid = fork do 
  puts "Starting Rails App" 
  Dir.chdir(Rails.root) do 
    exec("rails server --port 3001 --environment test") 
  end 
end 

sleep 2

at_exit do #kill the Ember and Rails apps 
  puts("Shutting down Ember and Rails App") 
  Process.kill "KILL", rails_pid 
  Process.kill "KILL", ember_pid 
end

Capybara.configure do |config| 
  config.run_server = false 
  config.default_driver = :webkit 
  config.app_host = 'http://localhost:4201' 
end

This solution works okay. Rails and Ember run on separate ports, so the development versions of the two servers can keep running. The only problem is the sleep 2. As the Rails and Ember apps grow, so will the time needed to wait for the servers to start. Also, this number is magic; it might take longer on another machine. How long will it take on the CI server?

What you really want to do is halt the tests until you know that Rails and Ember are running. However, after some investigation, you realize there is no way to know if Ember is running successfully. But then you notice how EmberCLI smoke tests itself.

  it('ember new foo, server, SIGINT clears tmp/', function() {
    return runCommand(path.join('.', 'node_modules', 'ember-cli', 'bin', 'ember'), 'server', '--port=54323','--live-reload=false', {
        onOutput: function(string, child) {
          if (string.match(/Build successful/)) {
            killCliProcess(child);
          }
        }
      })
      .catch(function() {
        // just eat the rejection as we are testing what happens
      });
  });

Now you know that Ember has booted when it outputs “Build successful” and have some insight as to how you might wait for Rails.

# features/support/env.rb
begin 
  DatabaseCleaner.strategy = :truncation 
rescue NameError 
  raise "You need to add database_cleaner to your Gemfile (in the :test group) if you wish to use it." 
end

ember_server = nil 
rails_server = nil

Dir.chdir("/Users/mwoods/hashrocket/todo-frontend") do 
  ember_server = IO.popen([{"API_HOST" => "http://localhost:3001"}, "ember", "server", "--port", "4201", "--live-reload", "false", :err => [:child, :out]]) 
end

Dir.chdir(Rails.root) do 
  rails_server = IO.popen(['rails', 'server', '--port', '3001', '--environment', 'test', :err => [:child, :out]]) 
end

# Before timeout loop to prevent orphaning processes
at_exit do 
  Process.kill 9, rails_server.pid, ember_server.pid 
end

# if it takes longer than 30 seconds to boot throw an error
Timeout::timeout(30) do 
  # wait for the magic words from ember  
  while running = ember_server.gets 
    if running =~ /build successful/i 
      break 
    end 
  end

  # when rails starts logging, it's running 
  while running = rails_server.gets 
    if running =~ /info/i 
      break 
    end 
  end 
end

Capybara.configure do |config| 
  config.run_server = false 
  config.default_driver = :webkit 
  config.app_host = 'http://localhost:4201' 
end

This is not a perfect solution, but it’s good enough for day one. On day two, you might decide to move the smoke tests to their own repository. After all, they aren’t Rails-specific, so they probably should be a separate project.

Problems like these and many others have happened to me. That’s the reason I’m introducing exhaust, a new gem that hopefully alleviates some of the headache associated with smoke testing an Ember+Rails stack. Enjoy!

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles.
Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.



We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

  • Hi Micah, thanks for your article. Is there a reason you don’t use the proxy-argument when using ember serve? Then you don’t need to add the CORS things…

    • Micah Woods

      Great question!

      Answer: I like smoke tests to exercise as much of the application stack as possible.

      Other dev’s at Hashrocket like the `–proxy` approach. But personally, I like to know that my application adapters `host` was set correctly. What if I mistype `config.APP_Host`? If I use `–proxy` I won’t catch this until staging (or worse production). Same goes for CORS, `–proxy` will run green even if CORS was never added.

      Now, if you are using https://elements.heroku.com/buildpacks/tonycoco/heroku-buildpack-ember-cli or https://github.com/ember-cli/ember-cli-deploy, `–proxy` might be just fine.

      If you make an issue for `–proxy` on the https://github.com/mwoods79/exhaust repo, I will implement. I’ve already started a refactor for strategies.

  • Pavlo Osadchyi

    Hey. Thanks for the article. I have a question. How do you actually ensure in your tests that Router finished the transition and your test makes an assertions against the expected page’s state? In ember integration tests we have `andThen` helper (which, IIRC, checks whether there are no pending ajax calls etc and it’s `run loop` aware). I’m asking because when I started doing Ember, I put Ember App inside of Rails app via gem `ember-rails`, and I tried to write tests via cucumber and capybara. As a result I ended up with putting `sleep` here and there, and the whole test sweet took approx an hour to pass – it was all because cucumber and capybara (with selenium driver for js) simply couldn’t know, when the page is ready for assertions. Why you may think it won’t happen to your setup? Thanks.

    • Micah Woods

      Good question(s). Let’s see if I can address all your problems.

      Checking transitions is better handled in Ember acceptance tests. I keep cucumbers to a minmum because they are slow. I usually only write one smoke test per api endpoint, just to know that the application works.

      The ember-cli-rails gem started here at Hashrocket after Pavel and I had a conversation. He was much more ambitious and wanted a Rails and Ember hybrid. Because of this it does some tricks. When your test suite starts, it builds the ember app, copies it to the asset pipeline, then Rails serves the compiled ember app. Because of the build process there is a ember-cli plugin that writes a “lock” file to disk and rails waits for that file to be removed before any rails request are served. This makes your tests start slower. On large apps it can be painful.

      Exhaust on the other hand starts Ember and Rails in parellel halting cucumber until they start. This is a much faster boot process and more straight forward.

      Just last week, I worked to fix a cucumber suite that ran over an hour. It is a backbone.js application. By removing sleeps, and disabling jquery animations the test suite runs in just over 18 minutes.

      1. Never sleep.

      a. Wait for something on the screen. For example, if you know that when you click a button you will get a notication, wait for that instead. `expect(page).to have_content(notification_text)`

      b. Wait for ajax to finish. If you click a button, and immediatly go to another page, you can wait for ajax to finish first. Read this blog post here. https://robots.thoughtbot.com/automatically-wait-for-ajax-with-capybara

      c. Increase `Capybara.default_max_wait_time = wait_longer`, but only if you absolutly have to. This is a last resort.

      2. Disable jQuery animations when testing. `jQuery.fx.off = true`. You will be shocked how much time this shaves off.

      Using these strategies should keep your cucumber test suite fast.

      • Pavlo Osadchyi

        All of this makes sense, thank you. That project I mentioned about was my first one, about 2 years ago. I just recalled that I had those issues with sleeps in my tests; after that project I completely moved to writing acceptance tests with `ember-qunit`; and having a separate test suite for API – I was just curious how you address those issues in your setup. Thank you very much for such an extensive answer.

  • Jack Lucky

    This article was really helpful. I’ve been trying to solve this problem for a while.

    However, I had some issues (following the article’s instructions and also using the exhaust gem). To get `And I see 4 todos` to pass, I had to tag the Cucumber scenario with `@javascript` and I had to change the line `config.default_driver = :webkit` to `config.javascript_driver = :webkit`.