bintz-integration_testing_setup/README.md

# Bintz's integration testing setup

_I'm writing this for my coworkers but putting it here in GitHub, because heck, why not? All of the
related projects are already here. Here's how I use them. YMMV. Do what you want, I don't care._

So for most purposes I only do integration testing using Cucumber and Capybara
now. I used to unit test very very heavily. I don't anymore. The reasons for this are:

* Integration testing focuses on the most important part of the application -
  how the end user interacts with it. If it doesn't work the way
  the user expects it to, no amount of code coverage or component testing will
  help it.
* My years of unit testing every possible thing in an application have honed my
  code smell detection abilities pretty well, so I tend to
  write smaller methods anyway, breaking code out to new classes when necessary.
* I made the feedback loop for integration testing with Cucumber and Capybara a
  *lot* smaller. Much of the process is automated and optimized for
  speed, at the cost of the occasional bit of code duplication in tests.
* A lot of the things that Rails wants you to test are silly. Did an instance
  variable get set in a controller action? Can a url get routed to a particular
  controller action? For routes testing, unless it's a complex route, it's usually just
  simpler to load the url in a browser and make sure you get what you want to
  get. For other things, a single integration test will catch everything that
  the other little isolated tests will catch, and will probably require a lot
  less code.

I still advocate full automated testing for applications, run as often as possible (for me,
before every commit). I just save unit
testing for individual pieces of the app where there are a ton of possible
inputs (like some sort of filter mechanism), and I try to get 90% or more of what
the app is going to do in user's hands written into Cucumber features,
saving that last ~10% to be caught during actual user testing and when the app is
in the wild.

The goal should be to replace as much of the development environment tweak-reload cycle that pretty
much all apps will have with something that is just as fast and infinitely repeatable.

## Silly code metrics!

I'm not a huge fan of code metrics-as-gospel, but they sure do help when you run them every once in a while to
find dead/untested/complex code. As an example of how much coverage you can get with as few well-designed tests as possible,
and with the discipline to only ever write enough code to get the test to pass, I ran SimpleCov on two apps built using
this method, without doing any cleanup work after this first metrics run:

<table border="1" cellpadding="2">
  <tr><th>Relevant LoC</th><th>Coverage</th><th>Executed Step Count</th><th>Run Time (:poltergeist driver)</th></tr>
  <tr><td>1453</td><td>94.22%</td><td>891</td><td>2m29.286s</td></tr>
  <tr><td>602</td><td>99.0%</td><td>72</td><td>0m20.228s</td></tr>
</table>

Not a lot of work needed to get these apps closer to 100%, just finding and eliminating some dead code and testing
a missed branch or two, probably inserted during moments of lacking discipline when I just hacked on a
project in `development` mode. I'm not perfect. :P

## Setting up your project

To get your project set up fast, this repo is also a gem that provides a binary that puts everything in how
I normally do things:

``` ruby
# Gemfile

gem 'bintz-integration_testing_setup', :git => 'git://github.com/johnbintz/bintz-integration_testing_setup.git'
```

`bundle install`, then run `bundle exec bintz-integration_testing_setup`. It's best to do this on a pristine
project, but it may work on one that already has some stuff in it.

## Pure Ruby project?

Want to use this setup with something that's not Rails? Pass the `--pure-ruby` flag to `bintz-integration_testing_setup`.

## Gem setup

You'll need several gems to pull this off, some of them custom gems:

### For front-end development

These gems make it very easy for Capybara to target parts of the web page to
interact with.

``` ruby
gem 'semantic_rails_view_helpers', :git => 'git://github.com/johnbintz/semantic_rails_view_helpers.git'
```

### For testing

These gems set up the Cucumber environment and get it ready for super-fast integration testing.

``` ruby
group :cucumber do
  gem 'cucumber'
  gem 'cucumber-rails'
  gem 'cuke-pack', :git => 'git://github.com/johnbintz/cuke-pack.git'
  gem 'foreman'
  gem 'guard'
  gem 'guard-cucumber'
  gem 'rb-fsevent' # => for mac systems
  gem 'persistent_selenium', :git => 'git://github.com/johnbintz/persistent_selenium.git'
  gem 'poltergeist'
  gem 'capybara-rails-log-inspection', :git => 'git://github.com/johnbintz/capybara-rails-log-inspection.git'
  gem 'rspec'
end
```

### If you need to access a remote service in any way

``` ruby
group :cucumber do
  gem 'webmock', '~> 1.8.0', :require => nil
  gem 'vcr', :require => nil
end
```

### What everything does

Some of my custom gems are pretty cowboyed up, but they work for me, so YMMV.

#### `semantic_rails_view_helpers`

[This gem](http://github.com/johnbintz/semantic_rails_view_helpers) provides view helpers and Capybara finders that, when used with a form builder gem like Formtastic, allow for
very fast form & show construction, and makes it easy to write tests to target elements on those views.

#### `cuke-pack`

[This gem](http://github.com/johnbintz/cuke-pack) helps you set up a lot of common Cucumber environment settings, like support for Timecop, FakeFS, and Mocha;
helps with confirming JavaScript dialogs; enables `cucumber-step_writer`; sets up Guard for a work-in-process workflow; provides a one-at-a-time failed test rerun feature.

#### `cucumber-step_writer`

[Exactly what it sounds like](http://github.com/johnbintz/cucumber-step_writer). Writes out your missing Cucumber steps into `features/steps/(given|when|then)` so you don't
have to copy-paste them anymore.

#### `foreman` and `guard`

For testing, starting both Guard and persistent_selenium is made easy with Foreman. During development, if you have any
dependent services like a Delayed Job worker or API tunnel, Foreman makes it a lot easier to work with.

#### `persistent_selenium`

[Keep a Firefox or Chrome window open](http://github.com/johnbintz/persistent_selenium) while you test, and don't close or reset the page after a failed step. Very helpful
in quickly diagnosing problems and fixing them. Occasional DRb errors, but that's on about one of ever 100 runs or so.

#### `poltergeist`

When you're ready to run your whole test suite, use [the Poltergeist driver](http://github.com/jonleighton/poltergeist), which uses
[PhantomJS](http://phantomjs.org) to run the tests in a super-fast (and slightly finicky in a good way) headless browser.

#### `caypbara-rails-log-inspection`

Normally, errors go in your `log/test.log`. [This](http://github.com/johnbintz/capybara-rails-log-inspection) redirects them to `$stderr`, so you can see exceptions in your Guard
output.

### Remote API integration testing

If your app has to connect to an remote API, it's better to use [VCR](https://github.com/vcr/vcr) to record the actual output from the
API, rather than mocking it directly using WebMock. You may need to mock certain things (Varnish BANs are a common one I have to mock up), but
for the most part, use the actual data from the actual API, and if you're developing both in parallel, rebuild those cassettes often.

## The process

For your continuous integration work, you would want to run `bin/wip_cucumber`, to use both Guard and persistent_selenium.

Let's say you want to add a feature to your application where you see a list of users.
A good Cucumber scenario tries to avoid the nuts and bolts of how a scenario should happen,
instead focusing on as high a level as possible, while needing as few steps as possible.

The approach below is brittle, as it depends on precise text and regexp matches:

``` gherkin
Feature: My feature
  Scenario: Do something
    Given the user "my username" exists with the password "123456"
    Given I am on the home page
    When I click "Login"
    And I fill in "Username" with "my username"
    And I fill in "Password" with "123456"
    And I click "Login"
    ...
```

Knowing the particulars of the username and password, and enumerating how a login works, is
probably overkill, unless you're actually testing login, and even then the approach above is still brittle.
Instead, use a approach like this:

``` gherkin
Feature: My feature
  Scenario: Do something
    Given I exist as a user
    Given I am logged in
    Given I am on the home page
    ...
```

This prevents needing brittle "websteps"-style steps, which were deprecated from Cucumber
years ago. These higher level steps can store the necessary user data in instance variables
for use later:

``` ruby
Given /^I exist as a user$/ do
  @username = 'my username'
  @password = '123456'

  # use bang!-style creates to help catch validation errors as you test.
  # your test setup data should be valid anyway
  @user = User.create!(:username => @username, :password => @password)
end

Given /^I am logged in$/ do
  visit '/'

  # use within liberally, and ensure that your pages have enough
  # identifiers to use it effectively! good use of within means
  # you won't need to use xpath searching
  within '#login-form' do
    # find by CSS selector, since text matching is brittle!
    fill_in 'input[name="user[username]"]', :with => @username
    fill_in 'input[name="user[password]"]', :with => @password

    # this comes from semantic_rails_view_helpers
    find_submit.click
  end
end

Given /^I am on the home page$/ do
  visit '/'

  # find all by itself is an easy way to sanity check a page.
  # it will raise an exception if the element does not exist
  # after Capybara.default_timeout
  find('body.home_page')
end
```

A note about steps that call other steps: one could wrap up all those login steps into a meta-step:

``` ruby
Given /^I am logged in$/ do
  step %{the user "my username" exists with the password "123456"}
  step %{I am on the home page}
  step %{I click "Login"}
  step %{I fill in "Username" with "my username"}
  step %{I fill in "Password" with "123456"}
  step %{I click "Login"}
end
```

But this is even more brittle, since now you have to worry about the formatting of steps within not just
feature files, but in step files as well. My experience with these is to avoid them like the plague.
They will bite you later.

Normally, once you build a feature and run it through Cucumber, either manually or using
Guard, you'll get a list of steps to copy-paste to somewhere in `features/steps`. With this setup,
`cucumber-step_writer` will create the step files for you, with one step per appropriately-named file
in `features/steps/(given|when|then)`:

```
features/
  steps/
    given/
      i_exist_as_a_user.rb
      i_am_logged_in.rb
      i_am_on_the_home_page.rb
```

This makes it very easy to find a particular step using your editor's "find by filename" function,
like with Command-T in Textmate, Sublime Text 2, or with the Vim plugin (surprise) Command-T.

Once the steps are written, you can start filling in the steps, one at a time. Here's what would
happen during your typical test run with each possible driver, given my experience:

* `:rack_test`: Tests move super-fast, but inspecing output HTML requires lots of `puts`. If you can write
  all your finders perfectly the first or second time, and have no need for any JavaScript anywhere, `:rack_test` is very fast.
* `:selenium` and members, like `:firefox` or `:chrome`: The startup time of the browser is a pain, and it's nearly
  impossible to inspect the page state on a failed test. I used to use a `pause` command to pause test execution
  after a failure, inserting `pause` into the failing step, to see what was going on. This involved a lot of test re-running
  and wasted time.
* `:poltergeist` and `:capybara_webkit`: Very fast, JavaScript support, but during development, has the same problem
  as `:rack_test`: inspecting HTML requires `puts`. I save these for pre-commit or whole suite tests. `:poltergeist` is a
  little more stable than `:capybara_webkit` in my experience, and requires less (read: no) things to compile when you install a
  new copy of the gem.
* `:sauce`-based browsers: For when you care about Internet Explorer. A necessary evil.
* `:persistent_selenium`: When used with Foreman, keeps the browser open while testing, retaining the page state on
  a failed test so you can see what happened. Suffers from DRb recycled object errors, and can rack up memory while
  you use it, but it's as close to using `development` mode and reloading something in a browser that you can get.

So my new approach is:

* `:persistent_selenium` while you write and refine and fix your tests/code
* `:poltergeist` once you're ready to commit and re-test the entire application

While testing using `:persistent_selenium`, focus on exactly one scenario at a time (maybe two) by tagging that scenario
with the `@wip` tag:

``` gherkin
Feature: My feature
  @wip
  Scenario: Do something
    Given I exist as a user
    Given I am logged in
    Given I am on the home page
    ...
```

This hijacks the original meaning of `@wip` that Cucumber defined, which is "I am working on this feature, but it's OK,
you can commit this to the repository even though it's not done." The new meaning is "I'm working on this and won't
commit any code until it's done." It's in your best interests at that point to keep features small and focused. If it
takes too much time to set up a feature and run it, perhaps the feature is too complex and some refactoring may be
needed. Or maybe it's just a big feature and you have to deal with it.

Once a test fails or is pending, Firefox will still be open, in the state before the failed/pending test. Use the
inspection tools to diagnose the problem or to build the code necessary to go on to the next step. Keep going
until the test is complete. If you get a `DRb recycled object` error, just run the test again.

Once the scenario is complete, go on to the next one. With regards to feature organization, if working with a CRUD
asset, I often have two features: one for without an existing object and one with an existing object:

```
features/
  users.feature
  users/
    with_user.feature
```

I then copy-paste (oh noes) whatever steps necessary to get `users/with_user.feature` working. Honestly, it really
shouldn't be that many, otherwise you're probably doing something wrong, like being too granular in your
steps. Ideally, those step definitions should be written with an eye toward reusability anyway.

Once you're ready to commit code, if you used the `bintz-integration_testing_setup` binary to set up your project,
the default Rake task runs the `cucumber:precommit` task. Ideally, you should always run all your features all the time.
Sometimes, for practical reasons, you may want to turn a few of the slower/unchanging tests off with the `@no-precommit`
tag. Just remember that you turned them off if you're starting to see problems in your app!

When running a standard Cucumber run (using the `default` profile) or a `precommit` run, if you are using the cuke-pack
`in_progress` hook and there are test failures, you'll get an `in_progress.txt` file in your project root. This
file works very much like Cucumber `rerun.txt` style of testing, except instead of running all features at once (`@rerun.txt`'s
normal behavior), it runs them one at a time until each one passes, eliminating the passing one from the list as you go.
Internally, this uses `@rerun.txt` within a now-smart Cucumber profile, but it only passes in one feature at a time. There's
no need to use `@wip` tags for this either. It will run and eliminate passing scenarios until everything in the `in_progress.txt`
list is fixed. This takes over `@wip` tag processing, and will let you use `@wip` tags again once the list is empty or you
delete `in_progress.txt`.

For now, to make this work with Guard, use [my fork's `paths_from_profile` branch](https://github.com/johnbintz/guard-cucumber/tree/paths_from_profile) of `guard-cucumber`. Once this gets some more human
testing, I'll submit it as a pull request against `guard-cucumber`.

## Testing things that Capybara can't test

Sometimes you need to check HTTP headers, or load a file that Poltergeist of a browser will choke on (loading CSS or JS
files directly into Poltergeist is an error, actually). Then you have to drop to Rack::Test directly. Be sure you
have `Capybara.app_host` set correctly and Rack::Test should "just work":

``` ruby
When /^I request the JavaScript file$/ do
  get Capybara.app_host + '/file.js'
end

Then /^the cache header should be correct$/ do
  last_response.headers['Cache-Control'].should include(@max_age.to_s)
end
```

## Random tips

* Some JavaScript widgets, like WYSIWYG editors, are hard to test. Have an option that replaces them with
  their equivalent native widget during testing. This also falls into the "don't test the framework" category:
  the creator of the widget should be ensuring that the widget starts up and works, not you.
* Don't be afraid to break things up using Plain Old Ruby Objects. ActiveRecord isn't the be-all, end-all design
  pattern. If it's getting too big and you're passing a state around various methods, break it out.
* Make new folders in `app` for things. I usually have one for Delayed Job job objects, one for value objects,
  one for behaviors or concerns (in Rails 4 they have dedicated `app/model/concerns` and `app/controller/concerns`,
  but I also have those apply to other objects in the app's world, hence a dedicated directory for `app/behaviors`.
* VCR note: the innermost cassette is the one that gets recorded to when you're nesting.
* Testing using external services, like [Feed Validator](http://feedvalidator.org/): Force the application to start up
  by `visit`ing a safe page that doesn't do anything, or is wrapped by a VCR call. Then build the URL
  to give to the remote service using `Capybara.app_host`, like you would with `Rack::Test`.
* Sometimes Poltergeist and PhantomJS don't click the right element. This is due to the WebKit renderer being
  silly sometimes. If you're having intermittent problems with a `click` on an element, use `dom_click` instead:

  ``` ruby
  find_submit.dom_click
  ```

  This will use a DOM click in drivers that support DOM event firing, and a regular `click` on other browsers (like Selenium).

## Writing less better organized code

Rails gives you a lot of built-in tools to keep you from writing a lot of code. But there are gems out there that
let you get away with writing even less code, which means you have less things to test. You should never "test the
framework" when you write tests -- assume that the author of the tools you use have tested them thoroughly, unless
otherwise discovered (and then help the author by pinpointing the problem and sending code to fix it!) That's
why I'm not a fan of things like model validation tests: the model file clearly says `validates :something`, there's
no need to test that directly, unless you think Rails itself is broken. Indirect testing through integration
testing is fine, though (but probably mostly unnecessary).

For a "typical" Rails application which provides some sort of HTML-based interface to end users, I'll use
most of, of not all of, the following gems:

* [inherited_resources](http://github.com/josevalim/inherited_resources): Rails controller generators are OK, but this
  is way faster. With this, you don't even have to feel guilty about not having controller unit tests that are
  essentially "I have set an instance variable and called the render method".
* [formtastic](http://github.com/justinfrench/formtastic): Writing forms is lame. Formtastic lets you do it a lot
  faster, makes it easier to target your form elements using Capybara and semantic_rails_view_helpers, and
  makes writing custom form elements a snap, even ones that have JavaScript in them.
* [draper](http://github.com/drapergem/draper): Views shouldn't have any more logic than the occasional `if` statement.
  Put the rest of that logic in decorators. Combine that with [decorates_before_rendering](http://github.com/ohwillie/decorates_before_rendering)
  for stupid-simple decorators.
* [Active Admin](http://github.com/gregbell/active_admin): If you don't need to write an admin interface, don't do it.
* [cocoon](http://github.com/nathanvda/cocoon) and [my fork](http://github.com/johnbintz/cocoon): `accepts_nested_attributes_for` get some well-deserved hate, but
  if you truly are only putting nested attributes into an object in a single way, and you're doing it with a gem like cocoon, then it's perfectly acceptable IMHO.
  Write a Form Object to accept the input if you really care that much about that sort of thing.
* [virtus](http://github.com/solnic/virtus): For things like View Objects and Value Objects and Form Objects, make those objects behave like
  ActiveRecord objects. (Read [this article](http://blog.codeclimate.com/blog/2012/10/17/7-ways-to-decompose-fat-activerecord-models/) for some background on that.)

For something that's more of a "service", I'll still use what tools I can, but the test flow may be different.

Of course, you'll have your authentication gems and your asset-related gems and your pagination gems, but those
aren't about writing less code or better organized code.

## Thing will go wrong and my code will fail. Get over it.

I fully expect that I will make mistakes during development, either through incorrect assumptions or just plain shoddy work.
I'm getting better at programming, but I'm not perfect. I use, at the very least, [exception_notification](https://github.com/smartinez87/exception_notification)
to catch problems. If I have access to Errbit or Airbrake or something similar for catching and reporting exceptions, I'll use it. When it breaks,
if my actual code is clean enough, it should be easy to create a new feature, or fix a bad one, and get the problem resolved.

## Changelog

* v0.2.1 (2013-02-18): Note about using external validation services during testing
* v0.2 (2013-02-14): Add `cuke-pack`'s `in_progress` mode.'
* v0.1.2 (2013-01-30): Small update for VCR and finishing a sentence
* v0.1.1 (2013-01-28): Tiny spelling and pucntuation changes
* v0.1 (2013-01-25): Initial brain dump