How to handle flaky specs

We'll use the following failure as an example:

  1) Poll Officing Officing dashboard available for multiple sessions
     Failure/Error: select 'DNI', from: 'residence_document_type'
     Capybara::ElementNotFound:
       Unable to find visible select box "residence_document_type" that is not disabled
     # ./spec/features/officing_spec.rb:160:in `block (3 levels) in <top (required)>'
     # ./spec/sessions_helper.rb:5:in `in_browser'
     # ./spec/features/officing_spec.rb:156:in `block (2 levels) in <top (required)>'

1. Confirm it's a flaky

Try to reproduce the spec failure in localhost (on the correct git branch)

2. Search for open/closed issues

Check if there's an open issue with the failure name Poll Officing Officing dashboard available for multiple sessions on the issue title https://github.com/ayuntamientomadrid/consul/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22Flaky+spec%22 or a closed one that should be reopened https://github.com/ayuntamientomadrid/consul/issues?q=is%3Aissue+sort%3Aupdated-desc+label%3A%22Flaky+spec%22+is%3Aclosed

3. Create a flaky spec issue

Open a new issue using the template https://github.com/ayuntamientomadrid/consul/issues/new?template=flakies.md&title=Flaky%20spec:%20&labels=Flaky%20spec,PRs-welcome&project=Roadmap and fill it:

• Append to existing Flaky spec: title the failing test name Poll Officing Officing dashboard available for multiple sessions so it will be easy to find in the future.
• Search at travis build Randomized with seed, copy that seed number and at the PR description replace FILL_WITH_RANDOM_SEED with the number
• Replace FILL_WITH_FAILURE_REPORT at the PR description with the complete failure (like the example code used at the beginning of this list)

4. Once we have a PR

Check the template questions have been answered:

• Explain why the test is flaky, or under which conditions/scenario it fails randomly
• Explain why your PR fixes it If not, please doubt the solution isn't a placebo.

Unless its pretty clear why the failure happened and that the fix its 100% effective... Comment on the PR saying "All flakies" (if there are some) and "Restarting travis build #1" and restart the travis build

Repeat the Comment & Restart 2-5 times (if we have the time, better to be sure than having a flakie issue reopened in the future)

5. Once the PR is approved and merged

Remember the author to Create a backport PR to consul/consul when the fixing PR is approved as stated in the issue template. Better with a comment on the issue after PR approval and merge.

6. Close issue but keep an eye on every failing travis build for reactivation of the flaky