What's a flaky test?
It's a test that sometimes fails, but if you retry it enough times, it passes, eventually.
Automatic retries and flaky tests detection
We also use a home-made
RspecFlaky::Listener listener which records flaky
examples in a JSON report file on
update-tests-metadata jobs), and warns when a new flaky example
is detected in any other branch (
flaky-examples-check job). In the future, the
flaky-examples-check job will not be allowed to fail.
This was originally implemented in: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13021.
Problems we had in the past at GitLab
rspec-retryis bitting us when some API specs fail: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9825
Sporadic RSpec failures due to
- FFaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!):
spec/mailers/notify_spec.rbmore robust: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10015
- Transient failure in spec/requests/api/commits_spec.rb: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9944
- Replace FFaker factory data with sequences: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10184
- Transient failure in spec/finders/issues_finder_spec.rb: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10404
Time-sensitive flaky tests
Array order expectation
- Be sure to create all the data the test need before starting exercize: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12059
- Bis: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12604
- Bis: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12664
- Assert against the underlying database state instead of against a page's content: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10934
Capybara viewport size related issues
- Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10411
Capybara JS driver related issues
- Don't wait for AJAX when no AJAX request is fired: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10454
- Bis: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12626
PhantomJS / WebKit related issues
- Memory is through the roof! (TL;DR: Load images but block images requests!): https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12003
- Flaky Tests: Are You Sure You Want to Rerun Them?
- How to Deal With and Eliminate Flaky Tests
- Tips on Treating Flakiness in your Rails Test Suite
- 'Flaky' tests: a short story
- Using Insights to Discover Flaky, Slow, and Failed Tests