“A liar will not be believed, even when he speaks the truth.”

Front-End Testing: Why Even Bother?

The Science Of Flaky Tests

“Ahh, that build. Nevermind, just kick it off again. It will eventually pass, somewhen.”

The Causes

1. Test-Side Causes

{
   "id": "f1d2554b0ce847cd82f3ac9bd1c0dfca",
   "name": "Variant product",
}

<!-- This is a text field I took from a project I worked on -->
<input type="text" id="sw-field--f1d2554b0ce847cd82f3ac9bd1c0dfca" />

// Please never do that unless you have a very good reason!
// Waits for 1 second
browser.pause(1000);

2. Environment-Side Causes

3. Product-Side Causes

Ways To Fight Flakiness

Focus On Your Team

1. Do nothing and accept the flaky test result.
   Of course, this strategy is not a solution at all. The test will yield no value because you cannot trust it anymore — even if you accept the flakiness. So we can skip this one pretty quickly.
2. Retry the test until it passes.
   This strategy was common at the start of my career, resulting in the response I mentioned earlier. There was some acceptance with retrying tests until they passed. This strategy doesn’t require debugging, but it is lazy. In addition to hiding the symptoms of the problem, it will slow down your test suite even more, which makes the solution not viable. However, there might be some exceptions to this rule, which I’ll explain later.
3. Delete and forget about the test.
   This one is self-explanatory: Simply delete the flaky test, so that it doesn’t disturb your test suite anymore. Sure, it will save you money because you won’t need to debug and fix the test anymore. But it comes at the expense of losing a bit of test coverage and losing potential bug fixes. The test exists for a reason! Don’t shoot the messenger by deleting the test.
4. Quarantine and fix.
   I had the most success with this strategy. In this case, we would skip the test temporarily, and have the test suite constantly remind us that a test has been skipped. To make sure the fix doesn’t get overlooked, we would schedule a ticket for the next sprint. Bot reminders also work well. Once the issue causing the flakiness has been fixed, we’ll integrate (i.e. unskip) the test again. Unfortunately, we will lose coverage temporarily, but it will come back with a fix, so this will not take long.

Keep Tests Isolated

// File: customer-login.spec.js
let customer = {};

beforeEach(() => {
    // Set application to clean state
    cy.setInitialState()
      .then(() => {
        // Create test data for the test specifically
        return cy.setFixture('customer');
      })
}):

Further Optimize The Test Structure

Wait! Test Retries Are Sometimes OK?

test:
    script: rspec
    retry:
        max: 2
        when: runner_system_failure

{
    "retries": {
        // Configure retry attempts for 'cypress run`
        "runMode": 2,
        // Configure retry attempts for 'cypress open`
        "openMode": 2,
    }
}

Using Dynamic Waiting Times

// Wait for changes in UI (until element is visible)
cy.get(#element).should('be.visible');

// File: checkout-info.spec.js

// Define request to wait for
cy.intercept({
    url: '/widgets/customer/info',
    method: 'GET'
}).as('checkoutAvailable');

// Imagine other test steps here...

// Assert the response’s status code of the request
cy.wait('@checkoutAvailable').its('response.statusCode')
  .should('equal', 200);

Debugging Flaky Tests

// Use in build Lodash to repeat the test 100 times
Cypress._.times(100, (k) => {
    it(`typing hello ${k + 1} / 100`, () => {
        // Write your test steps in here
    })
})

it('should be a Vue.JS component', () => {
    // Mock component by a method defined before
    const wrapper = createWrapper();


    // Print out the component’s html
    console.log(wrapper.html());

    expect(wrapper.isVueInstance()).toBe(true);
})

Fight Flakiness Nightmares!

Spotting The Red Flags

• The test is large and contains a lot of logic.
• The test covers a lot of code (for example, in UI tests).
• The test makes use of fixed waiting times.
• The test depends on previous tests.
• The test asserts data that is not 100% predictable, such as the use of IDs, times, or demo data, especially randomly generated ones.

Further Reading

• Articles about “flake,” Cypress.io
• “Retrying Your Tests Is Actually a Good Thing (If Your Approach Is Right),” Filip Hric, Cypress.io
• “Test Flakiness: Methods for Identifying and Dealing With Flaky Tests,” Jason Palmer, Spotify R&D Engineering
• “Flaky Tests at Google and How We Mitigate Them,” John Micco, Google Testing Blog