The nightmare of flaky tests

The nightmare of flaky tests. And how to solve it.

Remember those tests which SOMETIMES pass and SOMETIMES fail… without changes in code?

We start to ignore those tests… And lose trust in our test suite… And eventually, abandon it.

What are the SYMPTOMS of flaky tests?

– TEST ORDER DEPENDENCY: Tests are dependent on order of execution. This means you can’t execute one test in isolation, but need to run some other test before it.

– SHARED STATE IN PARALLEL TESTS: Tests interfere with each other. Suppose we have one test which is testing that we can place a new order, whereas another test is testing that we have 100 orders. Or perhaps the test data cleanup wasn’t done well.

– DEPENDENCY ON SYSTEM CLOCK: Tests are dependent on the system time. Suppose that we’re testing the calculation for electricty bills, and the price is dependent on the time of day. Or if we’re testing sales discounts which are dependent on months, dates…

– WAIT TIME & TIMEOUTS: Tests fail due to the assertion wait time. So imagine if you set the waiting time 500ms for a message to appear on the UI, perhaps there’s a network timeout.

– MEMORY LEAKS: If you have a memory leak in your application, it is possible for tests to start failing after you’ve reached a certain resource usage level.

What’s the ROOT CAUSE of flaky tests and how to SOLVE them?

– NO ISOLATION: When tests are isolated, they can be run in parallel in any order (i.e. unit tests). When tests are not isolated (e.g. due to database, file system, singletons and shared data) then we have to take care of starting with a clean slate before the test or cleaning up after the test.

– ASYNCHRONOUS SERVICES: When you are testing asynchronous services, the problem is that you need to wait for the asynchronous service to complete. If the asynchronous service enables you to take a callback, otherwise polling, or a hardcoded wait time limit.

– EXTERNAL SERVICES: When you’re integrating with external services, you don’t have much control. The external service may sometimes be running sometimes not. So you could use Test Doubles for external services to minimize these issues.

– TIME & RANDOM GENERATORS: You’ll get different results every time you call the current date and time. Similar issue occurs with random number generators. In the case of unit tests, Test Doubles enable you to have control.

– RESOURCE LEAKS: This one might be harder to detect, but you can set resource pool limits, then easier to control.

HOW to structure our test suite?

All the problems above can be prevented with unit tests because unit tests enable true isolation. This is why the majority of your tests should be unit tests. Only a smaller percentage of your tests should be integration tests, and smallest percentage E2E tests. This is where you need strategies to minimize flaking.

https://lnkd.in/dGUZ5BnA
https://lnkd.in/d4uTkmPF
https://lnkd.in/dhEHviFD