Top 10 Reasons for Flaky Automated Tests

Test Automation Published on:
Top 10 Flaky Test



Top Ten Reasons for Flaky Automated Tests

1. Not having a framework

2. Using hardcoded test data

3. Using X,Y coordinates or XPath for element recognition

4. Using shared test environments

5. Having tests that are dependent on one another

6. Test not starting in a known state

7. Test no managing their own test data

8. Not treating automation like any other software development effort

9. Failure to use proper synchronization

10. Badly written tests

 

9 responses to “Top 10 Reasons for Flaky Automated Tests”

  1. I’d say your list is pretty much on target. Only thing I could add is that people have a tendency to make their tests too complex and do too much. They need to break them down and make them more “directed” in their purpose. Use the KISS method of design/construction of an automated test. Get in, get it done and then get out.

    Also to the point of not starting in a known place, better known as State Restoration, people tend not to clean up after a test has been run (true State Restoration). Use the SEARCH method; Setup, Execute, Analyze, Report, Cleanup and Home.

  2. I agree with a lot of the 10 points except part of 3. I use xpath to locate elements almost exclusively and it is only a source of ‘flakiness’ when the attribute is changed. I should say I don’t use brittle xpaths, I use * and contains a lot. I try to stay away from using class names, mostly using our own ‘qaid’s for location of key components. One of the biggest sources of flakiness has been our test bed machines (cluster of VM systems ) having systemic issues. (chromedriver debris left after a test closes) And as we’ve honed our test image and clean up processes, that has been less and less an issue.

    Good to see you are still around, I haven’t been on SQAForums much since the change

    • Hi Steve! Good point – I guess I would call these good practices in general. In my experience with new development on a green field application the XPath is always changing. If you have an application where the XPath is always stable then I agree use it but I would still recommend using a unique ID or Name before you did. I can never remember my SQAforums user name and password but I should to go on it more often :)

  3. Hey Joe

    Great post.

    Hey Joe

    Great post.

    I would like to add on to point number 6: “Scripts not executing from known points”. In other words, not making scripts generic enough to account for unexpected states and outcomes.

    For example, suppose a script inputs data into a search screen but does not enter data into all of the fields and selects the result. That is all good and well until the data changes and more than one result is returned and the unwanted result gets selected.

    Laurence

  4. Hi,

    I was struggling with flaky tests as well. It may be a huge pain. All mentioned reasons in this article are so true. Knowing potential causes helps a lot with fighting with them.
    I was also using test quarantine and retests. You can read more about here:
    https://martinfowler.com/articles/nonDeterminism.html
    or here:
    https://qualityengineer.blog/2018/best-practices-1-test-quarantine/

    For me very helpful was the article from Google about their battle with flaky tests:
    https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Top 10 Flaky Test