Skip to main content

Flaky test


A flaky test is a quality assurance (QA) test that fails to produce consistent results. Typically, a flaky test using the same software code and the same configuration will produce both a passing result and a failing result.

Whenever new code is written to develop or update computer software, a web page or an app, it needs to be tested for quality assurance. Ideally, each time the code is tested, the results are consistent. The code will either work as expected and pass the test, or not work as expected and fail the test. Sometimes, however, QA tests on the exact same code, using the exact same configurations, will produce inconsistent results. When this happens, the test is labeled "flaky." Unfortunately, flaky tests are not uncommon -- Google, for example, reports that 16 percent of its tests show some level of flakiness.

Flaky tests can be caused by various factors, including:
  • an issue with the newly-written code
  • an issue with the test itself
  • some external factor that compromises the test results

Once a test is deemed flaky, there are different approaches to dealing with the muddled results. Some developers will ignore the flakiness entirely, assuming that the issue is with the test and not with the newly-written code. Others will rerun their test multiple times and only go back to investigate further if the test fails a certain number of times in a row, indicating to them a true failure.

However, the safest approach -- the only way to truly find out whether there is a bug in the code -- is to halt the development of the application, fully investigate the cause of the flaky test and resolve it. If left unresolved and there truly is an issue with the code, one problem has the potential to wind up leading to another and another as more is built onto the faulty code.

When investigating the cause of a flaky test, the developer will need to gather data to try to discover differences within the seemingly random results in order to isolate the cause of the failed tests. The code should be re-examined, as should the test itself, and if no issues are found then external factors will need to be looked at to see if they might be at the core of the problem. The developer might look at whether the tests that passed were run at a certain time of day whereas the ones that failed were run at a different time of day, whether certain programs were running on the developer's computer at the same time of failed tests that weren't running when the tests passed or whether the tests that failed did so at the same point in the test or at different times during the test.

Sometimes, the cause of the flaky test is simple to diagnose and can be quickly fixed. That's the best-case scenario. Other times, there is no easy fix, and though potentially costly and time-consuming, the developer may need to delete the test and rewrite it from scratch in order to ensure the accuracy of the test results.

Comments

Popular posts from this blog

Computer Operating Systems

When you turn on the power to a computer, the first program that runs is usually a set of instructions kept in the computer's  read-only memory  (ROM). This code examines the system hardware to make sure everything is functioning properly. This  power-on self-test  (POST) checks the CPU,  memory , and  basic input-output systems  (BIOS) for errors and stores the result in a special memory location. Once the POST has successfully completed, the software loaded in ROM (sometimes called the BIOS or firmware ) will begin to activate the computer's disk drives. In most modern computers, when the computer activates the  hard disk drive, it finds the first piece of the operating system: the  bootstrap loader . The bootstrap loader is a small program that has a single function: It loads the operating system into memory and allows it to begin operation. In the most basic form, the bootstrap loader sets up the small driver programs that interface ...

Shaping the Digital Economy: From Infrastructure Thinking to Ecosystem Orchestration

Digital transformation is often discussed in terms of systems, platforms, and technologies. But at a national level, the real challenge is not building infrastructure, it is ensuring that infrastructure becomes economy-shaping capability . As digital ecosystems mature, the focus is shifting from isolated innovation to system-wide orchestration . The question is no longer “What systems do we build?” It is becoming “How do we ensure everything works together as one economy?” Beyond Infrastructure: The Limits of Building Alone Most digital economies begin with infrastructure development: Payment switches Mobile money platforms Core banking systems Digital identity frameworks Connectivity layers These are necessary foundations, but they do not automatically create impact. Without alignment across institutions, infrastructure risks becoming: Technically functional but underutilized Interoperable on paper but fragmented in practice Innovative in design but slow in ad...

Artificial intelligence for IT operations (AIOps)

Artificial intelligence for IT operations (AIOps) is the use of deep learning and big data analytics to automate routine administrative tasks, including deployment, root cause analysis and problem resolution, for an information technology (IT) system. Ideally, an AIOps platform brings three important capabilities to the enterprise: The ability to recognize abnormal system behavior faster and with greater accuracy than humanly possible. The ability to use IFTTT business rules to automate routine tasks. The ability to streamline communication among stakeholders. How AIOps works AIOps tools gather information from the IT tools and devices already in place and apply detailed analytics and machine learning to that information in order to identify potential issues and correct them. Typically, AIOps data comes from network log files, cloud monitoring tools and helpdesk ticketing systems. Big data technologies aggregate and organize all of the systems' outpu...