Skip to main content

Flaky test


A flaky test is a quality assurance (QA) test that fails to produce consistent results. Typically, a flaky test using the same software code and the same configuration will produce both a passing result and a failing result.

Whenever new code is written to develop or update computer software, a web page or an app, it needs to be tested for quality assurance. Ideally, each time the code is tested, the results are consistent. The code will either work as expected and pass the test, or not work as expected and fail the test. Sometimes, however, QA tests on the exact same code, using the exact same configurations, will produce inconsistent results. When this happens, the test is labeled "flaky." Unfortunately, flaky tests are not uncommon -- Google, for example, reports that 16 percent of its tests show some level of flakiness.

Flaky tests can be caused by various factors, including:
  • an issue with the newly-written code
  • an issue with the test itself
  • some external factor that compromises the test results

Once a test is deemed flaky, there are different approaches to dealing with the muddled results. Some developers will ignore the flakiness entirely, assuming that the issue is with the test and not with the newly-written code. Others will rerun their test multiple times and only go back to investigate further if the test fails a certain number of times in a row, indicating to them a true failure.

However, the safest approach -- the only way to truly find out whether there is a bug in the code -- is to halt the development of the application, fully investigate the cause of the flaky test and resolve it. If left unresolved and there truly is an issue with the code, one problem has the potential to wind up leading to another and another as more is built onto the faulty code.

When investigating the cause of a flaky test, the developer will need to gather data to try to discover differences within the seemingly random results in order to isolate the cause of the failed tests. The code should be re-examined, as should the test itself, and if no issues are found then external factors will need to be looked at to see if they might be at the core of the problem. The developer might look at whether the tests that passed were run at a certain time of day whereas the ones that failed were run at a different time of day, whether certain programs were running on the developer's computer at the same time of failed tests that weren't running when the tests passed or whether the tests that failed did so at the same point in the test or at different times during the test.

Sometimes, the cause of the flaky test is simple to diagnose and can be quickly fixed. That's the best-case scenario. Other times, there is no easy fix, and though potentially costly and time-consuming, the developer may need to delete the test and rewrite it from scratch in order to ensure the accuracy of the test results.

Comments

Post a Comment

Popular posts from this blog

Understanding the Evolution: AI, ML, Deep Learning, and Gen AI

In the ever-evolving landscape of artificial intelligence (AI) and machine learning (ML), one of the most intriguing advancements is the emergence of General AI (Gen AI). To grasp its significance, it's essential to first distinguish between these interconnected but distinct technologies. AI, ML, and Deep Learning: The Building Blocks Artificial Intelligence refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. Machine Learning, a subset of AI, empowers machines to learn from data and improve over time without explicit programming. Deep Learning, a specialized subset of ML, involves neural networks with many layers (hence "deep"), capable of learning intricate patterns from vast amounts of data. Enter General AI (Gen AI): Unraveling the Next Frontier Unlike traditional AI systems that excel in specific tasks (narrow AI), General AI aims to replicate human cognitive abilities across various domains. I...
Post a Comment
Read more

Normalization of Database

Database Normalisation is a technique of organizing the data in the database. Normalization is a systematic approach of decomposing tables to eliminate data redundancy and undesirable characteristics like Insertion, Update and Deletion Anamolies. It is a multi-step process that puts data into tabular form by removing duplicated data from the relation tables. Normalization is used for mainly two purpose, Eliminating reduntant(useless) data. Ensuring data dependencies make sense i.e data is logically stored. Problem Without Normalization Without Normalization, it becomes difficult to handle and update the database, without facing data loss. Insertion, Updation and Deletion Anamolies are very frequent if Database is not Normalized. To understand these anomalies let us take an example of  Student  table. S_id S_Name S_Address Subject_opted 401 Adam Noida Bio 402 Alex Panipat Maths 403 Stuart Jammu Maths 404 Adam Noida Physics Updation Anamoly :  To upda...
Post a Comment
Read more

How to deal with a toxic working environment

Handling a toxic working environment can be challenging, but there are steps you can take to address the situation and improve your experience at work: Recognize the Signs : Identify the specific behaviors or situations that contribute to the toxicity in your workplace. This could include bullying, harassment, micromanagement, negativity, or lack of support from management. Maintain Boundaries : Set boundaries to protect your mental and emotional well-being. This may involve limiting interactions with toxic individuals, avoiding gossip or negative conversations, and prioritizing self-care outside of work. Seek Support : Reach out to trusted colleagues, friends, or family members for support and advice. Sharing your experiences with others can help you feel less isolated and provide perspective on the situation. Document Incidents : Keep a record of any incidents or behaviors that contribute to the toxic environment, including dates, times, and specific details. This documentation may b...
Post a Comment
Read more