Smoke testing

Smoke testing, also called build verification testing or build acceptance testing, is nonexhaustive software analysis that ascertains that the most crucial functions of a program work but does not delve into finer details.

Smoke testing is the preliminary check of the software after a build and before a release. This type of testing finds basic and critical issues in an application before critical testing is implemented.

The term smoke testing originates from a similarly basic type of hardware testing in which a device passes the test if it doesn't catch fire the first time it turns on. Daily build and smoke tests are among industry best practices advocated by the Institute of Electrical and Electronics Engineers in Code Complete by former IEEE editor in chief and software engineer expert Steve McConnell.

How smoke testing works

Quality assurance (QA) testers perform smoke testing after the developers deliver every new build of an application. If the code passes the smoke testing, the software build moves on to more rigorous tests, such as unit and integration tests. If the smoke test fails, then the testers have discovered a major flaw that halts all further tests. QA then asks developers to send another build. This one broad initial test is a more effective strategy to improve software code than if the team conducted specific and rigorous tests this early in the development process.

Smoke testing is also performed from the perspective of user experience (UX). This approach includes testing key functionalities, such as if the build is accessible or if the user interface (UI) and login mechanism function. Other key functionalities include if an action selection correlates with the intended action. For example, if a user adds an item to a shopping cart in an e-commerce web application, does the item appear in the cart?

Automation and smoke testing

Automated smoke testing can drastically cut down on testing time. Manual testing can take half a day depending on the number of smoke tests involved. After automating those tests, smoke testing can take only a few minutes. If developers make builds frequently or if continuous testing is implemented, then automating smoke tests will enable QA testers to give faster feedback on test builds.

Tools for automated smoke testing include Selenium and PhantomJS. Selenium is an open source software that can automate and run testing parameters on multiple web browsers. Selenium automates control of browsers on various operating systems (OSes). PhantomJS is an option for integration and automation with continuous integration (CI) tools, such as Jenkins and TeamCity. PhantomJS is not resource-intensive and works well in networking monitoring.

Smoke testing vs. regression testing

Regression testing is a separate way to verify good software code. Regression testing ensures that changes to a program do not add new bugs that inadvertently compromise the performance or integrity of the software. Similar to smoke tests, regression tests should be implemented often, with every new build. Unlike smoke tests, however, regression tests are in-depth and detailed.

Regression tests target specific operations of software and can take a few hours to complete. For example, regression testing explicitly checks each link on an updated webpage to verify that each link still works as intended. Some use cases in regression testing can be automated, such as verifying that adding an item to a cart works or that users can still navigate to their cart and pay.

Smoke testing and sanity testing

There is confusion between the terms smoke testing and sanity testing in the software industry. Some professionals use the term sanity testing as an equivalent to smoke testing, while others use sanity testing as a term that refers to a subset of regression testing. When the term sanity testing is used to mean a subset of regression testing, the scope of testing is often assumed to be narrow and in-depth.