Smoke Testing: Software Applications

Smoke testing refers to the process of performing a few tests to verify that the functionality of an application or website is working before proceeding to more extensive software testing.

Smoke testing is essential to ensure that it's efficient and provides quick feedback to developers about the stability of the build. A good platform for automated smoke testing will allow you to run tests in just a few minutes. Manual testing can take several hours. This is why many teams opt to do automated smoking testing instead of having someone interact with the software.

This tutorial will show you how to create basic smoke testing suites using QA. Each step will be illustrated using examples from Airbnb's website.

QA allows you to run automated smoke tests every month for five hours without any code. After that, it's $5/hour.

What's smoke testing?

Smoke tests, as mentioned earlier, are the first set to be run on a new build in order to verify that the application works properly in a production environment. This name could be a result of the early days in electronic hardware testing. You should turn the device on for the first time and ensure it doesn't catch on fire before proceeding with any further testing. Software engineering is a metaphorical term. Although you don't expect to see smoke, you still want to find similar catastrophic problems.

After a build passes integration testing, many teams move to smoke testing. This ensures that each module of code works in harmony.

Smoke testing, also known as build-verification testing, is usually performed by the developer who is working on the build before it goes to more thorough testing. Smoke testing is different from unit tests which are focused on specific sections of a build during the initial stages of development. It helps to catch any major bugs that may have been unintentionally introduced into other parts of the software.

Smoke testing has the advantage of identifying major flaws in a new build early in the development process. This prevents developers and testers spending time fine-tuning or further testing.

If a crucial functionality, such as the login process, was not working in the new build of the software, almost all the tests you run would fail. You can save money and time by running a few tests instead of running your entire regression suite, which could have more than 100 tests.

The smoke tests will be used to verify the functionality of five to ten of the major functions of the software. They will remain the same regardless of the product you are building.

How to Select Smoke Testing Cases

Smoke tests should include the top five to ten most common user paths for your software. These would include the most important features that customers use and/or functionalities that would prevent you from delivering your product or service to customers.

As an example, take Airbnb. Their smoke tests would consist of one or two cases that cover the following user paths:

• Looking for a place to stay?
• Reserve a room
• Hosts can be mediated
• Being a host
• Logging into a personal account

Once you have determined which test cases should be included in your smoke suite each test should cover the functionality with the shortest possible steps. This is so that you can quickly find major bugs in the basic functionality. Long, complicated test cases will slow down release and not significantly improve quality.

Here's a quick test to filter locations on Airbnb.

Smoke Testing vs. All Other Types of Testing

Regression testing and sanity testing are two additional types of testing that can be confused with smoke testing.

Smoke Testing vs. Sanity Testing

Software testing includes sanity testing. This is a subset of regression testing that covers the new functionality and bug fix. Smoke testing checks that the entire application is stable. Sanity testing verifies specific features in the new build.

Contrary to smoke tests which are static for every build, sanity testing will change with every new build or bug fix.

Although sanity testing is less common than smoke testing, it is still widely used. Many quality assurance teams do not include sanity tests in their formal testing strategies. However, some do what could be called "sanity" testing. This type of testing is usually done manually by the person who will be performing regression testing.

Let's take Airbnb as an example. A bug caused the'showmap' button to disappear. After the bug was fixed, a test of sanity would be performed to verify that the "show map" button was functional and visible.

Smoke Testing vs. Regression Testing

Regression testing is usually the last stage of the testing process. It involves thoroughly testing the entire application for any bugs that might have been introduced by new codes.

Because software development is complex, regression tests are essential. Changes to one component of an application may cause unexpected changes in other parts. While most developers will perform a quick manual inspection of the component of an application that is relevant to the feature being worked on, they don’t have the time to go through every user path to ensure the change didn’t cause any problems.

A regression test suite may contain 500+ tests for complex applications. However, it does not usually have to include a case for each user path. It is important to verify that all of the key features work properly.

This article on automation test coverage will help you choose which cases to include in your regression test suite. Software regression testing can be found here.

If we take Airbnb as an example, our regression suite would likely contain test cases that cover everything you need to know about logging in, searching and booking a stay, filtering your search, saving searches as favorites, booking, messaging hosts, and so on. As you develop new features for your software, you will continue to add test cases to your regression suite.

Comparison of Smoke Testing and Other Types Of Software Testing

Software testing begins with smoke testing. Once an application passes smoke testing it can move on to other types software testing such as:

• Acceptance Testing and System Testing are closely related types that test whether a new version of software meets the original requirements.
• Performance testing aims to verify system stability when there are high demands.

How To Automate Smoke Tests

Smoke tests are usually run more often than other types. Teams, especially those that are running continuous integration pipelines, may run their smoke suite multiple times per day. Regression suites will only be run once per week.

Although manual smoke testing is possible, many teams prefer to automate their tests. It's faster and easier. You can start the test and receive results within minutes. Instead of waiting for manual testing to be completed by human testers, you only need to initiate it.

Although we have written a detailed article on automated smoke testing, here is an overview of how QA can be used to automate smoke tests:

You must first select an action (e.g., double click, fill, etc.) to write or edit any test step. Select an action from the dropdown menu. If you don't see the action you want, you can search for it. Drag and drop the box with your mouse to take a picture of the element to which you wish to apply the action. This screen capture is used to verify, interact with and find elements during test runs.

This allows anybody to create smoke-test cases without having to write a single line code.

makes it easy to launch your smoke suite or any other set of tests whenever you are ready. It takes just a few clicks. All tests will be run simultaneously on our virtual machines network, returning results in an average of less than 4 minutes.

can be integrated into your CI/CD pipeline using our API, CLI or CircleCI Orb. This allows you make smoke testing suite a mandatory checkpoint in your release process. Your CI/CD software will automatically start the test run after the previous step is completed (e.g. Unit testing. We offer integrations with Slack or Microsoft Teams so that you can receive notifications about test failures from wherever you are most convenient.

Results of the Smoke Triage Test

After you decide to automate your smoke test, you need to think about what tools you have for understanding the reasons behind failures and categorizing them.

It can be difficult to understand why a test fails if you don't have the right tools. It is especially important to be able to quickly triage test results when you run smoke tests several times per day.

QA allows you to quickly identify the cause of a failure by providing video replays, detailed logs, and detailed logs on how the test was conducted and the result.

Quality Assurance stores every test run whether it passes or fails. You'll be able to see both the steps and the execution of each step in every recording.

This allows you to compare the results of a successful and failed test run. This is often a key to understanding why test failures occur. Video replays allow you to easily recreate the bug since they show everything that led up to it.

You can click on the "investigate action" button at any test step if you wish to dig deeper into that test step. This will open a new screen that provides additional information about the test step's result.

It is also possible to determine if the test failed due to an error in the test or an issue within the environment. QA provides HTTP logs as well as browser logs to help you identify problems in your test environment.

also has a Jira integration, which allows you to create an automatic ticket to send to the developers for bug fixes. The Jira ticket will contain the failed test steps, the screenshot of the failed step, HTTP logs and a link to the complete test results and video recording.

Free Automated Smoke Testing

Open-source frameworks such as Selenium that allow you to write and run tests are available. However, these tools don't usually offer all the functionality you need for functional testing. Software testing tools, for example, won't allow you to run multiple tests simultaneously. Most teams eventually end up paying something.

You can still use a free tool by clicking here.

The Professional plan provides five hours of unlimited testing each month. This is enough time for most testing teams, allowing them to run small sets of smoke tests every day. You won't be charged any time you spend writing, editing or managing your testing suite. Only the test duration will be charged. You can run several tests to get familiar with the tool, and you won't be charged for them.

If you use all five hours in your monthly free testing, you'll pay $5 per hour for any time you spend running your tests.

• Simple management of test suites (including tags, folders, filters)
• You can run multiple tests simultaneously on a network of virtual machines.
• Multiple operating systems and browsers are available for testing, including older and current versions of Chrome, Firefox and Internet Explorer.
• Built-in test data like randomized email addresses, credit card numbers, names, etc.
• There are many other options.

There is no limit to the number of tests that you can run per month, nor how many users you may add. You won't be tied down by a contract and can scale up or decrease testing at any stage of the process.