The Different Types of software Testing ​

Software Testing Principles ​

Everything in tech has principles. These are guidelines to help you build better software and avoid errors.

Here are some software testing principles you should follow when writing tests for your code:

Testing aims to show the presence of defects, not the absence ​

Software testing aims to spot software failures. This reduces the presence of faults and errors.

Software testing ensures defects are visible to the developer but doesn't guarantee defect-free software. Multiple types of testing can't even ensure error-free software. Testing can only decrease the number of errors.

Exhaustive testing is not possible ​

Exhaustive Testing is the process of testing software for all valid and invalid inputs and pre-conditions.

This method of testing is not realistic because test cases presume that the software is correct and it produces the correct output in every test case. If you truly try to test every aspect and test case in your software, it will take too much time and effort, and it's not practical.

Perform early testing ​

Testing your software at an early phase helps avoid minor bugs or errors. When you can spot errors at an early stage of the Software Development Life Cycle(SDLC), it's always less expensive. It is best to start software testing from the beginning of the project.

Defect clustering ​

Defect clustering refers to when most of the problems you find occur in just a few parts of the application or software. If you can identify the modules or areas where these defects occur, you can focus most of your testing efforts on them.

Keep the Pareto Principle in mind when testing your code: 80% of software defects tend to come from 20% of the modules.

Beware of the Pesticide paradox ​

This principle is based on a theory – "the more you use pesticide on a crop, the more immune the crop will eventually grow, and the pesticide will not be effective."

When you repeat particular test cases over and over, you will see fewer and fewer new bugs. So to find new bugs, update your test cases and run them once you add new test cases.

Testing is context-dependent ​

Testing is context-dependent, which means that you should test your software based on its needs, functionalities, and requirements.

Your test approach should depend on what your software does. Not every software needs the same type/method of testing because every application has its unique functionalities.

For instance, when testing an eCommerce web app, you will focus on its functionality to display products, so you will test how it shows products to end-users. When dealing with an API, you will focus on the response the API returns when an endpoint is called.

You wouldn't necessarily use the same test cases for both – that is what it means that testing is context-dependent.

The absence of errors is a fallacy ​

If you build software that is 99% bug-free, but it doesn't follow user requirements, it is not usable for end-users.

Know that it is very much necessary that your 99% bug-free software still meets or fulfills your user requirements. It is important to write test cases to find errors in the code, but you also need to test your software for your end-users (with them and how they'll use it in mind). The best way to do this is to carry out beta testing.

Unit Tests ​

Unit tests are very low level and close to the source of an application. They consist in testing individual methods and functions of the classes, components, or modules used by your software. Unit tests are generally quite cheap to automate and can run very quickly by a continuous integration server.

Unit testing is also the heart of an advanced software development process called test-driven development. In the test-driven dev process, DevOps professionals and developers write tests before the actual implementation. The goal is to have the specification of a single unit roll out before its realization.

Most languages have at least one unit testing framework recommended for itself (e.g. Java → JUnit, Python → PyUnit or PyTest, JavaScript → Mocha, Jest, Karma, etc.).

Integration Tests ​

We have learned that, in practice, the isolation property of unit tests may not be sufficient for some functions. In this case, one solution is to test how parts of the application work together as a whole. This approach is called integration testing.

The primary goal of integration testing is to ensure relationship integrity and flow of data between components or units.For example, it can be testing the interaction with the database or making sure that microservices work together as expected. These types of tests are more expensive to run as they require multiple parts of the application to be up and running.

For example, an integration test could use the connection to a database (a dependency in unit testing) to query and mutate the database as it usually would. You would need to prepare the database and read it out afterward correctly. DevOps often “mocks away” these external resources the way mocking is used in unit tests. This results in obscuring the failures caused by APIs beyond their control.

Integration testing helps find issues that are not obvious by examining the application’s or a specific unit’s implementation. Integration testing discovers defects in the interplay of several application parts. Sometimes, these defects can be challenging to track or reproduce.

Functional Tests ​

Functional tests focus on the business requirements of an application. They only verify the output of an action and do not check the intermediate states of the system when performing that action.

There is sometimes a confusion between integration tests and functional tests as they both require multiple components to interact with each other. The difference is that an integration test may simply verify that you can query the database while a functional test would expect to get a specific value from the database as defined by the product requirements.

End-to-end Tests ​

End-to-end testing replicates a user behavior with the software in a complete application environment. The primary goal of end-to-end testing is to ensure the entire application or system as a unit behaves how we expect it to, regardless of internal workings. It verifies that various user flows work as expected and can be as simple as loading a web page or logging in or much more complex scenarios verifying email notifications, online payments, etc...

In essence, unit and integration tests are typically “white box” (e.g. internals are known) whereas E2E tests are typically “black box” (e.g. we only verify input and output combinations). An example E2E test might be a generic user story like “Fetch a user’s data.” The input could be a simple GET request to a specific path, and then we verify that the output returned is what we expect. How the system fetched that data underneath is irrelevant.

End-to-end tests are very useful, but they're expensive to perform and can be hard to maintain when they're automated. It is recommended to have a few key end-to-end tests and rely more on lower level types of testing (unit and integration tests) to be able to quickly identify breaking changes.

For E2E tests, you typically use behavioral-based frameworks. You might use frameworks like Cucumber, Postman, SoapUI, Karate, Cypress, Katalon, etc. Note that a lot of API-testing frameworks are used for E2E testing because an API is typically how you programmatically interact with an app.

Acceptance Tests ​

Acceptance tests are formal tests that verify if a system satisfies business requirements. They require the entire application to be running while testing and focus on replicating user behaviors. But they can also go further and measure the performance of the system and reject changes if certain goals are not met.

It’s important to note that while acceptance tests can verify that the application behaves how a user wants it to, it does not verify the integrity of the system. Another caveat of user acceptance testing is there’s a limit to the corner cases and scenarios a person can come up with - this is why the previous automated testing methods are important since every single use case and scenario is codified.

Performance Tests ​

Performance tests evaluate how a system performs under a particular workload. These tests help to measure the reliability, speed, scalability, and responsiveness of an application. For instance, a performance test can observe response times when executing a high number of requests, or determine how a system behaves with a significant amount of data. It can determine if an application meets performance requirements, locate bottlenecks, measure stability during peak traffic, and more.

Smoke Tests ​

Smoke testing just refers to a smaller subset of checks to reasonably verify a system is working. Smoke tests are basic tests that check the basic functionality of an application. They are meant to be quick to execute, and their goal is to give you the assurance that the major features of your system are working as expected.

Usually smoke tests are run when users expect changes to not have made any significant impacts to overall logic and function. It can be expensive and time-consuming to run the full suite of all tests every single time, so smoke tests are used as an inexpensive safety measure that can be run more often.

Regression Tests ​

Regression testing is a testing method to verify if any previously-functional features have suddenly broken (or regressed).

This often includes running the entirety of all unit, integration, and system tests to ensure no functionality has changed unexpectedly. As we all know, sometimes software has the oddest way of breaking.

Regression tests are often time-consuming and can be very expensive, which is why sometimes people will run smoke tests instead, especially if recent changes are not logically expected to impact the whole system.

Often when people set up CI/CD, they will run smoke tests on almost every commit, whereas regression suites might run at set intervals or on large features to ensure continuous integration without issues.

Penetration Tests ​

Penetration testing (or pen testing) is a form of security testing that involves verifying the robustness of an application’s security.

Every manner in which an application can be compromised (cross-site scripting, unsanitized inputs, buffer overflow attacks, etc.) is exploited to check how the system handles it. Pen tests are an important part of making sure a company does not fall victim to serious breaches.

Load Tests ​

Load testing refers to testing an application’s response to increasing demand.

This includes testing sudden influxes of requests or users that might put unexpected strain on the system. Load testing is often done as a part of security tests to ensure an application and its system cannot be DDOS’d.

Load testing is also done to verify the maximum amount of data a system can handle at any given time. It’s integral for helping teams determine effective HA (high availability) implementation and scaling formulas.

White Box Tests ​

White box (also called structural or clear box) testing describes tests or methods in which the details and inner workings of the software being tested are known.

Since you know the functions, the methods, the classes, how they all work, and how they tie together, you’re generally better equipped to vet the logical integrity of the code.

For example, you might know there’s a quirk with the way a certain language handles certain operations. You could write specific tests for that, which you otherwise would not know to write in a black-box scenario.

Unit testing and integration testing are often white box.

Black Box Tests ​

In contrast, black box (also called functional, behavioral, or closed box) testing describes any tests or methods in which the details and inner workings of the software being tested are not known.

Since you don’t know any of the particulars, you can’t really create test cases that target specific niche scenarios or stress specific logic in the system.

The only thing you do know is that for a request or given piece of input, a certain behavior or output is expected. Hence, black box testing primarily tests the behavior of a system. End-to-end tests are often black box.

Gray Box Tests ​

Gray box testing is just a hybrid combination of black box and white box.

Gray box testing takes the ease and simplicity of black box testing (e.g. input → output) and targets specific code-related systems of white box testing.

The reason gray box testing exists is because black box testing and white box testing by themselves can miss important functionality.

Black box testing only tests that you get a certain output for a given input. It does not test the integrity of internal components — you could be getting the correct output purely by chance. White box testing focuses on the integrity of individual units and how they function together, but it is sometimes insufficient for finding system-wide or multi-component defects. By combining the two types together, gray box testing can encompass more complicated scenarios to really validate that an application is sound in structure and logic.

Conclusion ​

In conclusion, software testing is a crucial part of development. It can help save your team a lot of trouble, and it feels great to create a usable, bug-free product that users enjoy and recommend. A good testing suite should try to break your app and help understand its limit.

There is no one-size-fits-all rule for the level of test coverage that is appropriate for most organizations. Again, testing against five platforms is a good general guideline, but your testing needs will vary depending on how many platforms you are aiming to support, which type of app you develop, and other factors.

And finally, tests are code too! So don't forget them during code review as they might be the final gate to production.