Let’s talk about the different types of tests we may encounter in a software reliability/engineering management line of work.
For the sake of structure, we're going to describe them using an oversimplified view of company states:
startup → growing company → Plateau
Each state has its own list of testing practices that would be (in my opinion) the best fit to introduce during that company period.
There may be a reasonable question, so let’s do quick a Q/A:
Hey Mike! Your diagram is bolloks! Are you implying that you cannot do chaos testing as the company grows???
You can, but you still have to account for general company stability. Every small thing you add to the process will increase your uptime, but it will distract you from actually getting things done. The company stages are presented in order of decreasing need for agility.
During the startup stage, you have a time bomb on your table that will destroy the entire company if you do not become profitable in time.
During the growing stage, you have money. The probability that you will have to redesign your entire architecture is close to zero, so you can afford some reliability tools, but not all of them. The main problem is you are no longer invisible to potential competitors, and you have to conquer the market before industry leaders copy you and use their influence and connections to get a significant portion of your potential growth. Don’t worry, companies on the Plateau are painfully slow. That’s exactly why you cannot become one of them just yet.
So, let’s review all states and what's sufficient to do in each state.
There are only a few things you absolutely have to do at any company state:
Host Ping Tests
More like “host monitoring”, but it’s still called tests, so I guess it’s an honorable mention
You periodically send a request to the host to prove the host is reachable.
Smoke Tests
You define the critical user journeys, which are: what actions the first-time user of your service should take to give you money! Write them down. Now, before (or after, I guess) production deployment, you should check if you can do that on your website.
Congratulations! You have smoke tests.
Manual (I hope it’s done by QA) Tests
Most probably done by the CTO or, even worse, the developer who made a feature
At this stage, you are most probably improving your app with big chunks of work: new functionality or services introduced, added integrations, new pages on the website.
After you think you are done with the chunk, you just check if it works as expected. Check some border cases. Exploratory testing in its finest.
It is as far as it goes in a startup.
Money is short, but you can still lose it all. So you add a few checks here and there so you can at least provide a decent service on your side.
Automated Unit Testing
In the old days, it was one of the most tiring toils to do. You finished a task; now you mock up all external calls and write unit tests for all (or at least some of) new Service-type classes or functions.
Even then, IDEs had built-in functions or plugins that were able to generate this crap. Now, it’s just an AI job to do, with review and small tweaks here and there.
Integration Tests
Can be done manually if you like to spend a lot of money to spread human suffering.
These tests check that your code works with the things it talks to. There are two kinds:
Narrow: one piece of your code against one real dependency, usually in a throwaway container (Testcontainers). Examples are your queries against a real database and your consumer against a real message queue. They’re fast and stable.
Broad: Basically a Component test + Contract testing in one bottle. Several of your services running together and calling each other. They catch wiring problems early, but they’re slow and flaky, and when one fails, it’s hard to tell which service broke. In the future, you replace them with component and contract tests.
System Tests
Can be done manually if you like to spend a lot of money to spread human suffering.
The next logical step is to focus on the behavior of the whole system, rather than internal parts. You set up the full staging/pre-prod environment. External APIs still should be mocked, but now you can execute different scenarios of success and failure.
Can be complemented with Acceptance Tests.
Acceptance Tests
Technically speaking, it uses the same toolkit as system tests; the only difference is that we test based on expected business outcomes, not technical system behaviours. You will need to communicate more closely with Product Managers for this one to be valid.
If system tests answer the question: “are we building it the right way?”, the acceptance tests check “are we building the right thing?”.
This way, we have finished the most basic test pyramid.
DR (Disaster Recovery) Plan Tests
According to ISO 22301 and ISO/IEC 27031, there are 5 types of DR tests:
Checklist Review: You review your DR plan documentation to keep it up-to-date.
Tabletop Exercise: Stakeholders gather to verbally walk through a hypothetical disaster scenario step-by-step to expose gaps in role clarity, communication, and decision-making.
Walkthrough / Simulation: Teams practice specific technical steps and handoffs in a sandbox environment to test runbooks and coordination without affecting production.
Parallel Test: Secondary systems and data backups are brought online and validated alongside the live environment to ensure functionality without risking primary operations.
Full-Scale / Full-Interruption Test: The primary production environment is completely shut down or isolated to shift live operations to the recovery site, proving end-to-end resilience and measuring exact recovery times.
It ensures the effectiveness and readiness of DR measures in the event of an actual disaster. You choose how far you are willing to go. The standard answer is Tabletop.
DiRT Testing
Most mature and expensive DR testing
Disaster Recovery Testing - Google’s yearly exercise in which real systems and teams are deliberately put through disaster scenarios. Disaster Recovery Testing Maturity Assessment (DiRMA) framework provides structured paths to enhance DiRT implementation across people, processes, and tools, helping to overcome cultural resistance and metric measurement issues.
Component test + Contract testing
You retire broad integration tests (the ones that need several of your services running together) and split what they checked into two types:
Component tests: one service tested through its API, with the services it calls replaced by stubs.
Contract tests: each side is checked against a shared contract separately, and the two services never talk to each other during the test. Pact is the usual tool.
Narrow integration tests stay. They check your code against real infrastructure, such as a database, a message queue or an identity provider. They catch failures no other test type reaches cheaply: a missing unique index, an offset committed before processing, a key rotation nobody handled. System tests confirm the pieces work together; narrow tests cover each piece’s edge cases in seconds.
Automated E2E Testing
Can be done manually if you like to spend a lot of money to spread human suffering.
Good news! Your startup smoke tests (a user journey through the deployed system) already are manual E2E tests!
Now you automate most of these tests to speed the process up.
Ideally, you will do this so successfully that smoke tests become just a “check that the system is up and running“.
Performance Testing
Broad term to describe: Load testing, Stress Testing, and Soak Testing
Let’s describe all of them:
1. Load testing
We test that the system works well when many people use it at the same time. Guess the expected or peak load that the system will experience in production and simulate it using tools like LoadRunner.
1.1 Assured Capacity Load Testing
Validates whether your dedicated, reserved, or contracted infrastructure can actually sustain its maximum promised limits in a crisis.
You ask the cloud provider to do it as part of an audit before signing a big infrastructure contract.
Or, if you're not in the cloud, you test that your data centers can handle the assured load (especially the backup ones).
1.2 Real-World Traffic Load Testing
You can do this by:
Using tools or scripts to simulate user traffic and behavior.
(Which is literally just standard Load testing! BUT, now you can add it into your performance review)Capturing and replaying actual user traffic patterns.
Shadow traffic. Copying real-time traffic from your busiest production environment and sending it to the test site. Works great, as long as you block side effects such as payments, emails, and writes.
2. Stress testing
Push the system beyond its expected load limits and see how it behaves.
3. Soak testing
Guess the expected or peak load of the system, simulate it, and let it run under it for a day. Helps to identify potential memory leaks, stability issues and performance degradation.
Regular BCP Testing (run from alternate site)
Imagine: you have an alternate site that you can run during the event of disaster, and it covers the most critical functions of business until the main site is up and running again (Business Continuity Plan).
BCP Testing is called to verify that the backup site is sufficient, properly configured, and equipped to support critical business functions in the event of disaster.
Rollbacks/Rollforwards Tested
Can be done manually if you like to spend a lot of money to spread human suffering.
Tests whether the team can undo a release and then apply a corrected release safely.
In a test environment, deploy a change, return to the previous version, and then deploy the fixed version. Check that the service and its data are correct after each step.
Left Shift QA Testing (SDET)
SDET - Software Development Engineer in Test
An approach in which developers and testers agree on what should happen, then add automated checks as the feature is built. A feature is not done until the tests for it are written and pass without an exception.
Left Shift Performance Testing
Run a small performance check while the feature is still being developed. Compare its speed and resource use with earlier results.
Chaos Testing / Failure Testing in Prod
You break things on purpose, one at a time, to check that the system keeps serving users and recovers on its own. The goal is to find weak spots before a real outage finds them for you.
Before you start, you need three things:
Infrastructure that can, in theory, survive failures: redundancy, timeouts, retries with backoff, circuit breakers and graceful degradation. Chaos testing checks resilience; it doesn’t create it.
Monitoring and alerting that define “normal”, such as checkout success rate and p99 latency. If you can’t measure normal, you can’t tell whether an experiment hurt anything.
A pre-production environment close to production. Every experiment runs there first.
Each experiment follows the same loop:
Define “normal”. For example, checkout success rate above 99.5%.
Predict what will happen. “If one payment-service instance dies, checkout success stays above 99.5%.”
Inject one realistic fault. Kill an instance, add 2 seconds of latency to a dependency, make an external API return errors, or delay or drop queue messages (more experiment ideas).
Limit the damage. Start in pre-production. In production, start with one instance or a small share of traffic, during working hours, with the team watching.
Stop automatically. Tie the experiment to an alarm, so it ends the moment the metric drops below normal.
Fix what broke and repeat. Experiments that pass become regular automated runs, so resilience doesn’t quietly decay.
What you’re really testing: timeouts, retries, circuit breakers, failover, autoscaling, and whether alerts fire and on-call knows what to do.
Tools:
Netflix Chaos Monkey terminates instances at random. That proves you can lose any instance at any time, and that’s all it does. It also requires Spinnaker.
AWS Fault Injection Service runs planned experiments with many fault types (latency, API errors, network disruption, CPU and memory stress) and has built-in stop conditions.
Gremlin, Chaos Mesh and LitmusChaos do the same for other clouds and for Kubernetes.
Is chaos engineering the same as failure testing in production? They overlap. Failure injection is the technique; chaos engineering adds the prediction, the limited impact, and the automatic stop (Principles of Chaos Engineering). A test checks something you already expect. A chaos experiment looks for weaknesses nobody knew about.
Skip step 4 of the experiment, and you’re not doing chaos engineering. You’re just having an outage on purpose.
Fuzz Testing
You use a tool like RESTler to feed your service a bunch of unexpected, broken, or unusual input. Check that this service does not crash, freeze, or return business-harming results.
Cross-Service Transaction Testing
Test a task such as placing an order, which may update orders, stock, and payments. Check that all services agree on the result, including what happens if one step fails or is repeated.
Fin.