For the sake of structure, we're going to describe them using an oversimplified view of company states:

startup → growing company → Plateau

Each state has its own list of testing practices that would be (in my opinion) the best fit to introduce during that company period.

There may be a reasonable question, so let’s do quick a Q/A:

Hey Mike! Your diagram is bolloks! Are you implying that you cannot do chaos testing as the company grows???

You can, but you still have to account for general company stability. Every small thing you add to the process will increase your uptime, but it will distract you from actually getting things done. The company stages are presented in order of decreasing need for agility.

During the startup stage, you have a time bomb on your table that will destroy the entire company if you do not become profitable in time.

During the growing stage, you have money. The probability that you will have to redesign your entire architecture is close to zero, so you can afford some reliability tools, but not all of them. The main problem is you are no longer invisible to potential competitors, and you have to conquer the market before industry leaders copy you and use their influence and connections to get a significant portion of your potential growth. Don’t worry, companies on the Plateau are painfully slow. That’s exactly why you cannot become one of them just yet.

So, let’s review all states and what's sufficient to do in each state.

There are only a few things you absolutely have to do at any company state:

Host Ping Tests

^{More like “host monitoring”, but it’s still called tests, so I guess it’s an honorable mention}

You periodically send a request to the host to prove the host is reachable.

Smoke Tests

You define the critical user journeys, which are: what actions the first-time user of your service should take to give you money! Write them down. Now, before (or after, I guess) production deployment, you should check if you can do that on your website.
Congratulations! You have smoke tests.

Manual (I hope it’s done by QA) Tests

^{Most probably done by the CTO or, even worse, the developer who made a feature}

At this stage, you are most probably improving your app with big chunks of work: new functionality or services introduced, added integrations, new pages on the website.

After you think you are done with the chunk, you just check if it works as expected. Check some border cases. Exploratory testing in its finest.

It is as far as it goes in a startup.

Money is short, but you can still lose it all. So you add a few checks here and there so you can at least provide a decent service on your side.

Automated Unit Testing

In the old days, it was one of the most tiring toils to do. You finished a task; now you mock up all external calls and write unit tests for all (or at least some of) new Service-type classes or functions.

Even then, IDEs had built-in functions or plugins that were able to generate this crap. Now, it’s just an AI job to do, with review and small tweaks here and there.

Integration Tests

^{Can be done manually if you like to spend a lot of money to spread human suffering.}

These tests check that your code works with the things it talks to. There are two kinds:

• Narrow: one piece of your code against one real dependency, usually in a throwaway container (Testcontainers). Examples are your queries against a real database and your consumer against a real message queue. They’re fast and stable.

• Broad: Basically a Component test + Contract testing in one bottle. Several of your services running together and calling each other. They catch wiring problems early, but they’re slow and flaky, and when one fails, it’s hard to tell which service broke. In the future, you replace them with component and contract tests.

System Tests

^{Can be done manually if you like to spend a lot of money to spread human suffering.}

The next logical step is to focus on the behavior of the whole system, rather than internal parts. You set up the full staging/pre-prod environment. External APIs still should be mocked, but now you can execute different scenarios of success and failure.

Can be complemented with Acceptance Tests.

Acceptance Tests

Technically speaking, it uses the same toolkit as system tests; the only difference is that we test based on expected business outcomes, not technical system behaviours. You will need to communicate more closely with Product Managers for this one to be valid.

If system tests answer the question: “are we building it the right way?”, the acceptance tests check “are we building the right thing?”.

This way, we have finished the most basic test pyramid.

DR (Disaster Recovery) Plan Tests

According to ISO 22301 and ISO/IEC 27031, there are 5 types of DR tests:

• Checklist Review: You review your DR plan documentation to keep it up-to-date.

• Tabletop Exercise: Stakeholders gather to verbally walk through a hypothetical disaster scenario step-by-step to expose gaps in role clarity, communication, and decision-making.

• Walkthrough / Simulation: Teams practice specific technical steps and handoffs in a sandbox environment to test runbooks and coordination without affecting production.

• Parallel Test: Secondary systems and data backups are brought online and validated alongside the live environment to ensure functionality without risking primary operations.

• Full-Scale / Full-Interruption Test: The primary production environment is completely shut down or isolated to shift live operations to the recovery site, proving end-to-end resilience and measuring exact recovery times.

It ensures the effectiveness and readiness of DR measures in the event of an actual disaster. You choose how far you are willing to go. The standard answer is Tabletop.

DiRT Testing

Most mature and expensive DR testing

Disaster Recovery Testing - Google’s yearly exercise in which real systems and teams are deliberately put through disaster scenarios. Disaster Recovery Testing Maturity Assessment (DiRMA) framework provides structured paths to enhance DiRT implementation across people, processes, and tools, helping to overcome cultural resistance and metric measurement issues.

Component test + Contract testing

You retire broad integration tests (the ones that need several of your services running together) and split what they checked into two types:

Component tests: one service tested through its API, with the services it calls replaced by stubs.

Contract tests: each side is checked against a shared contract separately, and the two services never talk to each other during the test. Pact is the usual tool.

Narrow integration tests stay. They check your code against real infrastructure, such as a database, a message queue or an identity provider. They catch failures no other test type reaches cheaply: a missing unique index, an offset committed before processing, a key rotation nobody handled. System tests confirm the pieces work together; narrow tests cover each piece’s edge cases in seconds.

Automated E2E Testing

^{Can be done manually if you like to spend a lot of money to spread human suffering.}

Good news! Your startup smoke tests (a user journey through the deployed system) already are manual E2E tests!

Now you automate most of these tests to speed the process up.
Ideally, you will do this so successfully that smoke tests become just a “check that the system is up and running“.

Performance Testing

Broad term to describe: Load testing, Stress Testing, and Soak Testing

Let’s describe all of them:

1. Load testing

We test that the system works well when many people use it at the same time. Guess the expected or peak load that the system will experience in production and simulate it using tools like LoadRunner.

1.1 Assured Capacity Load Testing

Validates whether your dedicated, reserved, or contracted infrastructure can actually sustain its maximum promised limits in a crisis.
You ask the cloud provider to do it as part of an audit before signing a big infrastructure contract.
Or, if you're not in the cloud, you test that your data centers can handle the assured load (especially the backup ones).

1.2 Real-World Traffic Load Testing

You can do this by:

• Using tools or scripts to simulate user traffic and behavior.
  ^{(Which is literally just standard Load testing! BUT, now you can add it into your performance review)}

• Capturing and replaying actual user traffic patterns.

• Shadow traffic. Copying real-time traffic from your busiest production environment and sending it to the test site. Works great, as long as you block side effects such as payments, emails, and writes.

2. Stress testing

Push the system beyond its expected load limits and see how it behaves.

3. Soak testing

Guess the expected or peak load of the system, simulate it, and let it run under it for a day. Helps to identify potential memory leaks, stability issues and performance degradation.

Regular BCP Testing (run from alternate site)

Imagine: you have an alternate site that you can run during the event of disaster, and it covers the most critical functions of business until the main site is up and running again (Business Continuity Plan).

BCP Testing is called to verify that the backup site is sufficient, properly configured, and equipped to support critical business functions in the event of disaster.

Rollbacks/Rollforwards Tested

^{Can be done manually if you like to spend a lot of money to spread human suffering.}

Tests whether the team can undo a release and then apply a corrected release safely.
In a test environment, deploy a change, return to the previous version, and then deploy the fixed version. Check that the service and its data are correct after each step.

Left Shift QA Testing (SDET)

^{SDET - Software Development Engineer in Test}

An approach in which developers and testers agree on what should happen, then add automated checks as the feature is built. A feature is not done until the tests for it are written and pass without an exception.

Left Shift Performance Testing

Run a small performance check while the feature is still being developed. Compare its speed and resource use with earlier results.

Chaos Testing / Failure Testing in Prod

You break things on purpose, one at a time, to check that the system keeps serving users and recovers on its own. The goal is to find weak spots before a real outage finds them for you.

Before you start, you need three things:

1. Infrastructure that can, in theory, survive failures: redundancy, timeouts, retries with backoff, circuit breakers and graceful degradation. Chaos testing checks resilience; it doesn’t create it.

2. Monitoring and alerting that define “normal”, such as checkout success rate and p99 latency. If you can’t measure normal, you can’t tell whether an experiment hurt anything.

3. A pre-production environment close to production. Every experiment runs there first.

Each experiment follows the same loop:

1. Define “normal”. For example, checkout success rate above 99.5%.

2. Predict what will happen. “If one payment-service instance dies, checkout success stays above 99.5%.”

3. Inject one realistic fault. Kill an instance, add 2 seconds of latency to a dependency, make an external API return errors, or delay or drop queue messages (more experiment ideas).

4. Limit the damage. Start in pre-production. In production, start with one instance or a small share of traffic, during working hours, with the team watching.

5. Stop automatically. Tie the experiment to an alarm, so it ends the moment the metric drops below normal.

6. Fix what broke and repeat. Experiments that pass become regular automated runs, so resilience doesn’t quietly decay.

What you’re really testing: timeouts, retries, circuit breakers, failover, autoscaling, and whether alerts fire and on-call knows what to do.

Tools:

• Netflix Chaos Monkey terminates instances at random. That proves you can lose any instance at any time, and that’s all it does. It also requires Spinnaker.

• AWS Fault Injection Service runs planned experiments with many fault types (latency, API errors, network disruption, CPU and memory stress) and has built-in stop conditions.

• Gremlin, Chaos Mesh and LitmusChaos do the same for other clouds and for Kubernetes.

Is chaos engineering the same as failure testing in production? They overlap. Failure injection is the technique; chaos engineering adds the prediction, the limited impact, and the automatic stop (Principles of Chaos Engineering). A test checks something you already expect. A chaos experiment looks for weaknesses nobody knew about.

Skip step 4 of the experiment, and you’re not doing chaos engineering. You’re just having an outage on purpose.

Fuzz Testing

You use a tool like RESTler to feed your service a bunch of unexpected, broken, or unusual input. Check that this service does not crash, freeze, or return business-harming results.

Cross-Service Transaction Testing

Test a task such as placing an order, which may update orders, stock, and payments. Check that all services agree on the result, including what happens if one step fails or is repeated.

Fin.

The main idea in short terms:
Incoming traffic to your service is what your service’s logic depends on to work correctly.
There are 2 ways of how information can flow into your service:
1 - by receiving a request from another service;
2 - by receiving a response from another service.

By avoiding destructive changes in the parts of the contract that are responsible for inbound information flow, we allow our service to receive information in both structures: with the new changes and without.

It’s a tool for API contract migration: by holding back destructive changes for a certain amount of time, sprints or releases, we are allowing our services to move in production independently.

The result is a reduced amount of release communication needed for rollout or rollback.

This is the most elegant and simple explanation of backwards compatibility I’m able to condense.

By enforcing backwards compatibility, we are effectively breaking a dependency between different services, allowing different teams to deliver API changes at their own pace, while also creating a window of time when rollback can be performed without breaking an API contract (in case of major incidents).

You can enforce backwards-compatibility on all of your projects (backwards-compatibility by default), or you can apply it only to the most critical and interconnected projects. Creating what is effectively a release-candidate group.

You test projects from the same release-candidate group together and deploy them together. This way you can decrease the number of moving pieces in prod. Which may be good enough for your case.

At the same time, backwards compatibility by default is an important prerequisite to the left-shift deployment self-service, where the team leader of a certain service is responsible for this service delivery.

Pick your poison based on resources and company needs.

That’s it! You now know everything about backwards-compatibility.

Fin.

P.S.:

Yet, there still can be a question of why we should concentrate specifically on incoming traffic only.

Task: You need to make sure that you provide teams with simple-to-understand backwards compatibility rules.

Well, let’s look into all possible options of backwards-compatibility handling:

1. You can make teams responsible for doing all communication in backwards backward-compatible manner (incoming and outgoing).
   Sounds great on paper; if you have a backwards-compatibility problem between 2 projects of your company, you will have 2 teams doing homework! And the possibility of an issue arising again is twice as likely!
   Until you realize that: In situations when more than 1 team is responsible for something, nobody can be held accountable for the contract breach.
   Also, it’s unnecessary overkill.

2. The second way is to make teams responsible for doing only outgoing communication in a backwards-compatible.
   Which is essentially delegating responsibility for our service's correct work to other teams’ shoulders.
   ”Our service failed because the other service did not do its homework”
   This approach does not remove project dependency; it only shifts responsibility onto someone else.

3. Teams care only about making their service work, no matter what changes are introduced. It’s simple to assign accountability for failure. And it will be fair.

There is a several approaches to naming a version, like hashing, date and time of release, or increment of numbers, but for some reason, semantic versioning is one of the most popular of them.

Let’s figure out what we can cook with it. But first here is a quick referance for the rules:

https://semver.org/

So, you make a company-wide policy so that engineers will now decide on the version naming.

Everyone happy! Now we have a reliable way of versioning things, right?

R-right?

Aaaannnnd… the engineer never actually followed any rules.

It’s simple, yet there is so much bias happening when you give people power to change the version.

• Architects want to reward themself for big migrations facilitated

• Devs want to celebrate big refactoring or big chunks of work done

• PMs want a major version after every epic.

There is nothing particularly bad or abnormal with people's desire to mark the big effort behind their work with a numbered version. But my points is, if you changed the whole service and no API was affected:

1. You did a great job designing APIs

2. It’s a patch change

The real-world semantic versions are ugly. And there is so much pursuit of beauty in engineering; following those rules just feels wrong, even counterintuitive.

You need to be prepared for the fact that a few years deep into the regular cadence of scheduled project releases, you will have 100+ major version changes in most interconnected and important BE services.

There is just so much professional bias happening, the faultiest, stupiddest implementation of the automatic semantic version assignment will do more reliable work than any engineer ever will.

So, let’s go back to SemVer rules.

The main idea is: will another service notice the difference if we rollback?

For example:
If another service adds a new field in their request and we have not yet introduced it in our API contract deserialization data structure, we can probably just ignore it and move on. But, getting fewer data fields than we require on the input will instantly trigger deserialization errors.

The same goes for the Response DTO; we cannot just “rollback“ the newly added field. Other services are relying on its existence! But bringing back something we freshly removed? Sure, no problems here.

We are not talking about how the internal logic of a service will react to each of these use cases; we are simply checking contract compatibility, like puzzle pieces.

I really like semantic versioning, but it only describes the public API.

If it’s not a library but a service in a structured system, I would also prefer to check the external API we use. Sooo… Let’s upgrade it a bit.

Let’s also add an answer to this question:
”What changes in our representation of the external API is backwards incompatible?”

So we can fully assess the possibility of rollback for this particular service without breaking other services or going into a downtime.

Frankly speaking, it’s the same rules for outbound traffic; they are just reversed.

Changes here have a pattern. The moment you see minor and major version changes on different sides of one API communication route, you know that backwards-incompatible changes were introduced here.

You know that without looking into the code or documentation at all!

You can also understand the direction of changes, because breach of a contract happens when a request moves from Minor → Major version change. Not sure what use you can make from that information.. But hey! To some degree, the surplus of information confirms the accuracy of the source.

So, if you try to release your services on some synchronized cycle, SemVer++ starts to describe the state of internal communication. It will also give a source of truth about service compatibility, which will help when you move towards a proactive reliability approach.

That’s how we use SemVer as a reliable tool!

Now forget about all that shit and just use Protocol Buffers.

Subscribe now

Share Reliable Engineer

This is the motto! Software release process is a tool!

To create a successful tool that will resolve company problems, you need 2 things:

1. High willingness from teams to engage with your process.

2. Grant to technical stakeholders the ability to control their part of the process.

To get the first point, you need to resolve at least one day-to-day pain from every team involved in the process:

1. Software Engineers and DevOps want to communicate with each other in their technical language while refusing to learn each other’s.

2. Product and Project managers want CLEAR ETA’s and observability over production delivery.

3. Tech support wants to know which tasks are delivered and which are not.

4. etc. etc.

For the second part, the rule of thumb is: everything that makes the process live on its own, without your major involvement, is good.

When it comes to line managers, the tooling will be complex, but if (or when) you need to involve higher and higher levels of management, the tools will become simpler and simpler.

If the line management controls will be “stop release process until blocker X is done“, the CTO level will be a big red button, “FUCK, stop everything“.

This is the way.

Subscribe now