Delivery and ordering
- The same event delivered twice
- Events for one key arriving out of order
- A consumer restarting halfway through a batch
Microservices Testing
Microservices testing services
Once a product is split into services, most failures happen between them: a renamed field, a retried event, a timeout nobody set. Agnotic Technologies builds the contract, component, event and resilience tests that catch them before release, in each service team's own pipeline.
Consumer-driven contractsComponent tests with stubsEvent and retry testsFault injection
Trusted by global innovators
Software testing for microservices
What is microservices testing?
Microservices testing is checking that independently deployed services still work together: contracts between services, component tests with dependencies stubbed, event and retry handling, and behavior when a dependency fails. Agnotic Technologies builds this testing for the client and places each test in the pipeline of the team that owns the service.
What changes after a split is where failures hide. A monolith fails inside one process you can debug. A set of services fails between them, in a call, a message or a timeout.
This page covers the tests that exist only because the product is split into services. General test layers and framework choice are on our test automation page, and load tests across a chain of services are on the performance testing page.
Test placement table
Which test proves what, and which pipeline does it block?
The placement we agree with each service team. A test that blocks the wrong pipeline either slows every team down or catches nothing.
| Test type | What it proves | Who owns it | Pipeline it blocks |
|---|---|---|---|
| Unit | The service's own rules and calculations are right | The service team | Service build, on every commit |
| Component with stubs | The whole service works through its real API, with its own database real and other services stubbed or virtualized | The service team | Service build, before merge |
| Consumer contract | The calls and messages this service depends on are written down as a contract the provider can check | The consuming team | Consumer build; the contract is published with its version |
| Provider verification | The provider still meets every published consumer contract | The providing team | Provider build; a broken contract stops the merge |
| Messaging and events | Events follow the agreed schema, and consumers handle duplicates, reordering and poison messages | Producer and consumer teams | Both service builds, against a real broker in a container |
| Short-lived environment smoke test | The changed service starts, connects and serves a few key calls beside its real neighbours | The service team, on a shared platform | Release of that service |
| Resilience and fault injection | Timeouts, retries, circuit breakers and fallbacks behave as designed when a dependency is slow or down | Service team with the platform team | Release, on a schedule and before risky changes |
| End-to-end journey | A few journeys that earn money or keep users safe still work across all services | A small cross-team group | Release train or nightly; kept short on purpose |
Most confidence comes from the first five rows, which run without deploying the whole system. End-to-end journeys stay few, because each one depends on every service being up.
Asynchronous messaging
How do we test events, retries and dead-letter queues?
Message-based integration fails in ways a request and response never does. These are the checks we write for each producer and consumer.
Idempotency
- A repeated event produces one effect, not two
- Idempotency keys stored and checked before any side effect
- Payments, emails and stock changes tested with duplicates
Retries
- Retry count and backoff match the design
- Retries stop on errors that will never succeed
- A retry storm does not overload a recovering service
Dead-letter queues
- Poison messages land in the dead-letter queue with the reason
- An alert fires when the queue grows
- Messages can be inspected and replayed safely
Event schemas
- New fields added without breaking older consumers
- Removed or renamed fields caught before release
- Schema versions checked in both pipelines
Do you need a full copy of every service to test one change?
Usually not. For most changes we deploy the changed service and its direct dependencies into a short-lived environment, created for the pull request and removed when it closes, and stub or share everything further away.
On Kubernetes this is often one namespace per change, built from the same manifests or Helm charts as production, with seeded test data and an expiry so forgotten environments do not pile up.
A shared staging environment still has a place for release rehearsals. It just stops being the only place a change can be tested, which is what usually leaves teams queuing for their turn.
Fault injection
Which failures do we inject, and what should happen?
Each fault runs against a short-lived or staging environment. The expected behavior is written down before the test, so the result is a pass or a fail, not an opinion.
01
A dependency responds slowly
Example: The pricing service takes far longer than usual to answer
Approach: The caller times out at its configured limit, returns a clear error or a fallback, and does not hold threads or connections.
02
A dependency returns errors
Example: The inventory service returns server errors for a period
Approach: Retries stay within their budget, then the circuit breaker opens and calls fail fast until the dependency recovers.
03
A dependency is unreachable
Example: A service is scaled to zero or its network is blocked
Approach: Features that depend on it degrade as designed, and the rest of the product keeps working.
04
The message broker is unavailable
Example: Producers cannot publish events for several minutes
Approach: Events are kept in an outbox or buffer and published once the broker returns, with none lost or doubled.
05
A consumer falls behind
Example: A backlog builds after a slow deployment
Approach: Lag shows on dashboards and alerts, and the consumer catches up without breaking ordering per key.
06
A bad version is deployed
Example: A new release starts failing health checks
Approach: The rollout stops and rolls back, and the contract results show which change caused it.
Diagnosis
How do traces show which service failed a test?
When a test that crosses services fails, the first question is where. We wire tests into your tracing so the answer is one click away.
- The test runner starts a trace for each test and passes the trace context on every call it makes
- Each failed test report links to its trace, showing every service the request touched and where it broke
- Producer and consumer spans connect an event to the work it triggered, so asynchronous failures are traced too
- Logs from every service carry the trace ID, so they can be read in order for one test
- Test traffic is tagged, so it can be filtered out of production dashboards and alerts
- Slow spans seen in tests are logged as findings, even when the test itself passed
Contract workflow
How does a contract change reach production safely?
The order we set up between a consumer team and a provider team, so neither has to deploy the other's service to test a change.
The consumer records what it needs
The consumer's tests describe the requests or messages it sends and the parts of each response it reads, and a contract is generated from them.
The contract is published with a version
Contracts go to a shared broker, tagged with the consumer version and the environments that version runs in.
The provider verifies every contract
Each provider build replays the published contracts against the real provider code and reports which consumers it satisfies.
A deployment check runs before release
Before a service is deployed, the pipeline asks whether this version is compatible with the versions already running in the target environment.
Breaking changes are planned, not discovered
A provider that needs to remove a field adds the new one first, waits until no deployed consumer reads the old one, then removes it.
Failure points
Why do microservices test suites stop protecting releases?
Patterns we find when we review service testing that has grown without a plan.
Challenge
Every change waits for one shared staging environment.
Agnotic approach
Short-lived environments per change, with staging kept for release rehearsals.
Challenge
Hand-written mocks drift away from how the real provider behaves.
Agnotic approach
Consumer-driven contracts, so a stub is only trusted while the provider still passes it.
Challenge
Contract tests check the provider's business rules and break on every change.
Agnotic approach
Contracts cover only the shape and meaning of what the consumer reads; business rules stay in the provider's own tests.
Challenge
Event tests wait with fixed sleeps and fail at random.
Agnotic approach
Tests poll for the expected outcome with a time limit, or read the consumer's output directly.
Challenge
Retries were never tested, so a retried message charged a customer twice.
Agnotic approach
Duplicate delivery tests for every consumer that causes a side effect.
Challenge
A failing end-to-end test takes hours to trace to one service.
Agnotic approach
Trace context in every test, linked from the failure report.
Starting points
Which of these sounds like your team?
Situations that bring engineering leaders to us, and where we usually start.
Just split from a monolith
The old end-to-end suite still runs against everything and nobody trusts it. We start with contracts at the busiest service boundaries.
Releases blocked by shared staging
Teams queue for one environment. We start with component tests and short-lived environments for the services that change most.
An outage from a changed API field
A provider change broke a consumer in production. We start with consumer contracts for that provider and a deployment check.
Duplicate side effects from events
A retried message sent two emails or two payments. We start with idempotency and duplicate delivery tests on those consumers.
Microservices testing for startups
A handful of services and one team. We keep it small: component tests, a few contracts and one short end-to-end journey.
A move to Kubernetes
The platform changes under every service. We add resilience tests and per-change environments as part of the move.
Scope
What sizes a microservices testing engagement?
We do not publish prices. Microservices testing cost depends on these drivers, which we size on the scoping call.
Services and boundaries in scope
How many services, and how many consumer and provider pairs need contracts.
Messaging
Which brokers and event flows are involved, and how many consumers cause side effects.
Environment platform
Whether your platform can already create an environment per change, or that has to be built first.
Existing tests
Which unit, component and end-to-end tests exist, and which ones the teams trust.
Tracing
Whether services already emit traces, or instrumentation comes first.
Pipelines and owners
How many service pipelines need new gates, and which team owns each one.
Facts and sources
What do Pact, NIST, OpenTelemetry and Kubernetes say?
Short facts from the projects and standards behind this testing approach, each linked to its primary source.
Pact describes contract testing as checking each application in isolation to confirm the messages it sends or receives match a shared, documented contract, and Pact generates that contract while the consumer's automated tests run.
Source: Pact documentation: Introduction (Pact Foundation)NIST SP 800-204 lists the core features a microservices system needs, including service discovery, security monitoring, availability and resiliency techniques such as circuit breakers, and load balancing and throttling.
Source: NIST SP 800-204, Security Strategies for Microservices-based Application SystemsIn OpenTelemetry, a trace is the path of a request through an application, and a producer span marks a job that may be processed later; the parent of a consumer span is always a producer span.
Source: OpenTelemetry documentation: Traces (CNCF project)Kubernetes namespaces isolate groups of resources within a single cluster, and resource names only need to be unique within a namespace, not across namespaces.
Source: Kubernetes documentation: Namespaces
References
Which specifications do our service tests rely on?
The specifications and guidance behind the contracts, traces and environments on this page.
PactNIST SP 800-204OpenTelemetryW3C Trace ContextOpenAPIAsyncAPI
Frequently Asked Questions
An integration test runs two or more real services together. A contract test checks each side on its own against a shared contract, so a consumer and a provider are tested in their own pipelines without deploying each other. We use both, with far fewer integration tests.
They replace most of them. Contracts catch broken calls and messages between services early and cheaply. A short set of end-to-end journeys stays, to prove the most important paths work with everything deployed.
It depends on the number of services and on whether your platform can already create environments per change. We agree a first scope on the scoping call, usually the busiest service boundaries, and plan the rest from what that first work shows.
Yes. We add stages to your existing CI system for each service, publish contracts to a broker you host or choose, and reuse your deployment manifests for short-lived environments. We do not replace a pipeline that works.
Yes. Contract and event tests work across languages, because the contract describes the HTTP calls or messages, not the code behind them. Each team writes its side of the test in its own language.
Yes. A common first phase is one consumer and provider pair with contracts and a deployment check, plus duplicate delivery tests for one event consumer. It proves the pattern on your code before other teams adopt it.
AI assistants may draft stubs and test data from your API descriptions. An engineer reviews every generated test, and a test only gates a pipeline after it has been seen to fail against a deliberately broken build.