01
Severity: critical
Example: Data loss, a security hole or a core flow blocked for all users.
Approach: Stops the release; fixed or rolled back before anything else ships.
QA Consulting
QA consulting and test process audits
When releases keep breaking, adding more testers rarely fixes it. Agnotic Technologies audits how your team tests, from strategy and CI gates to flaky tests and defect triage, measures what escapes to production from your own data and hands you a prioritized roadmap.
Evidence from your own dataTest strategy and pyramidCI gates and flaky testsPrioritized roadmap
Trusted by global innovators
QA consulting
What is QA consulting, and when does a team need it?
QA consulting is an outside review of how a software team plans, runs and gates its testing, followed by a plan to fix what the review finds. Agnotic Technologies audits test strategy, automation balance, CI quality gates, environments, test data and defect handling, using your tracker and pipeline history as evidence.
The signal is usually the same: customers find defects first, releases get rolled back, or the regression suite is so slow and flaky that people stop trusting it.
This page is about assessing and setting up your QA process. If you need testers to run tests on your product, that is our QA and testing service; healthcare validation work sits with our healthcare quality assurance team.
Audit checklist
What does a QA audit check, and what are the warning signs?
The checklist we work through. Each area is scored from evidence, not from interviews alone.
| Area | Evidence we review | Warning sign |
|---|---|---|
| Test strategy and plan | Strategy document, test plans, product risk list | No written link between product risks and what gets tested |
| Pyramid balance | Test counts and run times by level | Most checks are slow end-to-end UI tests, with few unit or API tests |
| CI quality gates | Pipeline definitions and recent runs | Builds merge with failing or skipped tests, or gates are bypassed often |
| Flaky tests | Rerun history per test | Tests pass on rerun with no code change, and rerunning until green is normal |
| Environments | Environment list, booking and refresh records | Shared environments that stay broken or out of date for days |
| Test data | Data setup scripts and copies of production | Tests depend on hand-made data, or real personal data sits in test systems |
| Defect triage | Tracker workflow, severity and priority fields | Severity and priority treated as one field, and old defects never closed or rejected |
| Release criteria | Exit criteria, release checklists, rollback records | No agreed exit criteria, so the release date decides quality |
We measure defect leakage as defects found after release against all defects recorded for that release, from your own tracker. We do not compare you with industry averages.
How the audit runs
How does a QA audit run?
Three stages. You see findings as they appear, not only at the end.
1.
Collect evidence
Read access to your tracker, repositories and CI, plus short interviews with developers, testers and the release owner.
Evidence2.
Analyse and score
Each checklist area is scored from the evidence, with defect leakage and flaky test rates calculated from your own history.
Analysis3.
Report and roadmap
Findings ranked by risk and effort, and a roadmap that says what to fix first and who does it.
RoadmapDefect lifecycle
How should severity and priority work in a defect process?
Part of the QA process we set up for teams where every bug is urgent. Severity describes impact; priority is the order of work. They are separate fields.
02
Severity: major
Example: A key feature fails for some users and there is no workaround.
Approach: Usually high priority, but a product owner can defer it with a recorded reason.
03
Severity: minor
Example: A feature misbehaves and a workaround exists.
Approach: Prioritized with other work, and closed or rejected at an agreed review instead of left open.
04
Priority
Example: Set by the product owner, not by the tester.
Approach: Decides the order of fixes and can differ from severity, such as a minor typo on the pricing page fixed first.
QA vs QC
Is QA the same as QC in software?
No. Quality assurance builds confidence that quality requirements will be met, through process: strategy, standards and reviews of how the team works. Quality control checks whether a product meets them, through testing and inspection. Agnotic Technologies audits both, because a team can test thoroughly and still have no process that stops the same defects coming back.
| Quality assurance: the process | Quality control: the product |
|---|---|
| Writing the test strategy that links product risks to tests | Running the tests the strategy calls for |
| Agreeing a definition of done for each story | Checking each story against it before it closes |
| Setting code review standards | Reviewing a pull request against those standards |
| Auditing how the team plans, tests and releases | Inspecting a release candidate before it ships |
| Designing the defect lifecycle and triage rules | Logging and triaging the defects testing finds |
| Writing release criteria | Running release checks and signing off against them |
The ISTQB Glossary describes quality control as activities designed to evaluate the quality of a component or system. QA consulting works on the left column; our QA and testing service does the right.
Deliverables
What does the QA audit report contain?
Everything is written so your team can act on it without us.
- A score and the evidence for each checklist area
- Defect leakage, flaky test rate and pipeline run times from your own data
- A test strategy and plan outline, with entry and exit criteria for each stage
- A defect lifecycle with separate severity and priority, and triage rules
- Quick fixes your team can make in the next sprint
- A roadmap of larger changes, ranked by risk reduced and effort
Who it is for
Who asks us for a QA audit?
Teams that already test, but do not trust the result.
Teams with rising escaped defects
Customers find bugs the test suite should have caught.
Teams with slow, flaky pipelines
The regression suite takes too long and fails at random, so people rerun or skip it.
Products growing past one team
Several teams need a shared test strategy, shared gates and one defect process.
New QA leads
A newly hired QA lead wants an independent baseline and a plan to present.
Teams moving to continuous delivery
More frequent releases need automated gates that people trust.
After the audit
Who carries out the QA roadmap?
The audit is useful on its own, and you choose who does the work after it.
Your team
We hand over the roadmap and stay available for questions and a follow-up review.
Agnotic engineers beside your team
We fix CI gates, quarantine flaky tests and set up test data while your team keeps shipping.
Agnotic QA and testing services
For test execution or automation build, our QA and testing team takes on the work.
A follow-up audit
A second review measures the same indicators again, so you see what changed.
References
Which standards and measures do we refer to?
Published references behind the checklist and the measures.
ISO/IEC/IEEE 29119NIST SSDFDORA metricsTest pyramid
Facts and sources
Facts behind QA process measures
Delivery measures and testing practices, each linked to its primary source.
DORA measures software delivery instability with change fail rate, the ratio of deployments that need immediate intervention after release, such as a rollback or a hotfix.
Source: DORA: software delivery performance metricsDORA's deployment rework rate is the ratio of deployments that are unplanned and happen because of an incident in production.
Source: DORA: software delivery performance metricsDORA's failed deployment recovery time measures how long it takes to recover from a deployment that fails and needs immediate intervention.
Source: DORA: software delivery performance metricsNIST's Secure Software Development Framework task PW.8.2 calls for scoping and designing tests, performing them and documenting the results, including recording and triaging every issue found in the team's workflow or issue tracking system.
Source: NIST SP 800-218, SSDF version 1.1 (February 2022)The ISTQB Glossary defines quality assurance as activities focused on providing confidence that quality requirements will be fulfilled, a definition it takes from the ISO/IEC/IEEE 24765 vocabulary.
Source: ISTQB Glossary, quality assurance
Frequently Asked Questions
We do not publish prices. The audit is quoted as fixed work once we know how many teams, products and repositories it covers. Follow-on work depends on who carries out the roadmap.
Read access to the issue tracker, code repositories and CI system, plus short interviews with a few developers, testers and whoever owns releases. We do not need production data.
Defects found after release divided by all defects recorded against that release, taken from your tracker for recent releases. If your tracker does not record where a defect was found, adding that field is the first fix.
Severity is the defect's impact on users or data. Priority is the order in which it gets fixed, set by the product owner. Mixing them makes every defect urgent, so we keep them as separate fields.
Either. Many teams take the roadmap in-house; others ask our engineers to fix the pipeline, flaky tests and test data first while their own team keeps delivering.
Yes. QA consulting for startups is lighter: a test strategy that fits a small team, unit and API tests in CI, a few end-to-end checks on the paths that earn money and a simple defect workflow.
No. We compare your team with its own history and its own goals. Published averages use different methods and rarely match your product's risk.