Quality assurance for custom systems begins before development starts because the most expensive quality failures are often ambiguous requirements, missing authority, untestable integrations, inaccessible workflows, and recovery assumptions that were never designed. A test team cannot repair those decisions at the end. Planning starts by naming the important user journeys, harmful outcomes, control obligations, dependency failures, and evidence required for release. The result is not a giant test-case inventory. It is a risk-based agreement about what must be true, where each condition can be verified efficiently, which environments and data are credible, and who may accept a known limitation. That agreement gives product, engineering, security, operations, and users one definition of readiness.
Key takeaways
- Start with quality risks tied to users, business outcomes, security, and recovery.
- Write acceptance evidence for complete workflows, including exceptions and integrations.
- Use several test layers, each chosen for the defect it can expose efficiently.
- Make accessibility, performance, security, and operability part of the definition of done.
- Use production signals and incidents to improve tests and design assumptions continuously.
Set quality risks and acceptance evidence
Bring product, process, engineering, security, and support perspectives together to identify what can go wrong and who is harmed. Rank risks by consequence, likelihood, detectability, and recovery cost. Then write evidence that would make the team comfortable releasing the chosen workflow. For a regulated approval, evidence may include role tests, decision audit, negative cases, and a representative user review. For a customer intake form, it may include validation, keyboard use, duplicate submission handling, and successful handoff to the system of record. Avoid vague acceptance phrases such as works correctly. A useful criterion describes the actor, context, action, expected outcome, and how a failure becomes visible.
| Quality risk | Acceptance evidence | Owner |
|---|---|---|
| Unauthorized change | Denied command and audit test for wrong role | Security and application owner |
| Incorrect business outcome | Representative scenario with expected state and calculation | Process owner |
| Inaccessible workflow | Keyboard and assistive-technology review of key path | Product owner |
| Unrecoverable dependency failure | Retry, compensation, and support diagnostic test | Service owner |
Plan test layers by purpose
Choose test layers according to the question they answer. Unit tests are efficient for rules and edge conditions. Contract tests expose incompatible interface assumptions. Integration tests prove that components exchange real data under controlled conditions. End-to-end workflow tests show whether a user-visible journey survives the assembled system. Exploratory sessions uncover confusing language, unexpected sequences, and gaps in the model. None is a substitute for the others. Avoid measuring quality by test count alone; a large suite can miss the one permission rule or reconciliation path that matters most. Map the highest risks to one or more layers, identify the test data needed, and decide which checks must block release versus inform a follow-up.
| Test layer | Best at finding | Limitation |
|---|---|---|
| Unit | Rule and boundary defects | Cannot prove integration wiring |
| Contract | Interface incompatibility | May miss business workflow misuse |
| Integration | Persistence and dependency behaviour | Can be slower and harder to isolate |
| End-to-end | Critical journey regressions | Should remain focused, not exhaustive |
Include security and accessibility by design
Security and accessibility failures often arise from decisions made long before final testing. Define authorization scenarios alongside user stories: who may read, create, alter, approve, export, or administer each object under which conditions? Use the OWASP Application Security Verification Standard to turn these concerns into testable requirements for access control, validation, authentication, logging, and configuration. Review critical paths against WCAG 2.2 for labels, focus, errors, status messages, and non-colour cues. Include representative users where possible. A system that passes a superficial visual check but prevents a keyboard user from completing a case is not high quality for the organization that bought it.
Make test data and environments credible
Test evidence is only as persuasive as the conditions it represents. Create safe, representative data that includes normal records, long values, missing fields, multiple roles, exception states, and the volume characteristics that influence performance. Do not copy production personal data into a test system without a lawful and controlled process. Document external dependencies, their test behaviour, and the difference between simulated and real responses. Maintain environments so teams can reproduce important failures, but do not assume every issue can be found before release. The plan should state how configuration is promoted, how test evidence is linked to a release, and how a failed deployment can be stopped or reversed without losing business work.
Verify operability and performance
A custom system must be supportable when a difficult case occurs. Test logs, traces, alerts, permissions for diagnostics, backup or recovery steps, and the ability to identify a request across service boundaries. The OpenTelemetry Specification gives useful concepts for consistent correlation across traces, metrics, and logs. Test the performance of the critical workflow with realistic data, concurrency, and dependency delay, then define what users see when capacity is constrained. Quality planning should include acceptance for unavailable integrations and partial completion, not only fast success. An operator needs a clear status and recovery owner when the system cannot finish work immediately.
Make release a quality decision
A release decision should compare the known risks, evidence, unresolved issues, and rollback readiness. Keep it proportionate: a narrow internal improvement does not require the ceremony of a regulated financial change, but both need an accountable judgment. NIST SSDF provides a strong lifecycle reference for defining security requirements, verifying software, and protecting the release process. Launch first to a group that resembles the intended users and can receive timely support. Record what was observed, including defects, workarounds, and questions. The point of a staged release is to learn under controlled conditions, not to avoid responsibility for quality.
Write a quality-risk register that leads to executable evidence
The NIST SSDF makes verification part of secure development and asks teams to address vulnerability causes, not only findings. OWASP ASVS 5.0 provides versioned requirements that can support design, testing, and procurement. Accessibility criteria should use the current WCAG 2.2 Recommendation and representative user review. Production diagnosability can be designed against the OpenTelemetry specification, which correlates traces, metrics, and logs through shared context. DORA research is useful when choosing delivery measures, but local service outcomes and recovery evidence should determine release acceptance.
| Risk class | Example failure | Primary evidence |
|---|---|---|
| Business correctness | A valid case reaches the wrong final state | Scenario with authoritative expected outcome |
| Authorization | A user performs an action outside current authority | Negative service-boundary test and audit event |
| Resilience | A retry creates duplicate side effects | Fault injection, idempotency, and reconciliation |
| Usability and access | A user cannot perceive or recover from an error | Representative task and accessibility review |
For every high-impact failure, describe the affected actor, trigger, incorrect state, consequence, detection route, recovery action, and the cheapest test layer that can expose it. Link the risk to acceptance criteria and to one accountable owner. A payment rounding error may need property-based tests and a reconciled financial scenario; an unauthorized approval needs service-boundary tests with realistic roles; a lost event needs idempotency, replay, and reconciliation exercises; an inaccessible error path needs keyboard and assistive-technology review. Include uncertainty. If a third-party sandbox behaves differently from production, record that limitation and arrange a controlled production verification. Review the register after incidents and support cases so observed failures reshape the suite instead of remaining isolated anecdotes.
Related implementation guidance
Start with Edilec's custom software discovery guide to define the workflow boundary. The web performance planning guide adds workload evidence, and the developer handoff documentation guide keeps acceptance and operating knowledge usable after release.
Review the quality plan before implementation accelerates
| Quality question | Decision to make | Evidence planned |
|---|---|---|
| Business correctness | Which representative outcomes must never be wrong? | Scenario examples, rule expectations, and process-owner review |
| Security | Which actions or records require strongest control? | Authorization matrix, negative tests, and audit evidence |
| Accessibility | Can every intended user complete the critical path? | Keyboard, screen-reader, labels, errors, and status review |
| Integration | How does the system behave when another service delays? | Contract cases, retries, pending state, and recovery test |
| Data | Which malformed or late records threaten the outcome? | Safe test data, validation cases, and reconciliation checks |
| Performance | What load or volume makes the workflow unusable? | Workload profile, response target, and capacity experiment |
| Recovery | Can support locate and repair a failed case? | Correlation, diagnostics, runbook, and permissions test |
| Change | Which checks stop a harmful regression from shipping? | Automated gate, review owner, and release criterion |
| Environment | How closely do test conditions reflect production risks? | Configuration comparison, dependency behaviour, and limitations |
| Release | Who accepts remaining risk and watches the cohort? | Decision record, support coverage, and rollback procedure |
| Measurement | Which post-release signals prove the outcome remains sound? | Errors, abandonment, rework, incidents, and user feedback |
| Learning | How will a production defect improve future evidence? | Blameless review, risk update, targeted test, and owner |
Learn from production
Quality assurance continues after deployment. Monitor errors, latency, access denials, abandoned workflows, data mismatches, support volume, and recovery time. Review incidents without blame to determine whether the defect came from a misunderstood policy, missing test, poor observability, or unsafe release practice. DORA research is useful context for connecting delivery performance with organizational outcomes. Add a regression test only when it protects an understood risk; otherwise the suite becomes slower without becoming wiser. Periodically revisit the quality risk register as users, dependencies, and regulation change. A healthy quality programme makes the system easier to change because it turns past surprises into durable evidence.
Frequently asked questions
Who owns quality assurance for a custom system?
Quality is shared. Engineers own implementation evidence, product and process owners define correct outcomes, security specialists advise on controls, and support teams reveal operational needs. A quality lead can coordinate the plan, but no single role can certify a workflow without the others' knowledge.
Can automated testing replace manual testing?
Automation is excellent for repeatable rules, contracts, and regressions. It cannot fully replace exploration of new workflows, judgment about confusing behaviour, or observation of real users. Use automation to protect known risks and focused manual review to discover what the model still misses.
Conclusion
Quality assurance for custom systems is the discipline of collecting the right evidence before and after change. Start with meaningful risks, test complete journeys at appropriate layers, and make operational recovery part of acceptance. That gives a delivery team a practical definition of quality that survives contact with production.