← All insights

Whether You Use BMAD, Spec Kit, or Just Vibes, your AI-Generated Tests Have the Same Problem

Portia Canlas, FDE8 min readMethodology

Why test architecture is orthogonal to implementation methodology and what to do about it

The problem does not care how you wrote the code

There is a moment every engineering team using AI-assisted development eventually reaches. The CI pipeline starts failing intermittently. Tests that passed yesterday fail today on the same code. A refactor breaks fourteen tests simultaneously not because the feature broke, but because the tests were written against implementation details that changed. Coverage percentages look healthy but the team has no confidence in what the tests actually verify.

This moment arrives regardless of how the code was written. It does not matter whether your team uses BMAD's structured planning pipeline, Spec Kit's spec-driven development, GitHub Copilot's inline suggestions, or pure vibe coding prompting an AI agent and accepting what it produces. The failure mode is the same because it originates not in the implementation methodology but in how AI agents generate tests.

AI agents generate tests that are plausible, syntactically correct, and immediately passing. They are also, without intervention, systematically fragile. Understanding why is the starting point for understanding what TEA does and why it works independently of whatever implementation approach you are currently using.

Why AI agents generate fragile tests by default

An AI agent generating tests faces a fundamental constraint: it can only assert what it can observe. It observes the DOM, the rendered HTML, the class names, the element structure. It does not observe the engineering intent behind the design, the business rules that govern the feature, or the acceptance criteria the story was built against.

This constraint produces four consistent failure patterns regardless of the implementation methodology that preceded the test generation:

Brittle selectors. The agent chooses the most obvious locator, typically a CSS class or data attribute because that is what is visible in the DOM. page.locator('.property-card-submit') works today. After a CSS refactor it does not. The feature has not changed. The test has broken. The agent had no way to know that page.getByRole('button', { name: 'Submit' }) was the resilient choice because the semantic intent was not in the DOM it was reading.

Missing network assertions. For features that load data via API, the agent tests what appears on screen. If the API returns a 500 and the UI renders cached stale data, the test passes. The feature is broken. The test said it was fine. The agent did not know how to intercept the network call and assert the response because the network layer was not in the specification it was given.

Positive path only. The agent generates tests for the scenario described in the prompt. Nobody described the empty state, the validation error, the permission denied response, or the timeout behaviour. Those tests do not exist. The coverage percentage is high. The confidence is misplaced.

Non-deterministic timing. When an action triggers an asynchronous event, the agent inserts a wait. The wait is a fixed timeout await page.waitForTimeout(2000) because that is the simplest pattern. On a fast machine it passes. In CI under load it fails. The test suite becomes unreliable at exactly the moments when reliability matters.

These four patterns are not bugs in any particular implementation methodology. They are the default output of AI test generation without architectural guidance. BMAD teams see them. Spec Kit teams see them. Vibe coding teams see them at significantly higher frequency because there is less planning context constraining the agent. The tool is different. The failure mode is the same.

What TEA is and what it is not

TEA (Test Engineering Architect) is a test architecture module built as part of the BMAD ecosystem. The name and origin might suggest it only works within a BMAD workflow. That is a misreading of what TEA actually does.

TEA is a quality operating model encoded as executable workflows. It does not care how the implementation code was written. It cares about what the tests assert, whether the assertions are resilient, whether the coverage maps to requirements, and whether the release gate decision is backed by evidence.

The official TEA documentation confirms this explicitly: TEA can be used in "TEA Solo (Standalone)" mode brought to a non-BMAD project with your own requirements, specs, system-of-record pointers, or analysable source tree. The implementation methodology is irrelevant to TEA's operation.

What TEA requires is:

  • Tests to review (it audits what exists)
  • Acceptance criteria or requirements to map to (it traces coverage against these)
  • A codebase to analyse (it understands the system it is testing)

What TEA does not require:

  • BMAD planning artifacts
  • Spec Kit specifications
  • Any particular implementation methodology
  • Any particular CI/CD platform

TEA on a BMAD project

For teams using BMAD, TEA integrates at three points in the delivery lifecycle.

Before implementation begins, bmad-testarch-test-design analyses the PRD, architecture, and architectural decision records to produce a risk-based test plan with P0–P3 scenario prioritisation and NFR thresholds. bmad-testarch-atdd then generates failing test scaffolds from the acceptance criteria in the story file. The implementation agent has a verification mechanism before it writes a single line of code.

After implementation, bmad-testarch-automate expands coverage with Playwright specs and Page Object Models. bmad-testarch-test-review scores the generated tests 0–100 across Determinism, Isolation, Maintainability, and Performance.

At the release gate, bmad-testarch-trace maps tests to requirements and produces a Go/No Go gate decision backed by a traceability matrix.

The integration is deep because the planning artifacts BMAD produces story files, acceptance criteria, sprint-status.yaml give TEA rich context to work with. The quality that results is higher than what TEA can achieve on a project with no planning artifacts.

TEA on a Spec Kit project

For teams using Spec Kit, the integration is lighter but equally valid.

Spec Kit's spec.md contains the Why, Capabilities, Constraints, and Success signal for the change. This is precisely the structured specification TEA needs to generate ATDD scaffolds and map test coverage. TEA reads the spec, identifies the acceptance criteria, generates scaffolds before implementation begins, and maps coverage after.

test-review and trace operate the same way they do on a BMAD project, reviewing test files and making gate decisions because they do not depend on how those test files were produced.

The main difference from a BMAD integration is that Spec Kit projects typically lack system-level test design artifacts (the risk assessment and NFR planning that bmad-testarch-test-design produces in Phase 3). This means TEA's risk-based prioritisation is less informed. The team may want to run a lightweight test-design pass at epic level before implementation begins to identify the highest-risk scenarios.

TEA on a vibe coding project

This is where TEA's value is highest and where it is least likely to be considered, because teams that vibe code tend not to think architecturally about testing.

Vibe coding means prompting an AI agent with intent and accepting the output. The implementation is fast. The test generation is automatic. The coverage percentage looks fine. The problems accumulate invisibly.

For a vibe coding team, the typical state of the test suite after a few months of productive development is:

  • High coverage percentage, low coverage confidence
  • Hundreds of tests passing in local but failing intermittently in CI
  • No mapping between tests and the features they were supposed to verify
  • No gate mechanism, code ships when the team feels good about it

TEA addresses all four of these without requiring the team to adopt a planning methodology. The entry points are:

Immediate (TEA Lite): Run bmad-testarch-automate on existing features to generate structured Playwright tests with proper fixture architecture and accessible locators. Run bmad-testarch-test-review on existing test files to get an objective quality score and a list of specific issues to fix. This takes thirty minutes to set up and produces immediate, actionable results.

Short term: Run bmad-testarch-trace Phase 1 to establish a coverage baseline. This maps what currently exists to what is being covered and more importantly, what is not being covered. For a vibe coding team this is often the first time they have seen their actual coverage picture rather than a coverage percentage.

Medium term: Introduce bmad-testarch-atdd before new feature implementation. Even without formal acceptance criteria, the team can describe the expected behaviour in natural language. TEA converts that description into failing test scaffolds. The agent implements them. Implementation quality improves because the verification mechanism exists before the code is written.

None of this requires switching to BMAD. None of it requires writing a PRD or creating architecture documents. It requires writing down what a feature should do before implementing it which is good practice regardless of methodology and running a quality audit on the tests that come out.

TEA on a project inheriting legacy tests

A common scenario that does not fit neatly into any methodology category: a team inheriting a codebase with existing tests of unknown quality. The tests were written by previous engineers, possibly supplemented by AI generation, possibly migrated from an older framework. Nobody is sure which tests still reflect current behaviour.

TEA has an explicit brownfield engagement model for exactly this situation.

Start with bmad-testarch-trace Phase 1. This does not require any BMAD artifacts, it takes the existing test files and existing codebase and produces a coverage matrix showing which parts of the system have test coverage and which do not. For many teams this is the first honest picture of their test coverage they have ever had.

Then run bmad-testarch-test-review on the test files with the lowest confidence. The quality score identifies specifically which tests are brittle (hard waits, brittle selectors), which are non-deterministic (conditional flows, environment dependencies), and which are poorly isolated (shared state, order-dependent). The team has a prioritised list of what to fix.

The team does not need to adopt BMAD, Spec Kit, or any other methodology to get value from this. They just need to point TEA at their test files and their codebase. TEA does the rest.

The three questions every team should answer before shipping AI-generated code

Regardless of implementation methodology, every team shipping AI-assisted code should be able to answer three questions before each release:

Question 1: Are the tests testing what we think they are testing? Not "are the tests passing", they will pass. "Do the tests map to the acceptance criteria or requirements we committed to?" If there is no traceability, there is no honest answer to this question. bmad-testarch-trace provides the traceability.

Question 2: Are the tests themselves trustworthy? Not "did the tests pass in this run", they passed. "Will these tests give us a reliable signal the next time they run, and the run after that, and after a refactor?" If there is no quality audit, there is no honest answer. bmad-testarch-test-review provides the audit.

Question 3: Do we know which scenarios we are not testing? Not "is our coverage percentage above threshold", it is. "Do we know which edge cases, error states, and integration paths have no test coverage?" If there is no gap analysis, there is no honest answer. bmad-testarch-test-design and bmad-testarch-trace together provide the gap analysis.

Teams that cannot answer these three questions are making release decisions based on confidence rather than evidence. That confidence is usually misplaced. The tests look productive. But without a quality layer, they are measuring activity, not safety.

The adoption decision

TEA does not require a methodology change. It requires a quality mindset change, the recognition that generating tests and architecting tests are different activities, and that AI agents are good at the first one and need guidance for the second.

For teams using BMAD: add TEA to your Spec Readiness and sprint execution workflow. The integration points are defined. The value compounds with each sprint.

For teams using Spec Kit: run test-review and trace on every new test file. Add atdd before implementation for any non-trivial spec. The spec gives TEA enough context to work effectively without BMAD's full planning artifacts.

For teams vibe coding: start with TEA Lite (automate and test-review only). Get a quality score on your current test suite. See the specific issues. Fix the top three. The investment is thirty minutes and the return is an honest picture of where your test quality actually stands.

For teams inheriting legacy tests: start with trace Phase 1. Get the baseline. Know what you have. Everything else follows from there.

The implementation methodology is a choice your team has already made. The test quality layer is a choice you make on top of it and unlike the implementation methodology, it is a choice you can make without changing anything else about how your team works.

This article is based on ToolTwist's production experience building the Agentic Sprint Delivery (Brownfield) configuration.