Reporting so a failure is actionable

A test failed in CI, on a server you cannot see, at 2 a.m. Whether that failure gets fixed or ignored depends almost entirely on one thing: can someone understand it quickly? A failure that says "expected true, got false" on line 40 is a dead end; a failure that comes with the error, a screenshot, and a trace is a five-minute fix. Reporting — making failures legible and actionable — is what turns a CI suite from noise into a tool people act on. This lesson is reporting so a failure is actionable.

Why reporting is a QA concern, not an afterthought

It ties back to the theme of the whole automation part: a suite people do not trust is worse than none. Trust is lost not only to flakiness but to illegible failures — if every red build takes an hour to understand, people stop looking, and the suite stops protecting anything. Good reporting is how a failure earns a response instead of a shrug. So reporting is not decoration; it is what makes the suite usable, and a QA who invests in it multiplies the value of every test.

What a good failure report contains

When a test fails, the person triaging it needs to answer "what broke, and why?" fast. A good report gives:

Which test failed, clearly named. This is why test names should describe behaviour: "forbids an operator reading another agent's transaction (403)" tells you what broke; "test 7" tells you nothing. Name tests as sentences.
The assertion that failed — expected vs actual. "Expected status 403, received 200" points straight at the bug.
A screenshot (for UI tests) — the page at the moment of failure, so you see the actual state.
A trace / logs — the timeline of what happened, the network, the console (the debugging lesson).
How to reproduce — the environment, the seed, enough to run it again.

AgentPay's setup produces all of this: screenshot: 'only-on-failure', trace: 'on-first-retry', and the HTML reporter, uploaded as a CI artifact. So a failed E2E test in CI comes with a browsable report: the named test, the error, the screenshot, and a scrubbable trace — enough to diagnose without ever logging into the CI server.

Reporters: how the results are presented

A reporter decides how test results are formatted. Test runners ship several, and you choose per context:

Locally — a concise list or dot reporter, so you see pass/fail as you work. AgentPay uses list locally.
In CI — a machine-and-human format. AgentPay uses ['github'] (which annotates the failing lines directly on the pull request) plus ['html'] (a full browsable report uploaded as an artifact):
```
reporter: process.env.CI ? [['github'], ['html', { open: 'never' }]] : [['list']],
```
The HTML report — one page listing every test, and for each failure the error, screenshot and trace, all clickable (npx playwright show-report).
JUnit XML — a standard format many CI systems read to display test results natively and track them over time.

The pattern: a readable reporter locally, and in CI a format that both annotates the PR and preserves a detailed report as an artifact. That combination means a failure is visible right on the pull request and fully diagnosable from the uploaded report.

Making failures land where people see them

A report nobody looks at helps no one, so put the result where the team already is:

On the pull request — the github reporter annotates the failing line on the PR, and the required status check shows red. The person who caused the failure sees it on their own PR, immediately.
As downloadable artifacts — the HTML report, traces and screenshots attached to the CI run, so anyone can pull them and diagnose (AgentPay's upload-artifact step).
Notifications, judiciously — a message to the team's chat when main goes red. Useful, but only if the suite is reliable — notifying on flaky failures trains people to mute the channel, so earn reliability first.

The aim is that a failure is impossible to miss and easy to understand: visible on the PR, with a full report one click away. That is what makes CI a living safeguard rather than a formality.

Flaky failures poison reporting too

One caution that ties the whole part together: the best reporting in the world cannot save a flaky suite. If failures are often false alarms, people learn to disbelieve the reports — a detailed, beautiful report of a failure that "is probably just flakiness" gets ignored like any other. So reporting and reliability work together: reliable tests make each report worth reading, and good reports make each real failure fixable. Invest in both, and the suite becomes something the team genuinely relies on — which, after nine modules, is the entire point.

Check your work

Reporting = actionability. A suite loses trust to illegible failures as surely as to flakiness; if a red build is hard to understand, people stop looking. Good reporting makes a failure earn a fix, not a shrug — it multiplies every test's value.

A good report contains: the clearly-named failing test (name tests as behaviour sentences), the failed assertion (expected vs actual), a screenshot (UI), a trace/logs, and how to reproduce. AgentPay produces all of it (screenshot-on-failure, trace-on-retry, HTML report uploaded as an artifact).

Reporters format results: concise (list/dot) locally; in CI a format that annotates the PR (github) plus a preserved detailed report (html, JUnit XML). AgentPay: list locally, github + html in CI.

Put failures where people are: annotated on the PR (with a required check), downloadable artifacts (report/trace/screenshot), and judicious chat notifications (only once the suite is reliable). Visible and diagnosable in one click.

Reliability + reporting together: flaky failures make even great reports ignored; reliable tests make each report worth reading. Invest in both.

Practice

Take a poorly named test ("test 3") and rename it as a behaviour sentence; explain how that helps triage.
List everything AgentPay attaches to a failed E2E run in CI and how each helps diagnose it.
Configure a runner to use a concise reporter locally and a PR-annotating + HTML reporter in CI.
Explain how the github reporter and a required check put a failure in front of the person who caused it.
Generate an HTML report from a failing run and walk through diagnosing the failure from it alone.
Explain why great reporting cannot rescue a flaky suite, tying reliability and reporting together.

Official documentation

Playwright — Reporters — Built-in reporters, including github, html and JUnit.
Playwright — Continuous Integration — Producing and uploading reports in CI.
GitHub Actions — Storing workflow artifacts — Attaching reports and traces to a CI run.

Next: test environments and data, with Docker.