VisaSpark
Sign in
advanced21 min

Reporting, CI Execution, and Environment Configuration

Turning a test run's results into something a team can actually act on, and the specific settings that make a suite behave correctly and safely in CI rather than just on a laptop.

What you'll learn

  • Configure Playwright's HTML reporter and explain what it adds over raw console output
  • Identify CI-specific configuration a local-only setup doesn't need
  • Design environment-variable-based configuration that keeps secrets out of committed files

Prerequisites

Explanation

Playwright's HTML reporter (reporter: "html") produces a browsable report linking every test to its captured screenshots, videos, and traces directly — genuinely more useful than raw console output for anyone besides the person who happened to be watching the terminal when the run finished, since it survives the run and can be shared, reviewed later, or attached as a CI artifact. Other reporters ("list", "dot", "json", "junit") suit different needs: "junit" produces XML output many CI platforms and dashboards can natively parse and display, independent of Playwright's own HTML report; "json" is the right choice when a separate tool needs to programmatically process results. Configuring multiple reporters at once (reporter: [["html"], ["junit", { outputFile: "results.xml" }]]) is common and reasonable — a human-browsable report and a machine-parseable one serve genuinely different audiences from the same run.

CI-specific configuration exists because a CI runner's environment differs from a developer's laptop in ways that matter: process.env.CI is the conventional signal most CI platforms set automatically, letting playwright.config.ts branch its own behavior — retries: process.env.CI ? 2 : 0 (retry more readily in CI, where transient infrastructure noise is more common, but fail immediately and loudly on a local run, where the developer wants to see the real failure right away) and workers: process.env.CI ? 4 : undefined (explicitly bound parallelism to a CI runner's actual, often more limited, resources, rather than Playwright's local default of using most available cores) are two of the most common, load-bearing examples of this branch.

Environment-variable-based configuration (baseURL: process.env.BASE_URL, use: { httpCredentials: { username: process.env.HTTP_USER, password: process.env.HTTP_PASS } }) is what keeps secrets and environment-specific values out of committed config files entirely — the committed playwright.config.ts references the names of environment variables, never their actual values, and a .env.example file (committed, containing only variable names with placeholder or empty values) documents what a real .env (never committed, listed in .gitignore) needs to provide. This is the same discipline this platform's own .env.example/.env split follows, and getting it right is what prevents a real credential from ever ending up in version-controlled history.

Example

Modeling CI-vs-local config branching and the env-var-name-not-value discipline, as data.

function buildConfig(isCi) {
  return {
    retries: isCi ? 2 : 0,          // retry more readily in CI's noisier environment
    workers: isCi ? 4 : undefined,  // explicitly bound to the CI runner's known resources
    reporter: isCi ? [["html"], ["junit", { outputFile: "results.xml" }]] : [["list"]],
  };
}
console.log(buildConfig(true));  // CI config: retries, bounded workers, dual reporters
console.log(buildConfig(false)); // local config: no retries, default workers, simple list reporter

function referencesSecretSafely(configValue) {
  // A safe config value NAMES an env var; it never contains a literal-looking secret itself.
  return typeof configValue === "string" && configValue.startsWith("process.env.");
}
console.log(referencesSecretSafely("process.env.HTTP_PASS")); // true -- safe: a reference, not a value
console.log(referencesSecretSafely("sk-abc123real"));           // false -- a literal secret should never appear here

Try it yourself

Call buildConfig(true) and confirm the reporter array includes both 'html' and 'junit' entries.

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Guided exercise

Guided exercise

Write ciAwareRetries(isCi) returning 2 if isCi is true, otherwise 0. Then write ciAwareWorkers(isCi) returning 4 if isCi is true, otherwise null (modeling Playwright's local default of 'use most available cores', represented here as null meaning 'no explicit limit').

Checks: CI retries twice · local runs do not retry · CI bounds workers explicitly · local runs use no explicit worker bound

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Stuck? Get a hint.

Independent exercise

Independent exercise

Write auditConfigForSecrets(configValues) where configValues is an array of strings. Return an array of every value that looks like a LITERAL secret rather than an env-var reference -- a value is safe if it starts with 'process.env.'; otherwise, if it's a non-empty string, flag it as suspicious.

Checks: correctly flags a literal-looking value while ignoring env references and empty strings · flags nothing when every value is a safe env-var reference

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Stuck? Get a hint.

Guided local lab

Diagnose Failures Using Traces, Reports, Screenshots, and CI Artifacts

Runs on your computer
This lab runs on your own computer, in your own terminal and editor — not in your browser. VisaSparkSchools does not execute, run, or verify these commands for you. Follow the verification steps yourself to confirm your result.

Configure full diagnostic artifact capture on your Playwright project, deliberately break a test to generate real failure artifacts, and wire up a CI-ready configuration — the capstone of this course's debugging and operational work.

Required tools

  • Node.js (20.x or 22.x LTS)
  • @playwright/test (1.62.x)
  • A terminal (any)

Setup

  1. Continue from the pw-learning-lab project used in this course's earlier guided local labs.
  2. Add a deliberately failing test file to generate real diagnostic artifacts to inspect.

Project structure

pw-learning-lab/
  playwright.config.ts
  tests/
    homepage.spec.ts
    intentional-failure.spec.ts
  .env.example

Starter files

tests/intentional-failure.spec.ts

import { test, expect } from "@playwright/test";

test("a deliberately wrong assertion, to generate real failure artifacts", async ({ page }) => {
  await page.goto("https://playwright.dev");
  // TODO: assert something that is DELIBERATELY FALSE (e.g. a heading with text that
  // does not actually exist on the page) -- this is intentional, to produce real
  // screenshot/trace/video artifacts you'll inspect in the verification steps below.
});

.env.example

# TODO: list the environment variable NAMES this project would need in a real CI setup
# (e.g. BASE_URL=), with no real values -- this file is committed; a real .env is not.

Requirements

  • playwright.config.ts sets trace: 'on-first-retry', screenshot: 'only-on-failure', and video: 'retain-on-failure'.
  • playwright.config.ts branches retries and workers based on process.env.CI.
  • playwright.config.ts configures both the html reporter and the junit reporter.
  • intentional-failure.spec.ts contains a genuinely failing assertion that produces a real trace, screenshot, and video.
  • .env.example documents required environment variable names with no real values, and a real .env is excluded via .gitignore.

Commands to run

  • Run the suite once locally (the intentional failure will retry and produce artifacts)

    npx playwright test

  • Open the generated HTML report

    npx playwright show-report

  • Open a captured trace file directly

    npx playwright show-trace test-results/*/trace.zip

Expected behavior

The intentional failure test fails, retries once (producing a trace on that retry), and the HTML report shows the failure linked to a real screenshot, video, and trace you can open and step through — confirming you can go from 'a test failed' to 'here is exactly what happened' using only the generated artifacts.

Verify it yourself

  • npx playwright test

    Expected: intentional-failure.spec.ts fails (as designed); homepage.spec.ts still passes

  • npx playwright show-report

    Expected: Opens a browsable HTML report; the failing test links to a screenshot, a video, and a trace

  • npx playwright show-trace <path-to-trace.zip>

    Expected: Opens Trace Viewer, showing the step-by-step timeline, network activity, and DOM snapshots leading to the failure

Troubleshooting

  • No trace file was generated — Confirm trace is set to 'on-first-retry' (or 'on') in playwright.config.ts, and that retries is greater than 0 for this to trigger on the first retry specifically.
  • The HTML report doesn't open automatically — Run `npx playwright show-report` explicitly — it serves the report from the test-results output folder.
  • A real secret ends up in a committed file — Move it to a local, uncommitted .env file immediately, confirm .env is listed in .gitignore, and replace the committed reference with process.env.YOUR_VAR_NAME.

Stuck? Get a hint.

Extension challenge

Delete the intentional-failure.spec.ts file (its job is done), and instead configure retries: 1 with trace: 'on-first-retry' on the REAL homepage.spec.ts, verifying that a genuinely passing test produces no unnecessary trace overhead, confirming the setting correctly targets only actual failures.

When you've verified this locally, use the "Mark lesson complete" button below to record your progress.

Common mistakes

  • Setting trace to 'on-first-retry' but leaving retries at 0 -- with no retry ever happening, that specific trigger condition never fires, and no trace is captured despite the setting being present.
  • Relying only on raw terminal output in CI instead of configuring the html reporter (or an equivalent) as an uploaded CI artifact -- terminal output disappears once the CI job's log is no longer easily accessible; a saved report survives and can be reviewed later.
  • Committing a .env file (rather than only .env.example) or hard-coding a real value directly into playwright.config.ts -- both leak the actual secret into version-controlled history, which .gitignore-ing the real file and referencing only process.env.VAR_NAME in committed config prevents.

Knowledge check

Knowledge check

1. Why configure both an html reporter AND a junit reporter for the same test run?
2. Why does `retries: process.env.CI ? 2 : 0` make sense as a common pattern?
3. What should a committed playwright.config.ts contain when referencing a secret like an HTTP password?

Takeaway

A saved, shareable report (html plus a CI-parseable format like junit) survives past the moment a run finishes; CI-aware config branches (retries, workers) target genuine differences between environments; and committed configuration should only ever name environment variables, never contain the real secret values themselves.

Summary

The html reporter produces a browsable report linking every test to its artifacts; junit/json serve CI dashboards and tooling. process.env.CI lets config branch retries/workers appropriately per environment. Environment-variable-referenced configuration (never literal secrets) plus a committed .env.example keeps real credentials out of version control.

References

Your notes

Notes save automatically.

Finished this lesson?

Mark it complete to track your progress and schedule a future review.

Next: Flaky-Test Diagnosis, Accessibility-Aware Testing, and Maintainable Architecture →