VisaSpark
Sign in
advanced22 min

CI Pipelines, Quality Gates, and Sharding

Wiring a framework into a real CI pipeline as an actual quality gate (not just an informational report), and splitting a growing suite across shards to keep CI runtime reasonable as it scales.

What you'll learn

  • Explain the difference between a CI check that's genuinely a quality gate and one that's purely informational
  • Design a sharding strategy that splits a test suite for parallel CI execution without breaking suite-level assumptions
  • Explain what caching in a CI pipeline should and shouldn't be relied on for

Prerequisites

Explanation

No real CI pipeline runs in this lesson's exercises -- they model quality-gate and sharding decisions as data, using genuine JavaScript/TypeScript execution. This platform runs its OWN real CI, including a real Playwright suite, in its own repository infrastructure -- entirely separate from anything in a learner's browser exercise.

A CI check is only a genuine quality gate if it can actually block something from happening — merging a pull request, deploying a build — when it fails. A test suite that runs in CI and reports its result somewhere, but where a failure doesn't prevent a merge or a deploy, is purely informational: useful for visibility, but it provides none of the actual protection a quality gate is meant to provide, since a failing check that nobody is required to act on can be, and eventually will be, silently ignored. Turning a check into a genuine gate is typically a repository/CI-platform configuration decision (a required status check on a branch protection rule, for example) — but designing the underlying test suite to be honest and stable enough to safely gate on (not flaky, not slow enough to be routinely skipped under deadline pressure) is squarely a framework-engineering responsibility.

Sharding splits one large test suite across multiple parallel CI workers/machines, each running a subset of the full suite — this is what keeps a growing suite's total CI wall-clock time reasonable as the number of tests increases, since 4 shards running in parallel can finish in roughly a quarter of the time one worker running everything sequentially would take. Sharding does require the suite to already be genuinely isolated (Lesson 8) — tests assigned to different shards may run on completely different machines, at different times, with zero shared state whatsoever, so any test that secretly depended on another test's leftover state (even if that dependency happened to "work" when both ran on the same single worker) will break unpredictably once sharding is introduced, depending on which shard each ends up assigned to.

CI caching (of installed dependencies, browser binaries, build outputs) is a genuine, valuable way to reduce redundant, repeated work across runs — but it should be relied on specifically as a performance optimization, never as a substitute for correctness. A cache that's stale, corrupted, or keyed incorrectly should, at worst, produce a slower run (falling back to a full, uncached install) — a well-designed CI pipeline should never depend on a cache being present or valid for CORRECT results, only for a faster ones; this is exactly why CI pipelines typically also run correctly, if more slowly, on a fully clean cache-miss run.

Example

Modeling the gate-vs-informational distinction and estimating a sharded suite's parallel runtime, as data.

function isGenuineQualityGate(checkBlocksMerge, checkBlocksDeploy) {
  return checkBlocksMerge || checkBlocksDeploy;
}
console.log(isGenuineQualityGate(true, false));  // true -- a failure here actually prevents something
console.log(isGenuineQualityGate(false, false)); // false -- purely informational, easy to silently ignore over time

function estimatedShardedRuntimeMinutes(totalSequentialMinutes, shardCount) {
  // A simplified model -- real sharding overhead exists, but the core benefit is this rough division.
  return Math.ceil(totalSequentialMinutes / shardCount);
}
console.log(estimatedShardedRuntimeMinutes(40, 1)); // 40 -- one worker running the whole suite
console.log(estimatedShardedRuntimeMinutes(40, 4)); // 10 -- roughly a quarter of the time, run in parallel across 4 shards

Try it yourself

Call estimatedShardedRuntimeMinutes with totalSequentialMinutes 90 and shardCount 6, and observe the estimated parallel runtime.

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Guided exercise

Guided exercise

This models classifying a CI check's real gating strength only -- no real CI runs. Write gateStrength(blocksMerge, blocksDeploy, isFlaky): if isFlaky, return 'unreliable-gate' (a flaky gate is dangerous even if configured to block). Else if blocksMerge || blocksDeploy, return 'genuine-gate'. Else return 'informational-only'.

Checks: correctly identifies a genuine, stable gate · correctly flags a flaky check as unreliable, even if configured to block · correctly identifies a purely informational check

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Stuck? Get a hint.

Independent exercise

Independent exercise

This models deciding whether a CI pipeline correctly treats caching as a performance-only optimization -- no real CI cache is used. Write cachingIsSafe(pipelineWorksOnCacheMiss, resultsChangeBasedOnCacheHit): return pipelineWorksOnCacheMiss && !resultsChangeBasedOnCacheHit.

Checks: correctly identifies safe, performance-only caching · correctly rejects a pipeline that cannot function on a cache miss · correctly rejects a pipeline whose results depend on cache state

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Stuck? Get a hint.

Guided local lab

Add CI Quality Gates, Reporting, Sharding Guidance, and Failure Triage

Runs on your computer
This lab runs on your own computer, in your own terminal and editor — not in your browser. VisaSparkSchools does not execute, run, or verify these commands for you. Follow the verification steps yourself to confirm your result.

Extend the framework from earlier guided local labs with a real GitHub Actions workflow that runs the test suite, reports results clearly, and would genuinely gate a pull request -- real, local (and GitHub-hosted) CI configuration work. Every command below runs in YOUR terminal and your own GitHub repository; this platform does not execute any of them.

Required tools

  • Node.js (20.x or 22.x LTS)
  • Playwright (1.62.x)
  • Git (any current version)
  • A GitHub account (for the CI portion) (free tier is sufficient)

Setup

  1. Continue in the automation-framework project from earlier guided local labs (or recreate its basic structure if needed).
  2. Initialize git if not already done: `git init` (if this is a fresh project).
  3. Create the workflow folder: `mkdir -p .github/workflows`.

Project structure

automation-framework/
  .github/
    workflows/
      ci.yml
  src/
    ... (from earlier labs)
  tests/
    ... (from earlier labs)
  playwright.config.ts
  package.json

Starter files

.github/workflows/ci.yml

name: CI

on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    # TODO: add a strategy.matrix block with at least 2 shardIndex values and a matching shardTotal
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - name: Install dependencies
        run: npm ci
      - name: Install Playwright browsers
        run: npx playwright install --with-deps
      - name: Run tests
        # TODO: pass --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} to npx playwright test
        run: npx playwright test
      # TODO: add an "Upload failure diagnostics" step using actions/upload-artifact@v4,
      # gated on if: failure(), uploading the playwright-report/ folder

Requirements

  • ci.yml triggers on pull requests targeting main.
  • The workflow uses a shard matrix (at least 2 shards) so the suite genuinely runs split across parallel jobs.
  • npx playwright install --with-deps runs before the test step, so the CI runner actually has real browsers available.
  • On failure, the workflow uploads the Playwright HTML report as an artifact, so a failure's diagnostics are retrievable without re-running locally.
  • The workflow is pushed to a real GitHub repository, and a test pull request confirms it actually runs (this specific step is done in your own GitHub account, not on this platform).

Commands to run

  • Confirm the suite passes locally before pushing

    npx playwright test

  • Commit and push the workflow file

    git add .github/workflows/ci.yml && git commit -m "Add CI workflow with sharding" && git push

  • Open a pull request on GitHub and observe the workflow run in the Actions tab

    gh pr create --fill

Expected behavior

Opening a real pull request on GitHub triggers the workflow, which runs two shard jobs in parallel, each executing roughly half the suite. Both shards report their results in the PR's checks. Intentionally breaking a test and pushing again demonstrates the failure being clearly visible in the PR checks, with the Playwright HTML report available as a downloadable artifact from the failed run.

Verify it yourself

  • (in the GitHub PR's Checks tab) confirm 2 shard jobs ran

    Expected: both shardIndex 1/2 and shardIndex 2/2 jobs appear and complete

  • (after intentionally breaking one test and pushing) check the PR's Checks tab again

    Expected: the relevant shard job shows as failed, and a playwright-report artifact is available for download

Troubleshooting

  • The workflow doesn't trigger at all on a new pull request — Confirm ci.yml is on the branch the PR is FROM (not just main) and that the `on: pull_request: branches: [main]` target matches your actual default branch name.
  • `npx playwright install --with-deps` fails on the CI runner — ubuntu-latest should have the required system dependencies with --with-deps -- confirm the runs-on value is exactly ubuntu-latest and the Playwright version matches package.json.
  • Only one shard appears to run — Confirm the strategy.matrix block correctly lists both shardIndex values (1 and 2) and shardTotal is 2, matching the --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} flag.

Stuck? Get a hint.

Extension challenge

In your GitHub repository's branch protection settings, mark both shard jobs as required status checks for main, then confirm (by intentionally breaking a test in a new PR) that GitHub now actually blocks the merge button -- turning this from an informational check into a genuine quality gate.

When you've verified this locally, use the "Mark lesson complete" button below to record your progress.

Common mistakes

  • Treating a CI check as a safety measure without ever configuring it as a required, merge-blocking status check -- an informational-only check that nobody is required to act on will eventually be silently ignored.
  • Introducing sharding before the suite is genuinely isolated (Lesson 8) -- tests that secretly depended on another test's leftover state can break unpredictably once they're split across different shards/machines with no shared execution context.
  • Designing a pipeline that only works correctly when a cache hits, rather than treating caching as a pure performance optimization -- this turns an infrastructure convenience into a hidden correctness dependency.

Knowledge check

Knowledge check

1. What makes a CI check a genuine quality GATE, as opposed to purely informational?
2. Why does sharding require the test suite to already be genuinely isolated?
3. What should CI caching be relied on for, and what should it never be relied on for?

Takeaway

Configure a genuinely stable, reliable check as a required, merge-blocking status check to make it a real quality gate, not just informational visibility. Only introduce sharding once the suite is genuinely isolated. Treat CI caching strictly as a performance optimization -- a pipeline must still produce correct results on a cache miss.

Summary

A CI check is a genuine quality gate only when its failure actually blocks a merge or deploy -- a check with no such consequence tends to be silently ignored. Sharding splits a suite across parallel workers with zero shared state, which requires genuine test isolation to have already been achieved, or pre-existing isolation bugs will surface unpredictably. CI caching should be relied on strictly for speed -- a pipeline should always produce correct results on a cache miss, never depend on a cache hit for correctness.

References

Your notes

Notes save automatically.

Finished this lesson?

Mark it complete to track your progress and schedule a future review.

Next: Failure Triage, Ownership, and Code Review Standards →