Retry and Timeout Policy, and Managing Flaky Tests
Designing a framework-wide retry and timeout policy deliberately, and the honest, disciplined process for actually managing a flaky test instead of just silencing it.
What you'll learn
- Explain the real risk of a retry policy that's too generous, and the real cost of one that's too strict
- Design a timeout policy that distinguishes reasonable waiting from a genuinely stuck operation
- Apply a disciplined process for investigating a flaky test rather than reflexively retrying or skipping it
Prerequisites
Explanation
No real test retry or timeout executes in this lesson's exercises -- they model retry/timeout policy decisions and flaky-test triage as data, using genuine JavaScript/TypeScript execution.
A retry policy (rerunning a failed test automatically, a set number of times, before reporting it as truly failed) is a genuine, honest tradeoff, not a free safety net: retries can absorb truly transient, environment-level hiccups (a momentary network blip unrelated to the code under test) — but a retry policy that's too generous becomes a way to quietly hide a genuine, reproducible bug behind noise, since a test that fails 1 time in 3 but is retried 3 times will usually eventually "pass," reported as green despite a real, underlying problem. A well-designed policy keeps retries few (commonly just one or two) and, critically, treats every retry as signal worth investigating, not something to be silently absorbed and ignored — a framework that reports retry counts prominently (not just final pass/fail) preserves that signal for triage instead of discarding it.
A timeout policy needs to distinguish reasonable waiting (a page genuinely taking a few extra seconds to load under real, variable load) from a genuinely stuck operation that will never complete on its own. Too short a timeout produces false failures on entirely legitimate slow-but-working operations; too long a timeout means a genuinely stuck test wastes significant CI time before finally failing, and a whole suite's runtime can balloon if this happens across many tests. A framework-wide default timeout, with a small number of deliberate, justified per-test overrides for operations known to be legitimately slower, is the practical middle ground.
Flaky-test management is a disciplined process, not "add test.retry() and move on." A genuinely rigorous approach treats a flaky test the same way a real bug report is treated: reproduce it (run it repeatedly, ideally with tracing/screenshots enabled, until it fails again), form a specific hypothesis about the actual cause (an isolation bug from Lesson 8? a race condition the test's waiting strategy doesn't correctly handle? a genuine, intermittent bug in the application itself?), and fix the actual root cause — rather than reflexively adding a retry (which hides the symptom) or a .skip (which discards the test's coverage entirely). A retry or a skip can be a legitimate, temporary, explicitly time-boxed measure while a root cause is actively being investigated — but treating either as a permanent solution lets real flakiness accumulate silently across a growing suite.
Example
Modeling why a generous retry policy hides real failure signal, and a timeout-vs-legitimate-delay decision, as data.
function wouldReportAsPassing(failureRate, retryAttempts) {
// Models the probability a genuinely flaky test (given its real failure rate) eventually passes within N attempts.
const probabilityAllFail = Math.pow(failureRate, retryAttempts);
return 1 - probabilityAllFail; // probability of at least one pass among the attempts
}
console.log(wouldReportAsPassing(0.3, 1).toFixed(2)); // 0.70 -- with NO retries, this genuine 30%-failure-rate bug is visible 30% of the time
console.log(wouldReportAsPassing(0.3, 3).toFixed(2)); // 0.97 -- with 3 retries, it's reported as passing almost every time -- hiding the real bug
function timeoutIsAppropriate(observedP95Ms, configuredTimeoutMs) {
// A reasonable timeout should comfortably exceed normal, legitimate variance (p95), not be right at its edge.
return configuredTimeoutMs >= observedP95Ms * 1.5;
}
console.log(timeoutIsAppropriate(4000, 5000)); // false -- too close to normal p95 variance, risking false failures
console.log(timeoutIsAppropriate(4000, 8000)); // true -- comfortable margin above normal, legitimate varianceTry it yourself
Call wouldReportAsPassing with failureRate 0.1 (a much rarer, genuine flake) and retryAttempts 2, and observe how even a low real failure rate gets substantially masked by retries.
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Guided exercise
Guided exercise
This models flagging a retry policy as too generous only -- no real retries occur. Write isRetryPolicyTooGenerous(maxRetries), returning true if maxRetries is greater than 2 (this course's practical, deliberate ceiling for keeping retries as signal, not noise-absorption).
Checks: does not flag a minimal, deliberate retry count · does not flag the practical retry ceiling itself · flags an excessive retry count as too generous
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Stuck? Get a hint.
Independent exercise
Independent exercise
This models choosing the correct flaky-test response given available evidence only -- no real investigation occurs. Write flakyTestResponse(rootCauseIdentified, hasTimeBoxedPlan): if rootCauseIdentified, return 'fix-root-cause'. Else if hasTimeBoxedPlan, return 'temporary-time-boxed-retry-or-skip'. Else return 'investigate-before-any-action'.
Checks: prioritizes fixing a known root cause over any temporary measure · allows a temporary, time-boxed measure only when a real plan exists · defaults to investigation when neither a cause nor a plan exists
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Stuck? Get a hint.
Common mistakes
- Setting a generous retry count (more than 1-2) as a default 'safety net' -- this quietly absorbs a real, reproducible bug's failure signal, reporting it as passing more often than not.
- Setting a timeout right at the edge of normal, legitimate operation latency instead of with a comfortable margin -- this produces false failures on entirely legitimate, if slightly slower, operations.
- Reflexively adding test.retry() or .skip to a flaky test as a permanent fix, instead of a genuine, time-boxed stopgap while the actual root cause is being investigated -- this lets real flakiness accumulate silently across a growing suite.
Knowledge check
Takeaway
Keep retries few and deliberate, and always visible as signal, not silently absorbed -- a generous retry policy can hide a real, reproducible bug. Set timeouts with a comfortable margin above normal, observed latency. Treat a flaky test like a real bug: reproduce it, find its actual root cause, and fix that -- using a retry or skip only as an explicit, time-boxed stopgap, never a permanent solution.
Summary
A generous retry policy can systematically hide a genuine, reproducible bug's failure signal by letting it eventually pass within several attempts -- retries should stay few and remain visible for triage. A timeout should sit comfortably above normal, observed latency to avoid false failures on legitimately slower operations. Flaky-test management is a disciplined process of reproduction, root-cause hypothesis, and an actual fix -- a retry or .skip is a legitimate, temporary, explicitly time-boxed stopgap, never a default or permanent response.
References
Your notes
Notes save automatically.
Finished this lesson?
Mark it complete to track your progress and schedule a future review.
AI tutor
The optional AI tutor isn't enabled in this deployment. All lessons, exercises, quizzes, and search work fully without it.