Flaky snapshots

A snapshot is flaky when its screenshot changes between runs although the code didn’t: an animation caught halfway, a font that loaded late, a timestamp, a random avatar. Every flaky change shows up as Changed and asks a reviewer for a decision that doesn’t matter, until reviewers stop reading diffs carefully.

Fix flakiness where it starts, in this order:

1. Let the page settle before the screenshot.
2. Mask content that is meant to change.
3. Only then, raise the tolerance.

The last section explains the Flaky and Quarantined tags in a review.

Let the page settle

Before every screenshot, the Playwright client:

1. freezes CSS animations and transitions, hides the text cursor and turns off smooth scrolling;
2. moves every running animation to its last frame;
3. waits for web fonts to load;
4. waits up to 3 seconds for images to finish loading.

A step that doesn’t finish never fails the capture: the client takes the screenshot anyway. An invalid capture.animations value in .pixmoat.yaml does fail the snapshot.

Change the defaults in the capture section of .pixmoat.yaml. Only the client reads it:

capture:
  animations: end          # end | pause | allow
  wait_for_fonts: true
  wait_for_images: true
  image_timeout_ms: 3000

To change the mode for one snapshot, pass animations:

await pixmoat.snapshot(page, "loading-spinner", { animations: "allow" });

If your page’s Content Security Policy blocks the style the client injects, the client asks the browser for reduced motion instead and logs [pixmoat] stabilization degraded (CSP). Pages that respect reduced motion still settle. On pages that don’t, allow inline styles in your test environment or mask the animated parts.

Settling can’t fix content that differs on every load. Mask that instead.

Mask content that is meant to change

A masked area is left out of the comparison: pixmoat ignores every pixel inside it. There are three ways to mask, and they add up.

In test code

Pass locators or boxes to mask on the snapshot() call. Connect Playwright shows the option. Use it when only one test needs the mask.

In your markup

Add a data-pixmoat attribute to the element itself, so every test that captures it is covered:

<!-- Left out of the comparison -->
<span data-pixmoat="ignore" class="relative-time">3 minutes ago</span>

<!-- Made invisible before the screenshot, keeping its space in the layout -->
<img data-pixmoat="hide" class="avatar" src="/api/random-avatar" />

• ignore masks the element’s box, like a mask locator.
• hide sets visibility: hidden on the element before the screenshot. The space stays, so the layout around it doesn’t move.

The attributes do nothing outside a pixmoat capture, so you can leave them in production code.

In the review

When a changing area only shows up in the review, draw a mask on it. Select a screenshot, select Ignore Regions, and drag on the image. The button changes to Done Drawing while you draw. To remove a mask while you draw, select it and press Delete or Backspace.

• Drawn masks apply from the next build on, not to the build you’re reviewing. Decide the current screenshot as usual.
• The snapshot needs an approved baseline on this build’s branch. Ignore Regions only appears when the screenshot has a baseline. When that baseline comes from the default branch and the branch has none of its own yet, saving fails with “Approve a baseline first to save ignore regions”.

Masks drawn in the review show in amber on the image. Next to Ignore Regions, the toolbar counts them, and separately counts the areas masked with data-pixmoat="ignore" in this screenshot. Those attribute masks aren’t drawn on the image.

Raise the tolerance

Tolerance is the share of pixels that may differ before a screenshot counts as Changed, from 0 to 1. The default is 0.001, one pixel in a thousand. Set it per snapshot with tolerance on snapshot(), or for every snapshot with default_tolerance in .pixmoat.yaml.

Raise it only for small, spread-out differences such as anti-aliasing on text, and only for the snapshots that need it. A tolerance high enough to hide a flaky banner also hides a real change of the same size.

Flaky and quarantined screenshots

pixmoat can also detect flakiness itself, by comparing several captures of the same snapshot taken in one build. The Playwright client takes one capture per snapshot, so screenshots it uploads are never marked Flaky or Quarantined, and the flakiness settings have no effect on them. Use the steps above instead. This section explains the tags for screenshots uploaded with several captures.

• Flaky. The extra captures in this build differed from the first by more than Flakiness threshold (%). The screenshot list shows how much when you hover over or select it, for example 0.40% inter-capture, and the detail panel compares the first capture with the one that differs most from it.
• Quarantined. The snapshot was flaky in Quarantine after (builds) builds in a row. The Quarantined group is collapsed in the screenshot list. The snapshot leaves quarantine by itself after Recovery after (builds) stable builds.

Neither needs a decision. When the review finishes, pixmoat approves them automatically, but doesn’t update their baselines, and these approvals don’t count toward the approval policy. A Flaky screenshot doesn’t hold the commit status back. A quarantined snapshot’s New or Changed screenshot does when the project blocks on visual changes, until someone finishes the review.

To take a snapshot out of quarantine by hand, a project admin opens Settings → Flakiness, finds it under Quarantined Snapshot Keys and selects Unquarantine, then Confirm. The same section sets Captures per snapshot, Flakiness threshold (%), Quarantine after (builds) and Recovery after (builds). The flakiness section of .pixmoat.yaml can override only the threshold for a build. Quarantine and recovery always use the values in Settings.