Is Visual Regression Testing Worth It for a Marketing Site?

Is Visual Regression Testing Worth It for a Marketing Site?

For most marketing sites, no. For design systems and component libraries, yes. The technique catches unintended visual change, which is a real problem, but on a site where the content changes weekly by design, the false alarms outnumber the catches and the suite gets ignored.

We have set these up, we have inherited them, and we have switched several off. The pattern is consistent enough that we now ask a different question at the start: what exactly are you afraid will change without anyone noticing.

If you can answer that in one sentence, visual testing is probably right. If you cannot, you are buying screenshots.

What Is Visual Regression Testing?

It is automated screenshot comparison. A tool captures an image of a page or component, stores it as a baseline, and on every later run compares the new capture to that baseline. Any pixel difference beyond a set tolerance fails the build.

The appeal is obvious. Unit tests tell you a function returns the right value. They say nothing about a CSS change that pushed your pricing table off the screen at one breakpoint. Visual tests are the only automated check that looks at the rendered result.

The difficulty is equally obvious once you run one. Pages change constantly, and the tool cannot tell an intended change from an accident.

How Does the Tooling Actually Work?

Playwright is the common starting point because it is already in many stacks. Its assertion is expect(page).toHaveScreenshot(), and on a first run it takes screenshots until two consecutive ones match, saves the result, and reports that a snapshot did not exist and it wrote the actual image.

Those baseline images get committed to version control, which is the part teams underestimate. Your repository now carries binary files that change whenever the design does, and reviewing a pull request means looking at images rather than reading a diff.

Tolerance is configurable. Playwright's threshold option defaults to 0.2, defined as an acceptable perceived colour difference in the YIQ colour space, while maxDiffPixels and maxDiffPixelRatio are unset by default. It also disables animations and hides the caret by default, which removes two of the most common sources of noise.

Why Do These Suites Get So Flaky?

Because rendering is not deterministic across environments, and Playwright's own documentation is unusually blunt about it. It warns that browser rendering can vary based on the host operating system, version, settings, hardware, power source of battery versus power adapter, headless mode and other factors.

Read that list again. Whether the laptop was plugged in can change the screenshot. That is why the guidance is to run tests in the same environment where the baselines were generated, which in practice means running them in a container in continuous integration and never locally.

Teams that skip that step spend their first month approving diffs caused by their own machines, and by month two nobody looks at the results. That failure mode is not a tooling problem, it is a setup decision made once, badly.

What Does It Cost to Run Hosted?

Self hosting with Playwright is free and gives you the storage and review problem. Hosted services solve the review problem and charge by snapshot.

Chromatic's published pricing has a free plan with 5,000 billed snapshots per month on Chrome only. Its Starter tier is 179 dollars per month for 35,000 billed snapshots and adds Safari, Firefox and Edge, Pro is 399 dollars per month for 85,000 snapshots, and extra snapshots are priced at 0.008 dollars each. Enterprise is custom with unlimited snapshots.

The number that matters is not the tier, it is how fast your snapshot count multiplies. Every viewport and every browser you add multiplies captures per build, and every build multiplies again. A modest component library at several breakpoints across four browsers can consume a tier faster than the headline numbers suggest.

Where Does Visual Testing Genuinely Earn Its Place?

Design systems and component libraries, without question. A component is a stable, isolated unit with a small number of intended states. A change to a shared button component can affect forty pages, and no human is going to check forty pages.

That is the exact shape the technique is built for: stable baselines, high blast radius, and a change nobody can review by eye. The same logic covers shared layout primitives, form controls, and anything consumed by more than one team.

It also earns its place on a small set of pages where the stakes are high and the content is stable: the pricing page, the checkout flow, the signup form. Not the blog.

Why Is a Marketing Site Different?

Because change is the point. A marketing site's content is edited constantly, by people who are not developers, often through a CMS where a longer headline legitimately reflows a hero section.

Every one of those legitimate edits produces a failing visual test. The team's options are to approve dozens of diffs a week, exclude the dynamic regions until the test covers almost nothing, or stop looking. All three end in the same place.

There is a structural point here too. On a Webflow site or any CMS driven build, the content is not in the repository, so a baseline captured on Tuesday can fail on Wednesday with no code change at all. That is not a bug in the tool. It is a mismatch between the tool and the medium.

What Should You Do Instead on a Marketing Site?

Cover the things that break silently and are not supposed to change. Automated accessibility checks catch contrast and label regressions that nobody reviews by eye. Performance budgets in continuous integration catch the script somebody added. Link checking catches the migration you did last month.

Then use deploy previews and have a human look at the pages that changed. A reviewer who sees the actual page for thirty seconds catches more than a screenshot diff does, because they understand intent. That is the strongest argument for the workflow we describe in staging sites and deploy previews.

And keep a short manual checklist for launches. It is unfashionable advice and it works, which is why it sits at the centre of our approach to pre launch testing.

Is There a Lighter Version Worth Adopting?

Yes. Capture a handful of pages at two or three breakpoints, run it only on dependency updates and framework upgrades rather than on every commit, and treat it as a change detector rather than a gate.

This is where the technique quietly shines. When a CSS framework or a component library updates, you want to know what moved, and reading a changelog will not tell you. A small suite run at that moment answers the question in minutes. That is one of the reasons we are systematic about dependency updates on marketing sites.

Run in that mode, a dozen snapshots are enough, the cost stays inside a free tier, and nobody is approving diffs all week.

How Do You Know If Yours Is Working?

Count two things over a quarter. How many real regressions did it catch that would otherwise have shipped. How many diffs did someone approve without looking properly.

If the second number is much larger than the first, the suite is theatre and it is costing you attention you could spend elsewhere. Reduce the scope until the ratio makes sense, or turn it off and say so out loud rather than letting it rot in the pipeline.

A test nobody trusts is worse than no test, because it occupies the slot where a real check would go.

What Would We Recommend This Quarter?

If you maintain a component library, set up visual testing on the components and run it in continuous integration in a container. If you run a marketing site, skip it, and put the same effort into accessibility checks, a performance budget, and a deploy preview habit.

If you already have a failing suite that everyone ignores, the honest move is to cut it down to the ten things you genuinely cannot afford to have change, and make those ten matter again.

If you are weighing this up for a site rebuild and want a straight answer about whether it fits your setup, we are happy to give one. You can find us at phoenix.studio.