Every product owner learns the same lesson: write the acceptance criteria before the work starts. When you own a search product, you quickly learn that it breaks that core lesson on your first day. You can’t write typical acceptance criteria for “the results should be better,” but that is precisely what lands in your backlog.
This isn’t an issue with stakeholders or a failure of the user by making the request. It’s the nature of the product. A search experience is never ‘finished.’ Search as a product is balancing an iron triangle of a catalog that changes, a query stream that changes, and a set of user expectations that change. Owning it means owning a system that is never done, and the standard product owner playbook was not written for that.
The problem nobody can spec
Search is hard. The evidence that search is challenging to get right is not anecdotal. Baymard Institute’s 2026 e-commerce search benchmark, built on more than 10,000 performance ratings across 170 sites and apps, found that 56 percent of sites fail to adequately support users’ search needs. In their moderated testing, participants needed more than one search attempt 65 percent of the time, and 31 percent of the time they either failed to find the item or gave up.
These numbers are from the top of the market, having budgets, teams, and years of constant investment. All this, and they still ship a product that fails way too often.
What makes this a product ownership problem instead of an engineering problem? It’s how the work arrives as anecdotes. An executive runs a query, dislikes the first page, and the ticket is born. A merchandiser notices a competitor’s product ranking above a house brand. A customer emails support. These are real, and each of them is one query out of thousands. A product owner who accepts anecdotes as acceptance criteria will spend every sprint chasing the loudest voice and will never know whether the system got better.
Value is a number, not a feature
The Scrum Guide gives the product owner one core accountability: maximizing the value of the product resulting from the work of the Scrum Team. It deliberately leaves the definition of value open, because it varies by product. For search, the definition has to be a measurement.
The mechanism is the judgment list: a fixed set of queries, each paired with results that people have graded for relevance. Doug Turnbull describes it as the tool that moves relevance evaluation away from a subjective “looks good to me” review and toward something systematic, a test harness that lets a team change ranking logic and immediately see what improved and what regressed. He has called the practice test-driven relevancy, and the comparison to unit tests is exact. Elastic’s engineering guidance makes the same point from the workflow side: judgment lists power offline testing, which lets a team iterate quickly and cheaply before committing to slower live experiments.
With a judgment list in place, the search product owner can finally write a definition of done that survives contact with an opinion. A relevance change is done when it improves the target query class on the judgment set and does not regress the rest. The executive’s query can be added to the set and graded like any other. It counts, but it counts as one query, weighted the same as the others. That is the difference between a program and a complaints desk.
What the search backlog actually contains
A feature backlog is a list of things the team will build and users will see. A search backlog has four lanes, and only one of them is visible.
Relevance tuning. Ranking weights, synonym expansion, field boosts, query rewriting, and model retraining. This is the work most people mean when they say “make search better,” and it ships nothing a user can point to. The results just look different, sometimes subtly.
Data and content quality. Missing attributes, inconsistent taxonomy, thin product descriptions, and stale documents. Relevance work is capped by the quality of what is being ranked, and this lane is usually owned by someone outside the search team. The product owner’s job is to get it onto the shared backlog anyway.
Query intent. Autocomplete, spelling correction, zero-result handling, and query classification. This lane shapes what the engine is asked before ranking ever runs, and it is the cheapest place to move the numbers.
Experience and performance. Result layout, filters, response time, and mobile behavior. This is the lane that looks like normal product work and gets normal product attention.
The political problem is obvious once the lanes are laid out. Three quarters of the work is invisible. A sprint review that shows a relevance metric moving from 0.71 to 0.74 does not land the way a new filter panel does. The product owner has to make that number mean something to the room, which is a communication job as much as a prioritization job.
Have you seen the price of gold (judgment sets)?
Judgment sets are not free and are typically temporal in accuracy. The search team at AMBOSS, a medical education platform, published their costs: roughly one rater week to label a single test case, about an hour a week of ongoing maintenance because both the engine and the content keep changing, and around 1,500 euros per complete case. In this effort their judges were medical experts, which raises the cost, but expense holds for any domain where relevance requires highly compensated knowledge workers.
This puts the product owner in a difficult position. The testing suite is a recurring expense with its headcount, and its output decays. Queries that mattered last year can drift out of the logs. Products that were graded as relevant get discontinued. A judgment set that is not refreshed becomes a measurement of a system that no longer exists. Deciding how often to refresh, which query types to cover, and whether to spend on human raters or on model-assisted grading is a search product owner’s work. It belongs on the roadmap next to the features.
Stakeholders, boosts, and expiry dates
Search has a stakeholder that most products do not: the person who wants a specific result in a specific position. Merchandisers want the house brand on top. Content owners want their page first for their topic. Legal wants a document pinned. None of these are unreasonable, and all of them are ranking overrides that the engine did not choose.
The mistake is treating these as configuration. You should treat them as backlog items. Each boost or pin gets an owner, a stated reason, a query scope, and an expiry date. When it expires, it is reviewed against the judgment set like any other change. A boost that still earns its place stays. One that has quietly been dragging a category down for eighteen months gets removed with evidence rather than an argument.
The same discipline applies to the newest stakeholder request: an AI answer layer on top of results. The expectations arrive fully formed, and the underlying retrieval is the same system with the same judgment set. If the ranking is weak, the generated answer will be confidently weak. Getting the retrieval measured first is not caution. It is the prerequisite.
Offline first, online always
The cadence that works pairs every offline improvement with an online check. Offline evaluation on the judgment set tells the team whether the ranking got better. It does not tell them whether users noticed. Turnbull’s caution is worth keeping in mind: “Judgments assume each result is evaluated in isolation, and a perfectly relevant result set that arrives slowly can still lose an A/B test”.
So the sprint rhythm looks like this. Relevance changes are proposed against the judgment set. Those that pass go to a live experiment with a behavioral metric attached, whether that is click-through, add-to-cart, task completion, or reformulation rate. Only changes that clear both bars are declared done. It is slower than shipping a feature. It is also the only way to know the product got better rather than different.
Making quality legible
The search product owner’s real job is to make quality legible. Search fails quietly. The results page always renders, the engine always returns something, and the damage shows up as a customer trying a second query, then a third, then leaving. None of that raises an alert.
Judgment lists, offline metrics, live experiments, and expiring overrides are all ways of turning that silence into something the organization can see and argue about. Once the argument is about a number, the product owner can do the job the Scrum Guide describes: order the work by value. Until then, the backlog belongs to whoever complained last.
Product Ownership is about maximizing value. When the product is relevance the work needs to be prioritized, tracked and executed like any other feature of the product.