Skip to content

Latest test, 20 Oct 2026: Haiku 4.5 vs Haiku 5.5 vs Mercury 3. See the scoreboard →

Andrew Deighan

work / decision-intelligence

03Decision Intelligence

AtlasOra

A guest-facing feature that scores each home as a percentage match against a guest’s priorities, labels AI judgement separately from measured fact, and falls back to facts when a model fails.

1The problem

Guests choosing between homes on AtlasOra, a vacation rental marketplace, need to weigh them against their own priorities.

2What I built

A guest-facing feature that scores each home as a percentage match against the guest’s priorities, using measured distances, noise, sunlight, listing details and reviews.

Multi-model design: Jev, TypeSafe’s decision model, makes the scoring decisions, and a language model reads the guest’s request and explains the result. In production within weeks of Jev’s public launch.

AI judgement is labelled separately from measured fact.

Shipped in stages: a hand-written evaluation set with a 100% pass bar, then a two-day shadow run against a standard LLM pipeline, then a two-day canary over several hundred live requests.

The launch gate was zero false claims: never calling a home close to the beach when it is not.

The model’s instructions were iterated against the evaluation set to fix place ambiguity, such as an Irish bar matching an Irish cultural centre.

Guest asks, facts first, judgement labelled
Guest asks, facts first, judgement labelled1. Guest asks for a comparison (Person): Nothing runs until they do 2. Measured distances, noise, sunlight, listings, reviews (Data) 3. Score each home against priorities (Model): Jev decides; a language model explains 4. If it fails: A model fails: Falls back to measured facts alone 5. Shown as fact and judgement (Labelled): Judgement labelled separately 6. Guest decides (Person): It never books, never reordersGuest asks for a comparisonNothing runs until they dopersonMeasured distances, noise, sunlight, listings, reviewsdataScore each home against prioritiesJev decides; a language model explainsmodelA model failsFalls back to measured facts alonefallbackShown as fact and judgementJudgement labelled separatelylabelledGuest decidesIt never books, never reordersperson
stepa personfailure pathright-hand tag: model or rule used

3How the AI is controlled

It may
  • Score each home against the priorities a guest gives it, using measured data. d1
It may not
  • Run unless the guest asks. d2
  • Book anything, or reorder results. d2
  • Present its judgement as fact. AI judgement is labelled separately. d3
  • Call a home close to the beach when it is not. Zero false claims was the launch gate. d6
Who approves
  • The guest starts it and makes every decision. d2
When it fails
  • If a model fails, it falls back to measured facts alone. d8

4The result

Run log8 rows · source: fact file
MeasureValueFact
Runson guest request onlyd2
Bookings made by the AInoned2
Results reordered by the AInoned2
Evaluation set pass barHand-written, before the shadow run100%d5
Shadow run, then canaryCanary over several hundred live requests2 days eachd5
False claims allowed at launchzerod6
Model outage, Sep 2026On measured facts alonekept workingd8
Guest outcome dataUsage and bookings, by handover in October 2026not collectedd9

Guest outcome data (usage, bookings) had not been collected by handover in October 2026.