work / decision-intelligence
03Decision Intelligence
AtlasOraA guest-facing feature that scores each home as a percentage match against a guest’s priorities, labels AI judgement separately from measured fact, and falls back to facts when a model fails.
1The problem
Guests choosing between homes on AtlasOra, a vacation rental marketplace, need to weigh them against their own priorities.
2What I built
A guest-facing feature that scores each home as a percentage match against the guest’s priorities, using measured distances, noise, sunlight, listing details and reviews.
Multi-model design: Jev, TypeSafe’s decision model, makes the scoring decisions, and a language model reads the guest’s request and explains the result. In production within weeks of Jev’s public launch.
AI judgement is labelled separately from measured fact.
Shipped in stages: a hand-written evaluation set with a 100% pass bar, then a two-day shadow run against a standard LLM pipeline, then a two-day canary over several hundred live requests.
The launch gate was zero false claims: never calling a home close to the beach when it is not.
The model’s instructions were iterated against the evaluation set to fix place ambiguity, such as an Irish bar matching an Irish cultural centre.
3How the AI is controlled
4The result
| Measure | Value | Fact |
|---|---|---|
| Runs | on guest request only | d2 |
| Bookings made by the AI | none | d2 |
| Results reordered by the AI | none | d2 |
| Evaluation set pass barHand-written, before the shadow run | 100% | d5 |
| Shadow run, then canaryCanary over several hundred live requests | 2 days each | d5 |
| False claims allowed at launch | zero | d6 |
| Model outage, Sep 2026On measured facts alone | kept working | d8 |
| Guest outcome dataUsage and bookings, by handover in October 2026 | not collected | d9 |
Guest outcome data (usage, bookings) had not been collected by handover in October 2026.