SpotCheck

Finance analysts now use AI every day, but there is no shared standard for what “good” looks like for their role, and no easy way to check whether their actual AI-assisted work is trustworthy. People default to blind trust or reflexive distrust; mistakes go unnoticed and are often found late — in a review or a handover — with little time to fix them.

SpotCheck targets mid-career analysts doing spreadsheet work that ships. That surface was chosen because it is gradable: some outputs can be checked against control totals or the source data.

The core loop keeps the person in control and does not do the work for them:

1. 1. Pre-flight coach. It hardens the user's prompt and pins down assumptions. Anything it infers that the user did not state is marked as a bracketed [confirm: …] placeholder, never an assumed fact.
2. 2. Run it in your own AI tool. The user runs the prompt in ChatGPT, Copilot, or Gemini.
3. 3. Paste the output back (or upload the spreadsheet).
4. 4. Get a seven-dimension verdict plus a “judgment map” of what you caught and what you missed.
5. 5. Go deeper, optionally — a Mock Review, a Concept lesson drawn from your own check, or a short Daily Drill.

The response format is designed for coaching rather than completed answers — it prompts a review instead of doing the work. That boundary still needs testing against attempts to bypass it.

Each check is a yes/no test on a real AI output. The first four are non-negotiable; the last three flag quality rather than block.

• 1. Output verified — spot-checked rows, cross-footed the total, or tied to a known control figure.
• 2. Source integrity — cleaned data still matches the raw source: row counts, no dropped or duplicated rows, no silent type coercion.
• 3. Plausibility — magnitude, sign, and units are sane.
• 4. Instruction fidelity — it answered the exact question (right period, filter, and definition).
• 5. Assumptions surfaced — named and validated the hidden assumptions.
• 6. Reproducible — the steps can be redone and explained without the AI.
• 7. Ship-ready fit — right format, labels, rounding, and caveats.

Some of these checks can be compared with control totals or the source data. The usefulness of the broader rubric still needs validation with finance reviewers.

• Model: Gemini → Claude Haiku. Gemini's free tier capped at 20 requests a day and silently degraded the app to fallback content — invisible to users. Moving to pay-as-you-go Claude Haiku removed the cap and the silent failure.
• Schema-constrained responses. The application uses a defined response schema to make results easier to validate and render. A consistent structure does not guarantee a correct assessment.
• Model its own guardrail. Inferences the user did not state become [confirm: …] placeholders — a fix for a defining bug (below), not a nicety.
• Spreadsheets are parsed in the browser. Large sheets are reduced to a bounded summary — shape, per-column stats, and a few sample rows — and that summary is what the model receives, so token cost stays flat regardless of file size. The full file is not uploaded.
• Fake-door pricing. A paid tier is captured as an interest signal rather than built before demand is proven. No gamification.

Next.js and React on the front, with Claude Haiku on the server behind API routes (the seven-dimension check, the prompt coach, concept lessons, the mock review, and the welcome email). Each client helper degrades to authored fallback content if the model is unavailable, so no screen dead-ends. Spreadsheets are read client-side with SheetJS. Supabase stores insert-only events. PostHog handles analytics. It ships on Vercel. At the August 2026 build, Haiku responses measured roughly 2–10 seconds, so client timeouts were set to 30 seconds against Vercel's 60-second serverless ceiling.

While testing a quarterly-revenue example, the pre-flight coach invented data-specific filters — “exclude intercompany, exclude refunds” — as if it knew the user's ledger. It was committing the exact silent-assumption sin the product exists to catch.

The fix reshaped the product: anything the coach infers now becomes a [confirm: …] placeholder under “pin these down first,” never an asserted directive. The lesson: a coaching product has to model its own guardrail — don't let the coach do the thing it is teaching users to avoid.

• It does not do the work, and it cannot tell you a number is correct — it checks whether the process was sound.
• The rubric is directional (a first version), not yet confirmed against real finance work.
• Ratings do not yet persist to a queryable table.
• Email is sent from a Gmail sender without domain authentication, so some inboxes route it to Promotions.