How to Grade an AI Deal Screener Against the Deals Your Team Actually Pursued
A deal screener once scored at zero the one deal the team went on to sign. That single fact changed how I think about grading one. Here is the method: grade against real decisions, per criteria version, starting with your best deal.
On this page
CMore Flo
Grade against real decisions
Real Estate
How do I know if an AI deal screener is any good?
Grade it against the decisions your team already made. Take every deal you signed and every deal a person moved into the pipeline, and check how the screener scored each one under the criteria version it used. A screener that scores your best deals low is missing what your team sees, whatever its average looks like.
I have been building deal screening with Dallas Whitaker of Carter Funds, a Tampa commercial real estate firm. Deals arrive by email, a screening step scores them against the fund's written criteria, and a person moves the promising ones into the pipeline.
When I went back through the history, one fact stood out. In July the screening step had scored one deal at zero. It was the deal the team went on to sign. The only signed agreement in the system, and the screener would have buried it.
That miss is the most useful test result the screener has produced, and it shaped the plan for grading it.
Grade against the deals your team acted on
Use real decisions as the answer key: every signed deal and every deal a person moved forward. If the screener would have buried one of them, that is the most important number you have.
The usual instinct is to build a test set: a pile of deals, labeled good or bad by someone, and a score for how often the screener agrees. The problem is that the labels are opinions made after the fact.
A better answer key already exists: what the team actually did. Every signed agreement, starting with that one, and every email a person chose to move into the pipeline. Those are real decisions, made with money and time on the line. The first grading question is simple: of the deals your team acted on, how many would the screener have surfaced?
In the language of classifiers, that is recall, the share of real positives the model catches, and Google's machine learning course says to favor it when a false negative costs more than a false positive. In deal flow it almost always does.
Compare
Where it comes from
What it tells you
Signed deals vs their scores
Your signed agreements and the screener's log
Whether it would have buried your best deals
Pipeline moves vs their scores
Emails a person moved into the pipeline
Whether it ranks what your team chose to read
Scores by criteria version
The version recorded with each score
Whether a change to the criteria helped or hurt
High scores nobody pursued
Top-scored deals with no pipeline move
What it values that your team does not
A grading sheet built from decisions the team already made. No deal names or counts shown.
Grade each criteria version separately
Criteria change as the fund learns. Record which version scored each deal, and grade versions separately, or an old miss and a new fix blur into one average that describes neither.
The fund's criteria are not fixed. They get written down, tried, adjusted. If every score is graded as one pile, a miss under last spring's criteria and a fix under this month's blur into one average that describes neither.
So each score is kept with the version of the criteria that produced it, and each version gets its own grade. The criteria are the fund's; the method of grading a screener against them is the part I bring.
Ask what made the deal worth moving on
The fix for a missed deal is usually a criterion nobody wrote down. Ask the person who pursued it what they saw, and decide with the fund whether it belongs in the next version.
A score of zero on a signed deal usually means the team saw something the criteria never said. So the one question for Dallas was simple: what made that deal worth moving on?
The answer becomes a candidate criterion, written in the fund's words. The fund decides whether it goes into the next version, and the grading shows whether it helped.
What stays human, and when not to bother
The screener ranks the reading; it does not decide. A person still opens the deal, and the fund still owns the criteria and every investment call. The screener gives no investment advice.
If your team signs a handful of deals a year and reads every email anyway, a screener may not be worth grading yet. Start by writing the criteria down. You will learn more from that than from any score.
Common questions
Answers to what people ask.
What is an AI deal screener?
A system that reads incoming deal flow, usually offering memos and broker emails, and scores each deal against the fund's written buying criteria so the team reads the most promising ones first. A person still decides which deals move forward.
What data do I need to grade a deal screener?
The deals your team actually acted on: every signed deal and every deal a person moved into the pipeline, plus the score the screener gave each one and the version of the criteria it used at the time.
Is a missed deal worse than a false alarm?
Usually. A false alarm costs a few minutes of reading. A missed deal costs the deal, because nobody opens an email the screener buried. Measure how many of your real deals it catches before you care about anything else.
Who should own the screening criteria?
The fund. The criteria are the fund's investment judgment, written down and versioned. The consultant brings the method of grading the screener against them, and the fund decides when the criteria change.
Chris Moreno builds custom AI systems for business operations. His writing draws on the work behind these systems: intake, follow-up, document workflows, and the checks that keep people in control.
The approach behind this work follows the four phases of Flo OS, our operating methodology for turning messy business workflows into systems that run cleanly and compound over time.
