Does AI idea scoring work? We measured our own and published the number.
By Eli Fayerman, attorney and software engineer · Last updated
We scored the same 25 business ideas twice, with the same code and the same prompts. The rank correlation between the two runs was -0.05, which is statistically indistinguishable from no relationship at all. In plain terms: the number was meaningless. A user who saw “6.2” reasonably read it as a winnable idea, but a rerun of the same idea could just as easily have said 2.4. That is worse than learning nothing: it is confidence built on a dice roll. This page is the full account: how we caught it, the fix that worked partially, the fix that made things worse, and what we ship now instead of a decimal.
Why test your own scoring at all?
Our scanner assigns each business idea a winnability rating: can a solo builder with no audience actually capture this, separate from whether demand exists. When we wrote publicly that this rating had judged none of our own 3,400 accumulated ideas an easy win, the most common pushback was the right one: is that score truth or vibes? Freeze the model, run a blind test, publish the result.
The product's whole premise is telling you the truth about an idea. That obligation points inward too. So we built the test.
How do you test an AI-generated score?
We started with a blinded desk check: 25 ideas banded top, middle, and bottom by score, judged pass or fail by a human who never sees the numbers, with the statistics fixed in code before any verdict existed so the analysis could not be tuned after seeing the result.
That test never got judged, because preparing it surfaced a simpler and more damning question first: does the score even agree with itself? Before asking whether a score matches human judgment, ask whether it is reproducible at all. Same inputs, same code, run twice. This is test-retest reliability, and it is the cheapest honest question you can put to any AI-generated number.
What did the measurement show?
| Measure | Result |
|---|---|
| Rank correlation between two identical runs | -0.05 |
| Average score change per idea | 2.16 points (on a 0-10 scale) |
| Largest single change | 6.0 points |
| Ideas keeping their band (top / middle / bottom) | 24%, worse than random assignment |
The arithmetic behind the score is deterministic, so none of this came from the formula. It came from the model choosing different categorical judgments each run: whether a funded incumbent owns the lane, what kind of moat winning requires. One flipped judgment moved the score by up to four points. The product had been presenting one draw from that distribution as a measurement, to two decimal places, for six weeks. Nobody caught it because nothing ever scored the same idea twice.
Can you fix an unstable score by sampling?
Partially. We changed the pipeline to ask the validation question several times in parallel and keep the majority answer:
| Samples | Between-run correlation | Band agreement |
|---|---|---|
| 1 | -0.05 | 24% |
| 3 | +0.38 | 45% |
| 5 | +0.47 | 49% |
Going from one sample to three bought most of the available gain. Three to five bought almost nothing, and the reason is the diagnosis: at five samples, the typical winning answer took only two of five votes. The model rarely agrees with itself. Extra draws settle on its most common answer; they do not make that answer any more grounded. Sampling turned an unusable number into a usable label, not into a precise score.
Do stricter prompt instructions help?
No. We tested the obvious theory, that the instability came from vague wording, by rewriting the judgment prompts as strict ordered decision rules. The result was worse on every measure that matters: correlation dropped from +0.38 to +0.14, and the share of ideas where the samples could not reach any majority rose from 31% to 47%. More procedural instruction produced more disagreement, not less. We archived that prompt with its numbers and kept the simpler one.
What we changed in the product
- The decimal is gone. Winnability now reads as a label: Winnable, Contested, or Hard to win. Measured on the same test set, labels keep their value across runs 65% of the time against a 45% chance baseline, rising to 84% when the samples agree. A label is the finest resolution the underlying signal supports.
- Disagreement is displayed, not averaged. When the repeated samples cannot agree on the signals at all, the card says "Signals disagreed" instead of laundering the disagreement into a confident-sounding word.
- The reasoning is a sentence, not a score. Each rating carries a plain-language line, for example: "Why: free or built-in options already do this for $0; no funded player owns this exact lane; winning needs an audience a new entrant lacks."
- The facts stay strict. Competitor names and prices are the part of the output that can be checked, so we check them: any price the pipeline cannot attribute to a real retrieved source is shown as "not found" rather than displayed as fact.
What this means if you use any idea-scoring tool
Ours or anyone else's, the same three questions apply:
- Ask for test-retest numbers. If the same idea scored twice gives different answers, the precision on screen is decoration. A vendor who has never run the same input twice has not measured their own product.
- Separate facts from judgments. Named competitors with checked prices are verifiable. A synthesized 0-10 judgment is one draw from a distribution you cannot see. Weight them accordingly.
- Prefer tools that show uncertainty. An honest "we could not settle this one" is more useful than a confident number that changes on reload.
Related questions
Is the demand score affected too? The demand score is computed from different inputs (research evidence, pricing signals, viability checks) and was not the subject of this test. The instability we measured was specific to the winnability judgment, which depends on a small number of categorical calls with large point swings.
Why publish a result that makes your own product look bad? Because the alternative is shipping a number we know is meaningless and hoping nobody checks. The scanner's value is that it tells you the truth about ideas, including ours. See how the research pipeline works and the build-in-public roadmap for the rest of the checking layer.
Will better AI models fix this by 2030? Partially, and the partial matters. Consistency will become cheap: more samples, steadier models, and snapshotted retrieval can make a score return the same answer every time. But a consistent score is a stable record of what the model thinks, not proof that it is right. Whether any winnability judgment predicts real outcomes requires outcome data that accumulates over years, and the target itself moves as markets open and close. Expect future scores to look more trustworthy without necessarily being more true, which makes the test-retest question more important to ask, not less.
What would change the rating's reliability? Nothing we can promise. The theoretical fix is outcome data: which scanned ideas people actually pursued and how those bets went. We do not have it, users do not report it, and no idea-scoring tool has it at any useful scale. So we matched the display to what we measured and left it there.
The scores are honest about their limits. The competitor research is checked against real sources. Run one and read it yourself.
Run a research-backed scan →