How Do You Tell If AI Research Is Any Good?

Count how many of your sources actually disagree with each other, and check how long anything they describe has been running. Those two questions catch most bad research, and neither takes more than a few minutes. Volume is not the test. Fifteen sources that all read the same original are one opinion wearing fifteen hats, and it feels exactly like consensus from the inside.

The Failure Looks Identical To Success

I was deciding whether to build a private AI knowledge system, the kind with a database and a graph and a few moving parts you maintain forever. I gathered fifteen sources before committing: the person who originated the approach, an engineering blog from a real company, two critics, and ten practitioner build videos.

Every one of them was positive. That is a properly assembled source list by any normal standard, and I would have defended it to anyone who asked. So I took the question as settled and started planning the build.

It was not consensus. It was one six month old idea being passed around, and every source I had was downstream of the same original. Nothing about the research process was going to reveal that, because the process was measuring how much I had read rather than how many separate opinions I had reached.

Test One: Count Independent Sources, Not Sources

Open your source list and ask a blunt question about each one. Would this person have reached this conclusion if the original article did not exist?

Most of the time you cannot answer that with certainty, which is itself informative. What you can usually see is whether the sources share vocabulary, share a diagram, or share the same three examples. When ten people illustrate a concept with the same example, they read the same thing.

The number that matters is not fifteen. It is how many genuinely separate positions you have found, and in my case the honest answer was one, plus two critics I had noted and moved past because the majority disagreed with them.

Test Two: Ask How Long It Has Been Running

Anything genuinely useful accumulates a track record. Anything new accumulates enthusiasm. Those look the same in a search result and completely different at twelve months.

So ask directly, of any approach you are considering: who has been running this for a year, and what happened? If nobody in your source list can answer that, you are not looking at a proven method. You are looking at a proposal that a lot of people found interesting.

When I applied this to my fifteen sources, only two of the systems described had any record of running past twelve months. Everything else was a build video, an announcement, or a description of an intention.

The Change That Actually Fixed It

Neither test helps if your research keeps returning the same body of writing, and mine was. The fix was one sentence, and it is the most useful thing in this article.

I stopped asking for information about the thing and started asking for postmortems about it. Abandonment stories. Forum complaints. People writing about why they gave up on it. Same tool, same topic, same afternoon.

Forty-nine sources came back instead of fifteen.

The material was there the whole time. It was not hidden and nobody was concealing it. It simply is not what you get back when you ask for information, because the writing that exists about any new technique is overwhelmingly written by the people introducing it.

What The Forty-Nine Said That The Fifteen Did Not

Three findings changed the decision, and none of them appeared in the original research.

Only two of these systems had any record of running past twelve months. One person's index file quietly hit a size limit and cut itself off, and their AI read the incomplete version for weeks without ever mentioning it was incomplete. And when somebody measured error rates properly, two out of three wrong answers were not the AI inventing things. They were the AI faithfully repeating something wrong that was already filed.

That last one is the finding I think about most. The failure mode of a knowledge system is not that it makes things up. It is that it remembers your mistakes perfectly and repeats them with total confidence, which is much harder to notice than an obvious invention.

I Did Not Build It

I kept the plain text files I already had. No database, no graph, nothing running in the background. Eleven client projects run on them, they cost nothing, and there is so little there that there is very little to break.

That was not the answer I expected or particularly wanted. I had gone into the exercise braced to find out I was years behind, and the finding was that I should stop reading and get back to work.

The Two Questions, To Keep

Before you act on any AI research, including research you did carefully:

How many of these sources would have reached this conclusion independently? And who has been running this for a year, and what happened to them?

If the answers are "I cannot tell" and "nobody", you have found enthusiasm rather than evidence. That is not useless, but it is not a reason to spend money or months.

Related reading: how to verify AI output before you act on it covers the next problem along, which is checking the work once you have decided the source is worth trusting. And how to make AI remember your business between sessions is about the notes you keep afterwards, where the same failure shows up in your own files.

---