The Machine That Cannot Know

Why AI Will Never Solve the Knowledge Problem

There is a parlour trick that large language models perform with extraordinary fluency. You ask a question. You receive an answer. The answer is grammatically perfect, often well-structured, sometimes even insightful-sounding. And so you conclude — as millions now conclude daily — that the machine knows something.

It does not.

What it does is produce syntax. It arranges tokens in sequences that are statistically likely given its training data. It is, in the most literal and technical sense, a machine that generates well-formed sentences without understanding a single one of them. This is not a metaphor. It is an architectural description. And the failure to grasp this distinction — between syntax and semantics, between the form of knowledge and the substance of it — is now distorting how we think about institutions, markets, governance, and the future of human coordination.

John Searle made a version of this argument with his Chinese Room thought experiment in 1980. A person in a room manipulates Chinese symbols according to rules, producing outputs that native speakers accept as fluent Chinese, without understanding a word. The room processes syntax. It does not possess semantics. Searle was arguing about hypothetical AI. We now have the actual thing — systems that process syntax at planetary scale — and the argument is no longer hypothetical. Every large language model is a Chinese Room. The room has merely become very fast and very large.

The consequences of this are not abstract. They are institutional, economic, and political.

I have spent the better part of two decades working at the intersection of computer science, economics, and law. I hold doctoral degrees across these fields. I have managed teams of PhD-level researchers and secured thousands of patents in cryptographic systems and blockchain technology. I say this not to brandish credentials but to establish that when I tell you the emperor has no clothes, I have spent enough time in the imperial wardrobe to know what I am looking at.

The argument I want to make is simple in outline and devastating in implication: artificial intelligence cannot solve what Friedrich Hayek called the knowledge problem, and every institutional arrangement premised on the assumption that it can — or soon will — is building on sand.

What Hayek Understood That Silicon Valley Does Not

In 1945, Hayek published “The Use of Knowledge in Society,” one of the most consequential essays in the history of economic thought. His central insight was that the knowledge required for economic coordination is not the kind of knowledge that can be centralised. It is dispersed across millions of individuals, embedded in particular circumstances of time and place, largely tacit, and often impossible to articulate even by those who possess it.

The shipper who knows that an empty vessel is available. The factory foreman who can feel when a machine is about to fail. The trader who senses a shift in market sentiment before it shows up in any data feed. This is the knowledge that makes economies function, and it is precisely the knowledge that no central planner — human or artificial — can aggregate.

The entire history of the socialist calculation debate, from Mises through Hayek to the collapse of the Soviet Union, is a demonstration of this principle. Centralised systems fail not because their planners are stupid but because the knowledge required for rational economic calculation is constitutionally uncentralisable. It exists only in distributed form, only in context, only in the lived experience of the actors who hold it.

Now consider the implicit promise of contemporary AI: that sufficiently powerful models, trained on sufficiently large datasets, can replicate or even surpass this distributed knowledge. That a foundation model with access to the world’s text can “know” what the economy needs. That algorithmic recommendation systems can replace the price mechanism. That centralised intelligence can coordinate better than decentralised markets.

This is the socialist calculation problem wearing a hoodie and raising venture capital.

Mises made the original argument in 1920: without private property in the means of production, there are no genuine market prices; without market prices, there is no rational economic calculation; without rational economic calculation, there is no way to allocate resources efficiently. Hayek deepened this by showing that the problem was not merely computational but epistemic. Even if a central planner had unlimited computational power, the relevant knowledge does not exist in a form that could be fed into any computer. It is not written down. It is not in any database. It lives in the heads, hands, and habits of millions of people who do not even know they possess it.

The tech industry’s answer to Hayek is, implicitly: “But now we have the data.” The entire premise of the AI revolution is that with enough data, enough compute, and enough parameters, the knowledge problem dissolves. This premise is wrong, and I want to explain precisely why, drawing on the most recent academic work in the field.

Five Recent Studies and What They Actually Show

The academic literature on AI’s epistemic limitations has matured considerably in the past five years. Five papers published in Humanities and Social Sciences Communications between 2020 and 2025 converge on a single conclusion from five different angles: AI does not know. It cannot know. And the gap between what it does and what knowledge requires is not a gap that more data or bigger models will close.

Ragnar Fjelland, writing in 2020, returned to Hubert Dreyfus’s critique of artificial intelligence — a critique that the AI community spent decades trying to bury and that keeps proving correct. Dreyfus argued, drawing on phenomenology, that human expertise depends on embodiment, on being physically situated in a world, on having passed through childhood, cultural immersion, and the slow accumulation of experience that no dataset can replicate. Fjelland examined the poster children of AI success — Deep Blue, Watson, AlphaGo — and showed that every one of them is narrow artificial intelligence. They do one thing. They do it within rigid boundaries. And the moment those boundaries are breached, they fail.

The most telling example is Watson Health. IBM poured billions into applying Watson to oncology, promising that AI would revolutionise cancer treatment. It did not. Oncologists found that Watson’s recommendations were often wrong, sometimes dangerously so, because the system could not replicate the situated judgment that comes from years of clinical practice. Watson could process the literature. It could not examine the patient. It could correlate symptoms with studies. It could not exercise the tacit knowledge that an experienced oncologist brings to a treatment decision — the kind of knowledge that Polanyi captured in his famous observation that “we can know more than we can tell.”

This is not a failure of implementation. It is a failure of category. The knowledge that Watson lacked is not the kind of knowledge that can be encoded in a training set, because it was never encoded in language in the first place.

Saurabh Bhardwaj, in a 2025 paper, extended this analysis by identifying four specific limitations of AI reason. First, AI cannot perform abductive inference — the Peircean “intelligent guessing” by which humans generate hypotheses from incomplete data. AI does statistical induction. It finds patterns in what it has seen. It cannot make the creative leap to what it has not seen. Second, AI fails with thin data and edge cases, precisely the situations where human judgment is most needed. Third, AI cannot acquire tacit knowledge, because tacit knowledge is acquired through embodied practice, not through data ingestion. Fourth, AI cannot reason by analogy or metaphor, which is how humans bridge between domains and generate genuinely novel understanding.

Bhardwaj invoked what he called the “Clever Hans” effect — named after the horse that appeared to do arithmetic by reading its handler’s unconscious body language. AI creates the illusion of understanding based on superficial cues. It responds to patterns in the question, not to the substance of the problem. This is not intelligence. It is an extraordinarily sophisticated form of mimicry.

Leszek Porębski and Grzegorz Figura, also in 2025, drove the point home from the architectural side. They examined the decoder-only architecture of large language models and pointed out what should be obvious but apparently is not: these systems are designed to generate text, not to comprehend it. A decoder-only model takes a sequence of tokens and predicts the next token. That is what it does. That is all it does. The fact that this process produces grammatically correct, contextually appropriate, sometimes brilliant-sounding text does not mean the system understands the text any more than a piano understands a sonata.

They coined a useful term: “semantic pareidolia.” Just as humans see faces in clouds and Jesus on toast, we project understanding onto systems that produce human-like output. We cannot help it. Our cognitive architecture is built to attribute intentionality to anything that behaves as if it has intentions. But the attribution is false. The machine is not thinking. It is autocompleting.

Porębski and Figura also noted the inconsistency problem. The same large language model, asked the same question in different contexts, will produce contradictory answers. If the system had genuine beliefs — genuine semantic content — this would be impossible. A being that believes P cannot simultaneously believe not-P. But a token-prediction engine can produce P in one context and not-P in another without any contradiction, because it has no beliefs at all. It has statistical distributions over token sequences. Those distributions shift with context. There is no underlying semantic state to be contradicted.

Jörg Noller, in a 2024 paper, offered a different but complementary perspective. He argued for understanding AI as an extension of human agency — not as a separate intelligence but as a tool that extends human purposes, much as a telescope extends human vision. The key insight was his insistence that databases are not epistemologically neutral. Every dataset encodes choices about what to include and exclude, how to categorise, what to privilege and what to ignore. These choices transport human biases, stereotypes, and assumptions into AI systems, which then amplify and systematise them.

This matters for the knowledge problem because it means that AI systems do not access some neutral, objective body of knowledge. They access a curated, encoded, inevitably partial representation of knowledge — a representation shaped by the purposes and limitations of those who constructed it. The map is not the territory, and the training set is not the world.

Finally, Moritz Renftle and colleagues, also in 2024, examined what Explainable AI algorithms actually do, stripped of the rhetoric. Their conclusion was sobering. Despite all the language about “interpretability,” “transparency,” and “trust,” XAI algorithms can answer exactly one question: “How can one represent an ML model as a simple function that uses interpreted attributes?” That is it. All the grander claims — that XAI generates trust, ensures fairness, delivers transparency — rest on an unexamined inference chain in which capabilities are assumed to produce goals without any demonstration that they do.

Renftle et al. identified two fundamental challenges. The approximation challenge: the simple, interpretable function that XAI produces is necessarily a lossy approximation of the actual model. You gain interpretability by sacrificing accuracy. The translation challenge: converting technical attributes (the features the model actually uses) into interpreted attributes (features humans can understand) is often impossible for complex models. The very features that make a model powerful are the ones that resist human comprehension.

This is the syntax-semantics gap expressed in formal terms. The model operates on technical attributes — pure syntax. The human needs interpreted attributes — semantics. And the translation between them is, for the most interesting and most consequential applications, intractable.

What Renftle and colleagues have done, whether they intended it or not, is provide the computer science proof of Hayek’s philosophical claim. The knowledge that matters for decision-making — the “interpreted” knowledge, the knowledge that connects to human purposes and human contexts — is exactly the knowledge that complex AI models cannot deliver. They can deliver technical outputs of extraordinary sophistication. They cannot deliver understanding. And the gap between output and understanding is not narrowing. As models become more complex, it widens.

Why This Demands Provenance, Not Prophecy

If AI cannot know — if it produces syntax without semantics, form without substance, pattern without understanding — then every claim it makes is, epistemically, ungrounded. Not necessarily false. But ungrounded. The machine cannot distinguish between a true claim and a false claim that has the same statistical profile in the training data. It cannot verify. It cannot check its work against reality. It can only check its output against the probability distributions it has learned.

This creates an institutional demand. If AI-generated content is going to proliferate — and it is, at scale — then we need mechanisms to verify the provenance and authenticity of information. We need to know where claims come from, who or what generated them, and whether they have been verified by processes that involve genuine understanding.

This is not a technical problem that AI itself can solve. You cannot use a syntax machine to verify semantics. You cannot use a pattern-matching engine to confirm truth. The verification must come from outside the system — from institutions, protocols, and human judgment.

The blockchain, properly understood, offers one such mechanism. Not as the speculative casino it has been turned into by people who understand neither the technology nor the economics, but as a system for establishing immutable, timestamped, publicly verifiable records of provenance. A system that does not require trust in any central authority because the verification is distributed — much as knowledge itself is distributed in Hayek’s economy.

The parallel is not accidental. The knowledge problem and the provenance problem have the same structure. In both cases, the critical information is dispersed, context-dependent, and resistant to centralisation. In both cases, the solution is not a more powerful central processor but a better-designed distributed system. And in both cases, the people promising that AI will solve everything are making the same mistake that central planners have always made: assuming that what cannot be centralised can be computed.

Consider what happens when AI-generated content enters a decision chain without provenance verification. A large language model produces a legal memorandum. The memorandum cites cases. Some of the cases do not exist — the well-documented “hallucination” problem. A lawyer relies on the memorandum without checking. The fabricated citations enter a court filing. This has already happened, multiple times, in jurisdictions around the world.

Now scale this. AI-generated financial analysis with fabricated data points. AI-generated medical summaries with invented studies. AI-generated policy briefs with hallucinated statistics. In each case, the output looks correct — the syntax is impeccable — but the semantic content is unreliable. And without provenance mechanisms that allow the recipient to trace every claim to its source, there is no way to distinguish the reliable from the fabricated.

The traditional solution to this problem is institutional trust: we trust the doctor because of the medical licence, the lawyer because of the bar admission, the journalist because of the editorial process. But AI-generated content bypasses all of these institutional filters. It arrives looking exactly like human-generated expert content, with none of the institutional scaffolding that made expert content trustworthy in the first place.

This is why provenance is not a nice-to-have but a structural necessity. As AI-generated content floods information ecosystems, the ability to verify the origin, processing history, and human oversight of any given claim becomes the critical infrastructure of epistemic integrity. Without it, the information environment degrades into noise — syntactically perfect, semantically meaningless noise.

The Stakes

I am not making an anti-technology argument. I use AI tools daily. They are useful for what they are — sophisticated text manipulation engines, pattern-recognition systems, productivity amplifiers. The problem is not with the technology but with the claims made on its behalf.

When venture capitalists claim that AI will replace doctors, lawyers, and economists, they are claiming that syntax can replace semantics. When governments propose AI-driven policy systems, they are proposing that statistical inference can replace situated judgment. When technologists promise artificial general intelligence by 2030 or 2035 or 2040, they are promising to solve a problem that Dreyfus identified as unsolvable in 1965, that Polanyi identified as unsolvable in 1966, and that Hayek identified as unsolvable in 1945.

The knowledge problem is not a bug. It is a feature of the world — a consequence of the fact that reality is more complex than any representation of it, that experience is richer than any encoding of it, and that understanding requires a kind of contact with the world that no machine, however sophisticated, can achieve.

Hayek understood this. Polanyi understood this. Dreyfus understood this. Wittgenstein, who observed that “if a lion could speak, we could not understand him” — because understanding requires shared forms of life, not merely shared grammar — understood this. The five papers I have discussed here, published in the last five years by researchers working independently across multiple continents, all converge on the same conclusion.

The conclusion is not that AI is useless. It is that AI is limited in a way that matters profoundly for how we organise institutions, govern economies, and maintain the epistemic integrity of public discourse. The limit is not one of scale or speed or data. It is one of kind. The machine operates on syntax. Knowledge requires semantics. And no amount of syntactic sophistication will bridge that gap.

What we need is not artificial intelligence. What we need is institutional intelligence — the wisdom to build systems that respect the limits of computation, that verify what AI produces, and that preserve the distributed, embodied, tacit knowledge on which every functioning society depends.

The price mechanism works not because it is computationally efficient but because it is epistemically honest. It aggregates dispersed knowledge into a signal — price — without requiring any central authority to understand or even access that knowledge. It is a system that respects the knowledge problem rather than pretending to solve it.

We need the digital equivalent. Systems that verify without centralising. Protocols that authenticate without requiring trust. Infrastructure that preserves provenance across an information ecosystem increasingly flooded with syntactically perfect, semantically ungrounded content.

The machine that cannot know can still be useful. But only if we stop pretending it knows.