On The Blunders Of Learning Theory

Once you start looking at AI through the lens of Recorded Intelligence, certain other conceptualizations in and around the field of AI begin to look ill-posed. In this essay I will explore the ideas and decades-long historical arc of one of these misconceptualizations of AI – the founding framing of Statistical Learning Theory.

I say “founding framing” here because the field of statistical learning theory has given us numerous useful conceptual tools and terms, and it has over its roughly 40-year history rolled with the punches and largely moved past the blunders of its initial conceptualization. However, this initial conceptualization and multiple decades of theoretical work focused obsessively on a type of proof called a “uniform convergence bound” does, to a Recorded-Intelligence-minded person, seem like one of those obvious-in-hindsight massive blunders.

The Founding of Statistical Learning Theory

In the mid 1980s, Leslie Valiant published a now-famous paper which has given rise to a certain mental and mathematical model of what machine learning is doing. This framing of ML is now known as the Probably Approximately Correct (PAC) learning framework. In this framework, we have some phenomenon that we model as a mathematical function. In the world of LLMs, this function would be the function mapping a sequence of words or actions to some probability of what word or action comes next. In PAC learning we also have a hypothesis class, which is a set of possible functions that serve as the palette with which we are going to paint an approximation of the function we want to model. In LLMs, this hypothesis class is often a specific transformer model architecture. PAC learning says we want some algorithm which can, by looking at data representative of our target function, can learn to select one out of the possibly-infinite set of hypotheses well. The definition of well here is where PAC learning gets its name – we probably arrive at a choice which is approximately as good as the best choice we could have landed on out of our hypothesis class.