Occam's Razor

Positivism requires [39] that a theory does not contain any more postulates than are necessary to express, in compact form, the observations that have been made⁸. In positivism, theories with few postulates are preferred to theories with many postulates. This meant, for example, that before observations, such as [58,25] that of the photo-electric effect in metals, were made, which were incompatible with Newton's laws of classical mechanics, positivism would have favoured classical mechanics, which has [26] two independent postulates, over quantum mechanics, which needs [58] five postulates to specify the behaviour of mechanical systems. This principle is known as Occam's razor.

A typical implementation of Occam's razor is the form of hypothesis testing attributed to Cournot [11]. In this method, at any given time, one theory $T_n$, known as the null hypothesis, is accepted as correct. If the likelihood $P(E\vert T_n)$ of the evidence $E$, given the null hypothesis, drops below some small, critical value $\epsilon{}$, then the null hypothesis is abandoned in favour of the next theory in line, $T_{n+1}$. Occam's razor then appears as the ordering of theories, $T_n$ being a theory of $n$ postulates; as Anderson [3] explained, in his exposition of Bayesian analysis, the null hypothesis is the simplest available theory. This system of ordering theories is here called the positivist Occam's razor, in order to distinguish it, as MacKay [42,43] already has, from the form of Occam's razor which arises naturally in Bayesian statistics, through the Occam factor. The difference is that, while the positivist Occam's razor uses the prior probability distribution to punish theories for having many postulates, irrespective of their relationship to the evidence, the Bayesian Occam factor directly takes into account, without reference to number of postulates or interference in the prior probability distribution, any adaptation which has been made in a theory in response to evidence, as a reduction in the extent to which the same evidence can be used to argue in favour of the adapted theory.

This bears closer examination. From a Bayesian perspective, the boundary between believing in $T_n$ and believing in $T_{n+1}$ must occur where

$$P(T_n\vert E)=P(T_{n+1}\vert E)$$ (12)

$$\Rightarrow{}\frac{P(E\vert T_n)P(T_n)}{P(E)}=\frac{P(E\vert T_{n+1})P(T_{n+1})}{P(E)}$$ (13)

$$\Rightarrow{}P(T_{n+1})=\frac{P(E\vert T_n)P(T_n)}{P(E\vert T_{n+1})}\textrm{.}$$ (14)

In Cournot's [11] method, this boundary occurs where $P(E\vert T_n)=\epsilon{}$. It is reasonable to assume that the specific, non-adjustable $T_{n+1}$ under consideration is that with the highest likelihood, and further, that the evidence will not, at the stage where some $T_n$ is still being entertained, have pulled the likelihoods in all of the $n+1$ postulate theories away from unity, because the disjunction of all available $n+1$ postulate theories has a vastly greater ability to adjust to evidence than the disjunction of all available $n$ postulate theories. Therefore, $P(E\vert T_{n+1})=1$. Equation 14 then becomes

$$P(T_{n+1})=\epsilon{}P(T_n)\textrm{.}$$ (15)

Equation 15 provides a recursion relation, associated with Cournot's [11] implementation of the positivist Occam's razor, for the prior probability of a specific, non-adjustable theory of $n$ postulates. Once the number of available theories of any given number of postulates is known, it therefore gives the absolute value of the prior probability of a theory of $n$ postulates, as a function of $n$. The crucial point is not the form of this function, but the fact that positivism, implemented in this way, can be associated with a particular prior probability distribution at all. The assertion of correctness for a particular prior probability distribution places this implementation of positivism in a tradition known as logical probability, which conflicts (section 3.1) with modern Bayesian statistics.