Hayek on Prediction in the Social Sciences

Updated

From Hayek’s Nobel Memorial Lecture given at Stockholm (11 December, 1974):

“We cannot be grateful enough to such modern philosophers of science as Sir Karl Popper for giving us a test by which we can distinguish between what we may accept as scientific and what not – a test which I am sure some doctrines now widely accepted as scientific would not pass. There are some special problems, however, in connection with those essentially complex phenomena of which social structures are so important an instance, which make me wish to restate in conclusion in more general terms the reasons why in these fields not only are there only absolute obstacles to the prediction of specific events, but why to act as if we possessed scientific knowledge enabling us to transcend them may itself become a serious obstacle to the advance of the human intellect.

The chief point we must remember is that the great and rapid advance of the physical sciences took place in fields where it proved that explanation and prediction could be based on laws which accounted for the observed phenomena as functions of comparatively few variables – either particular facts or relative frequencies of events. This may even be the ultimate reason why we single out these realms as ‘physical’ in contrast to those more highly organized structures which I have here called essentially complex phenomena. There is no reason why the position must be the same in the latter as in the former fields. The difficulties which we encounter in the latter are not, as one might at first suspect, difficulties about formulating theories for the explanation of the observed events – although they cause also special difficulties about testing proposed explanations and therefore about eliminating bad theories. ….

A simple example will show the nature of this difficulty. Consider some ball game played by a few people of approximately equal skill. If we knew a few particular facts in addition to our general knowledge of the ability of the individual players, such as their state of attention, their perceptions and the state of their hearts, lungs, muscles etc. at each moment of the game, we could probably predict the outcome. Indeed, if we were familiar both with the game and the teams we should probably have a fairly shrewd idea on what the outcome will depend. But we shall of course not be able to ascertain those facts and in consequence the result of the game will be outside the range of the scientifically predictable, however well we may know what effects particular events would have on the result of the game. This does not mean that we can make no predictions at all about the course of such a game. If we know the rules of the different games we shall, in watching one, very soon know which game is being played and what kinds of actions we can expect and what kind not. But our capacity to predict will be confined to such general characteristics of the events to be expected and not include the capacity of predicting particular individual events.

This corresponds to what I have called earlier the mere pattern predictions to which we are increasingly confined as we penetrate from the realm in which relatively simple laws prevail into the range of phenomena where organized complexity rules. As we advance we find more and more frequently that we can in fact ascertain only some but not all the particular circumstances which determine the outcome of a given process; and in consequence we are able to predict only some but not all the properties of the result we have to expect. Often all that we shall be able to predict will be some abstract characteristic of the pattern that will appear—relations between kinds of elements about which individually we know very little. Yet, as I am anxious to repeat, we will still achieve predictions which can be falsified and which therefore are of empirical significance.

Of course, compared with the precise predictions we have learnt to expect in the physical sciences, this sort of mere pattern predictions is a second best with which one does not like to have to be content. Yet the danger of which I want to warn is precisely the belief that in order to have a claim to be accepted as scientific it is necessary to achieve more. This way lies charlatanism and worse. To act on the belief that we possess the knowledge and the power which enable us to shape the processes of society entirely to our liking, knowledge which in fact we do not possess, is likely to make us do much harm. In the physical sciences there may be little objection to trying to do the impossible; one might even feel that one ought not to discourage the over-confident because their experiments may after all produce some new insights. But in the social field the erroneous belief that the exercise of some power would have beneficial consequences is likely to lead to a new power to coerce other men being conferred on some authority. Even if such power is not in itself bad, its exercise is likely to impede the functioning of those spontaneous ordering forces by which, without understanding them, man is in fact so largely assisted.”
F. von Hayek, Nobel Memorial Lecture, Stockholm,11 December 1974
http://www.nobelprize.org/nobel_prizes/economic-sciences/laureates/1974/hayek-lecture.html

There is a curious mix of good ideas and bad ones here.

First, the good ideas which Post Keynesians share: in the social sciences and economics, it is often not possible to make quantitative predictions of the type made in the natural sciences.

Nevertheless, it is still possible to make predictions in economics which are qualitative (or, as Hayek says, “pattern predictions”) and empirical, which can be verified or falsified.

But this passage seems to show that Hayek’s ultimate view of uncertainty is an epistemological, not an ontological, one:

“A simple example will show the nature of this difficulty. Consider some ball game played by a few people of approximately equal skill. If we knew a few particular facts in addition to our general knowledge of the ability of the individual players, such as their state of attention, their perceptions and the state of their hearts, lungs, muscles etc. at each moment of the game, we could probably predict the outcome.”

If Hayek is saying here that the natural sciences could exactly predict everything about the game given enough information (including presumably the brain states of human players), then he is committed to the view that the world is completely deterministic, and that our uncertainty about it is merely epistemic and caused by the insuperable difficulties of gathering enough information for calculations.

In contrast to this, Post Keynesians emphasise the ontological nature of uncertainty, and this commits them to a philosophical position quite different from that of Hayek.

The notion of “spontaneous ordering forces” that bring order to markets (like Smith’s “invisible hand” metaphor) seems overrated too.

“Spontaneous ordering forces” must be understood as emergent properties. That complex social and economic systems can display emergent properties that result in greater stability is not in doubt, but other emergent properties (e.g., the outcome of the paradox of thrift or distress selling in a market crash) can also be highly deleterious and destabilising. Hayek badly neglected such destabilising forces in his rhetoric about markets.