Aspects of scientific method
A heritage of thought about the process of scientific learning comes to us from such classical writers as Aristotle, Galen, Grossteste, William of Occam, and Bacon who have emphasized aspects of good science and have warned of pitfalls.
Iteration between theory and practice
One important idea is that science is a means whereby learning is achieved, not by mere theoretical speculation on the one hand, nor by the undirected accumulation of practical facts on the other, but rather by a motivated iteration between theory and practice such as is illustrated in Figure A(1).
Matters of fact can lead to a tentative theory. Deductions from this tentative theory may be found to be discrepant with certain known or specially acquired facts. These discrepancies can then induce a modified, or in some cases a different, theory. Deductions made from the modified theory now may or may not be in conflict with fact, and so on. In reality this main iteration is accompanied by many simultaneous subiterations.
Flexibility
On this view efficient scientific iteration evidently requires unhampered feedback. The iterative scheme is shown as a feedback loop in Figure A(2). In any feedback loop it is, of course, the error signal–for example, the discrepancy between what tentative theory suggests should be so and what practice says is so—that can produce learning. The good scientist must have the flexibility and courage to seek out, recognize, and exploit such errors–especially his own. In particular, using Bacon’s analogy, he must not be like Pygmalion and fall in love with his model.
Parsimony
Since all models are wrong the scientist cannot obtain a “correct” one by excessive elaboration. On the contrary following William of Occam he should seek an economical description of natural phenomena. Just as the ability to devise simple but evocative models is the signature of the great scientist so overelaboration and overparameterization is often the mark of mediocrity.
Worrying selectively
Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.
Role of mathematics in science
Pure mathematics is concerns with propositions like “given that A is true, does B necessarily follow?” Since the statement is a conditional one, it has nothing whatsoever to do with the truth of A nor of the consequences B in relation to real life. The pure mathematician, acting in that capacity, need not, and perhaps should not, have any contact with practical matters at all.
In applying mathematics to subjects such as physics or statistics we make tentative assumptions about the real world which we know are false but which we believe may be useful nonetheless. The physicist knows that particles have mass and yet certain results, approximating what really happens, may be derived from the assumption that they do not. Equally, the statistician knows, for example, that in nautre there never was a normal distribution, there never was a straight line, yet with normal and linear assumptions, known to be false, he can often derive results which match, to a useful approximation, those found in the real world.
It follows that, although rigorous derivation of logical consequences is of great importance to statistics, such derivations are necessarily encapsulated in the knowledge that premise, and hence consequence, do not describe natural truth. It follows that we cannot know that any statistical technique we develop is useful unless we use it. Major advances in science and in the science of statistics in particular, usually occur, therefore, as the result of the theory-practice iteration.
The researcher hoping to break new ground in the theory of experimental design should involve himself in the design of actual experiments. The investigator who hopes to revolutionize decision theory should observe and take part in the making of important decisions. An appropriately chosen environment can suggest to such an investigator new theories or models worthy to be entertained. Mathematics artfully employed—The researcher’s purely mathematical ingenuity is likely to be exercised more, not less, by the fact of his dealing with genuine problems—can then enable him to derive the logical consequences of his tentative hypotheses and his strategically selected environment will allow him to compare these consequences with practical reality. In this way he can begin an iteration that can eventually achieve his goal. An alternative is to redefine such words as experimental design and decision so that mathematical solutions which do not necessarily have any relevance to reality may be declared optimal.