There are two answers to the question “Why is science so successful?” The most popular, and probably mistaken, answer is that science has a method for guiding the discovery process. The other answer, associated with Michael Polanyi and Paul Feyerabend, is that the success of science is due to the scientific community having a particular kind of social organization (works according to certain norms).
Willmoore Kendall describes these norms in the following way:
[T]he discussion process works in those situations in which men who are products of the tradition organize themselves for a serious venture in the pursuit of truth ... Such men demonstrably proceed on some such principles as these:
(a) The pursuit of truth is indeed forwarded by the exchange of opinions and ideas among many; helpful suggestions do indeed emerge sometimes from surprising quarters; but one does not leap from these facts to the conclusion that helpful suggestions may come from just anybody.
(b) The man or woman who wishes to exercise the right to be heard has a logically and temporally prior obligation to prepare himself for participation in the exchange, and to prepare himself in the manner defined by the community.
Moreover (c), from the moment he begins to participate in the exchange, he must make manifest, by his behavior, his sense of the duty to act as if the other participants had something to teach him-the duty, in a word, to see to it that the exchange goes forward in an atmosphere of courtesy and mutual self-respect.
Next (d), the entrant must so behave as to show that he understands that scholarly investigation did not begin with his appearance on the scene, that there is a strong presumption that prior investigators have not labored entirely in vain, and that the community is the custodian of-let us not sidestep the mot juste - an orthodoxy, no part of which it is going to set lightly to one side.
(e) That orthodoxy must be understood as concerning first and foremost the frame of reference within which the exchange of ideas and opinions is to go forward. That frame of reference is, to be sure, subject to change, but this is a matter of meeting the arguments that led originally to its adoption, and meeting them in recognition that the ultimate decision, as to whether or not to change it, lies with the community.
(f) The entrant, insofar as he wishes to challenge the orthodoxy, must expect barriers to be placed in his way, and must not be astonished if he is punished, at least in the short term, by what are fashionably called "deprivations"; he must, indeed, recognize that the barriers and the deprivations are a necessary part of the organized procedure by which truth is pursued.
(g) Access to the channels of communication that represent the community's central ritual (the learned journals, that is to say) is something that the entrant wins by performing the obligation to produce a craftsmanlike piece of work.
(h) The ultimate fate of the entrant who disagrees with the orthodoxy but cannot persuade the community to accept his point of view is, quite simply, isolation within or banishment from the community.
[The "Open Society" and Its Fallacies, The American Political Science Review, 54(4), 1960, pp. 972-979]
Lee Smolin describes the norms in the following way:
Science has succeeded because scientists comprise a community that is defined and maintained by adherence to a shared ethic. It is adherence to an ethic, not adherence to any particular fact or theory, that I believe serves as the fundamental corrective within the scientific community.
There are two tenets of this ethic:
1) If an issue can be decided by people of good faith, applying rational argument to publicly available evidence, then it must be regarded as so decided.
2) If, on the other hand, rational argument from the publicly available evidence does not succeed in bringing people of good faith to agreement on an issue, society must allow and even encourage people to draw diverse conclusions.
I believe that science succeeds because scientists adhere, if mperfectly, to these two principles. To see whether this is true, let us look at some of the things these principles require us to do.
* We agree to argue rationally, and in good faith, from shared evidence, to whatever degree of shared conclusions are warranted.
* Each individual scientist is free to develop his or her own conclusions from the evidence. But each scientist is also required to put forward arguments for those conclusions for the consideration of the whole community. These arguments must be rational and based on evidence available to all members. The evidence, the means by which the evidence was obtained, and the logic of the arguments used to deduce conclusions from the evidence must be shared and open to examination by all members.
* The ability of scientists to deduce reliable conclusions from the shared evidence is based on the mastery of tools and procedures developed over many years. They are taught because experience has shown that they often lead to reliable results. Every scientist trained in such a craft is deeply aware of the capacity for error and self-delusion.
* At the same time, each member of the scientific community recognizes that the eventual goal is to establish consensus. A consensus may emerge quickly, or it may take some time. The ultimate judges of scientific work are future members of the community, at a time sufficiently far in the future that they can better evaluate the evidence objectively. While a scientific program may temporarily succeed in gathering adherents, no program, claim, or point of view can succeed in the long run unless it produces sufficient evidence to persuade the skeptics.
* Membership in the community of science is open to any human being. Considerations of status, age, gender, or any other personal characteristic may not play a role in the consideration of a scientist's evidence and arguments, and may not limit a member's access to the means of dissemination of evidence, argument, and information. Entry to the community is, however, based on two criteria. The first is the mastery of at least one of the crafts of a scientific subfield to the point where you can independently produce work judged by other members to be of high quality. The second criterion is allegiance and continued adherence to the shared ethic.
* While orthodoxies may become established temporarily in a given subfield, the community recognizes that contrary opinions and research programs are necessary for the community's continued health.
...
I would call this kind of community, in which membership is defined by adherence to a code of ethics and the practice of crafts developed to realize them, an ethical community. Science, I would propose, is the purest example we have of such a community.
But it is not sufficient to characterize science as an ethical community, because some ethical communities exist to preserve old knowledge rather than to discover new truths. Religious communities, in many cases, satisfy the criteria for being ethical communities. Indeed, science in its modern form evolved from monasteries and theological schools — ethical communities whose aim was the preservation of religious dogma. So if our characterization of science is to have teeth, we must add some criteria that cleanly distinguish a physics department from a monastery.
To do this, I would like to introduce a second notion, which I call an imaginative community. This is a community whose ethic and organization incorporates a belief in the inevitability of progress and an openness to the future. The openness leaves room, imaginatively and institutionally, for novelty and surprise. Not only is there a belief that the future will be better, there is an understanding that we cannot forecast how that better future will be reached.
Neither a Marxist state nor a fundamentalist religious state is an imaginative community. They may look forward to a better future, but they believe they know exactly how that future will be reached. ... An imaginative community believes that the future will bring surprises, in the form of new discoveries and new crises to be overcome. Rather than placing faith in their present knowledge, its members invest their hopes and expectations for the future in future generations, by passing along to them the ethical precepts and tools of thinking, individual and collective, that will enable them to overcome and take advantage of circumstances that are beyond the present powers of imagination. Good scientists expect that their students will exceed them. ...
The scientific community is thus both an ethical and an imaginative community.
What should be abundantly clear from this description is that controversy is essential for the progress of science. My first principle says that when we are forced to reach a consensus by the evidence, we should do so. But my second principle says that until the evidence forces consensus, we should encourage a wide diversity of viewpoints. This is good for science — a point that Feyerabend made often, and I believe correctly. Science proceeds fastest when there are competing theories. The older, naive view is that theories are put forward one at a time and tested against the data. This fails to take into account the extent to which the theoretical ideas we have influence which experiments we do and how we interpret them. If only one theory is contemplated at a time, we are likely to get stuck in intellectual traps created by that theory. The only way out is if different theories compete to explain the same evidence.
Feyerabend argued that even in cases where there is a widely accepted theory that agrees with all the evidence, it is still necessary to invent competing theories in order for science to progress. This is because experiments that contradict the established view are most likely to be suggested by a competing theory and perhaps would not even have been conceived were there not a competing theory. So competing theories give rise to experimental anomalies as often as the reverse.
Therefore Feyerabend insisted that scientists should never agree, unless they are forced to. When scientists come to agreement too soon, before they are compelled to by the evidence, science is in danger. ...
Science moves forward when we are forced to agree with something unexpected. If we think we know the answer, we will try to make every result fit that preconceived idea. It is controversy that keeps science alive, keeps it moving. In an atmosphere filled with controversy over rival views, sociological forces are not enough to bring people into agreement. So on those rare occasions when we do come to consensus on something, it is because we have no choice. The evidence forces us to do so, even if we don't like it. That is why progress in science is real. ...
While the progress of science relies on the possibility of achieving consensus in the long term, the decisions an individual scientist makes as to what to do, and how to evaluate the evidence, are always based on incomplete information. Science progresses because it is built on an ethic recognizing that in the face of incomplete information we are all equal. No one can predict with certainty whether an approach will lead to definite progress or years of wasted work. All we can do is train students in the crafts that experience has shown to lead most often to reliable conclusions. After that, we must leave them free to follow their own hunches and we must make time to listen to them when they report back. As long as the community continually opens up opportunities for new ideas and points of view and adheres to the ethic that in the end we require consensus based on rational argument from evidence available to all, science will eventually succeed.
The task of forming the community of science will never be finished. It will always be necessary to fight off the dominance of orthodoxy, fashion, age, and status. There will always be temptations to take the easy way, to sign up with the team that seems to be winning rather than try to understand a problem afresh. At its finest, the scientific community takes advantage of our best impulses and desires while protecting us from our worst. The community works in part by harnessing the arrogance and ambition we each in some degree bring to the search. Richard Feynman may have said it best: Science is the organized skepticism in the reliability of expert opinion.
[The Trouble with Physics, chapter 17, “What Is Science?”, pp. 301-307]
