The way science is usually taught in schools, one would think that knowledge grows at a linear rate, always accumulating, always going forward. However, a cursory look at the history of science shows that the truth couldn't be more different: the road toward knowledge is crooked and, sometimes, may even go backward when a wrongheaded notion persists well beyond its time.
This may happen for many reasons, from political censorship — as in the case of Lysenkoism, the notion promulgated by Trofim Lysenko during Stalinist years in the Soviet Union whereby acquired characteristics of vegetables could pass to future generations — to inbuilt ideologies in the scientific class, the insistence by influential members that a new idea can't be right.
In fact, the strategy of presenting science in a sort of triumphant mode takes away what may be one of its great attractions to people without their own scientific curiosity: the human drama.
The notion that scientists are always eager to receive new knowledge and drop the old is na´ve. Case in point, the famous Michelson-Morley experiment of 1887 was designed to measure the motion of the Earth along the aether, the medium everyone believed permeated space and propagated light waves. Just as sound waves need air to propagate, it was suggested that light waves need some sort of medium to support their propagation, the aether.
So, Albert Michelson and Edward Morley set up their apparatus, designed to detect the aether drift, something similar to the air speed you feel when you ride a bicycle. (Assume there is no wind that day.) If you throw a ball with the same speed in the direction of your motion or perpendicular to it, the ball will have a different final speed. For this experiment the ball was a light ray, propagating at the speed of 186,282 miles per second. (The apparatus was a bit more complex, but we can skip the details.) Well, the duo rotated their apparatus and, to their shock and amazement, there was absolutely no change in the speed of light.
The scientific community was completely mystified. There were many attempts to explain the results. In 1892, the great Dutch physicist Hendrik Lorentz wrote to Lord Rayleigh:
"I am totally at a loss to clear away this contradiction, and yet I believe that if we were to abandon Fresnel's theory [of the aether], we should have no adequate theory at all. ... Can there be some point in the theory of Mr. Michelson's experiment which has as yet been overlooked?"
A few years later, Lorentz proposed a "fix" to explain away the results of the experiment. He assumed that the arms of the apparatus could shrink in the direction of the motion. Ironically, Lorentz was right for the wrong reasons. The shrinking really is there, but the aether is not, as Einstein made clear in his 1905 paper on the special theory of relativity.
Faced with a shocking experimental results, scientists try first to accommodate it within the accepted framework. To let go of cherished ways of thinking is very difficult, and sometimes painful. Even in his last paper, written in 1927, Michelson still had difficulty letting the aether go: "Talking in terms of the beloved aether (which is now abandoned, though I personally still cling a little to it) ... "
When should an idea be abandoned? That is a hard question to answer, especially in cases where there are multiple lines of observation that are consistent with each other.
The recent observations casting doubt on the existence of dark matter in our galaxy is an example (see Adam's post here a few days ago). Astronomers found results which are in direct contradiction with the general expectation that our galaxy is surrounded by a nearly spherical halo of Dark Matter that we can only perceive through their gravitational attraction on normal matter. Dark Matter is supposed to be made of something different from the protons and electrons we are made of, although we don't know what.
Efforts to detect them here on Earth have so far failed. If the astronomical observations are correct they would explain this: there hasn't been a detection because the particles are not around. (Or they remain out of experimental bounds. There is much that is invisible to the eye or to our tools of exploration.)
But Dark Matter particles are consistent with other observations, for example, of the motion of galaxies in galactic clusters (groups of many galaxies held together by their mutual gravity) and by observations of the cosmic microwave background, the echo of microwave radiation left over when the first hydrogen atoms formed some 13.7 billion years ago.
So we seem to have a crisis, which is not a bad thing in science. Is it possible that Dark Matter doesn't exist? Until we detect it, yes, it's possible. Or perhaps we need to modify our theory of gravity? Also possible, and there have been many different attempts, although they always seem to run into one or another problem.
Either way, what does seem clear is that we are in for some sort of surprise that will force us into a new mode of scientific thinking about the universe. It will be interesting to see how hard will it be to accept evidence for some radically new idea in the face of current ideology.
You can keep up with more of what Marcelo is thinking on Facebook.