XP and ESP: The Truth is Out There!
By Ed Johnson
“Eclipses occur, and savages are frightened. The medicine men wave wands – the sun is cured – they did it.”– Charles Fort 1
People have a vast capacity for self-deception. Even members of the scientific community, from whom we expect objectivity, can unwittingly allow their personal beliefs and preconceptions to color their interpretation of data. Professional ambition and wishful thinking can turn their stance from one of neutral observance into passionate adherence to a position, sustained by willful ignorance of contrary evidence. Such attitudes are common amongst the ranks of pseudo-scientists and paranormal researchers. Enthusiasts in this domain reward these ersatz scientists by buying their books and journals in numbers proportionate to the impressiveness of the alleged experimental findings. In doing so, they become complicit in their own deception.
Many of these enthusiasts labor under the misimpression that the existence of ESP, PK and other paranormal phenomena has been “proved” by creditable scientists. Many of the researchers are similarly deceived.
Curiously, we may be seeing exactly the same effects currently at work in the software development community with regard to XP. If there is sufficient desire to find “evidence” favorable to XP, it will be found. If there is sufficient reward for publication of XP success stories, they will be published. The belief that XP has been “proved” in the field can develop, if there is sufficient desire to believe it. And if sustaining that belief makes it necessary to ignore conflicting evidence and censor stories of failure, then that will also occur.
Be it XP trials or ESP experiments, there are two sorts of bias that make it possible to find significance where there is none, and sustain false belief. This post examines how these biases manifest in both domains.
Positive Outcome Bias: Embrace Change Or Exaggerate Chance?
Positive outcome bias is defined as:
The tendency of researchers and journals to publish research with positive outcomes much more frequently than research with negative outcomes.2 Suppose 100 researchers conduct an experiment in ESP. Each professor chooses a single subject who believes they have ESP and asks them to “sense” a series of randomly chosen Zener cards being “sent” to them by the person who selects the cards. Suppose that in 50% of these experiments, the subject achieves an accuracy greater than that which could be attributed to chance alone. The 50 researchers conducting those experiments are intrigued, and decide to conduct a further round of tests with the same subject. The other 50 researchers, knowing that failed attempts to detect ESP are unlikely to get them published, abandon their experiments. In the next round of experiments, the same pattern occurs, and 25 more researchers give up. Eventually, all the researchers give up, but not before one has witnessed his subject beat chance in 6 or 7 consecutive experiments - which is quite a spectacular result! Deciding to neglect the final experiment that caused him to stop (figuring the subject was probably tired, anyway) the researcher writes up his results and sends them to the editor of the Journal of Parapsychology, in which they are published.
Consider the deception which results: • The PSI research community’s pro-ESP bias has been further confirmed by their receipt of this latest research evidence • The readers of the Journal of Parapsychology are impressed with the evidence, and any pre-existing belief in ESP is further cemented. • Other researchers, perhaps even some outside the PSI community, conclude “Maybe there’s really something to this ESP stuff after all” and decide to conduct their own experiments in ESP, thereby propagating the effect into another round of investigations.
Note that neither the researcher who was published, the research community, nor any of the readers of the Journal of Parapsychology ever become aware of the 99 experiments that were abandoned because they were deemed unpublishable. Taken in isolation, the published result may be impressive. But taken in the context of the other 99 experiments that have silently failed, the published result may simply be an outlier whose occurrence was actually quite likely.
The following factors contribute to positive outcome bias:
- Researchers who conduct uncontrolled experiments
- Researchers who self-censor negative results
- Researchers who can justify to themselves the imposition of optional starting and stopping conditions.
- A publication environment that favors success stories
All three of these are features of the environment in which the software development community examines and reports on your favorite methodology and mine, XP:
- XP is often trialed on a single project, on a non-comparative basis (controlled experimentation would be prohibitively expensive).
- When an XP project fails, it will probably fail quietly. Companies and individuals have reputations to protect.
- In a series of XP-related experiences, initial negative experiences are dismissed as “teething trouble”. For an example, see Laurie William’s pair programming experiment. Her dismissal of the last of four data sets, and devaluing of the first of those four data sets, is a good example of “optional starting and stopping conditions.”
- There can be no doubt that the IT media just loves those “XP saves the day” stories. Success stories sell magazines.
In such an environment, XP enthusiasts will declare “Wow, everywhere you look, XP is succeeding” – which is true. But it’s in the places that you haven’t looked that the real story lies.
Confirmation Bias
Confirmation bias is defined as:
The tendency to notice and to look for what confirms one’s beliefs, and to ignore, not look for, or undervalue the relevance of what contradicts one’s beliefs.
When it is pointed out to PSI researchers who claim to have successfully demonstrated ESP, that hundreds of non-PSI researchers have tried to replicate their results and failed, they sometimes attribute this to the ostensible influence that the attitude of both experimenter and subject can have over the results. An experimenter who is hostile towards the concept of ESP, they claim, can exert a negative influence over the results, thereby counteracting any positive ESP effects that may be present. This is one of the many “outs” PSI researchers have developed that enable them to attribute negative results to extraneous causes, and preserve only the data that is favorable to their preferred hypotheses.
We see exactly the same thing happening in the XP community’s evaluation of experience reports from the field.
When presented with a claim of success using XP, the community accepts it without challenge, for it is a welcome confirmation of pre- existing beliefs. However, a claim that XP has failed is an unwelcome affront to their personal convictions. So these claims are scrutinized until an “out” is found - some extraneous factor to which the blame for failure can be assigned. If all else fails, one can claim, as PSI researchers are wont to do, that the attitude of the participants is to blame for the failure.
To illustrate, consider the tabulation below of the four types of experience reports that the XP community can be presented with. The columns represent the two basic modes of XP usage – full and partial. Either you’re doing all the XP practices or you’re only doing some of them. The rows represent the claimants assessment of the project outcome – success or failure. The table shows the interpretation an XP proponent can confer upon each type of experience report so as to confirm their pre- existing belief in XP. Success Failure Full XP XP has succeeded! You weren’t doing xxx as well as you could have, or You weren’t committed enough, or There’s something wrong with you etc. Subset of XP See how powerful XP is? Even a subset of the practices can yield success! You weren’t doing all the practices, so you weren’t really doing XP. The XPers have all their bases covered. No matter what the experience report, there is no need to ever cast doubt upon XP itself – there are always rival causes to be blamed.3 In this way, XP becomes non-falsifiable.
Conclusion
There is an “essential tension"4 between being so skeptical of new technologies and methods that we miss the opportunity to exploit genuine innovations, and being so credulous that we are ourselves exploited by those willing to subjugate integrity to self-interest. Given the software industries’ history of fads, trends and passing enthusiasms, we would be wise to approach claims of innovation with caution – where those claims are accompanied by fanaticism and zeal, doubly so. As Thomas Henry Huxley warned:
Trust a witness in all matters in which neither his self-interest, his passions, his prejudices, nor the love of the marvelous is strongly concerned. When they are involved, require corroborative evidence in exact proportion to the contravention of probability by the thing testified.
There is no logical basis for dismissing out of hand every “next big thing” that comes along. But an awareness of confirmation bias, positive outcome bias and their contribution to the development of false beliefs should encourage us to seek evidence beyond that provided by popular media and effusive testimonial.
1 Cited in Voodoo Science, Robert Park, Oxford, 2000 2 The Skeptic’s Dictionary, Robert Carroll, Wiley, 2003 3 http://c2.com/cgi/wiki?IfXpIsntWorkingYoureNotDoingXp 4 Why People Believe Weird Things, M. Shermer, Owl Books, 2002