The Amazing Wozzlescope

clipped from chicagoboyz.net Effective peer review requires that the reviewers have a deep familiarity with the instruments, protocols and procedures used by the experimenters. A chemist familiar with the use of a gas-chromatograph can tell from a description whether the instrument was properly calibrated, configured and used in a particular circumstance. Today, each instance of custom-written scientific software is like an unknown, novel piece of scientific hardware. Each piece of software might as well be an “amazing wozzlescope” for all that anyone has experience with its accuracy and precision. No one can even tell if it has subtly malfunctioned. As a result, the peer review of scientific software does not indicate even a whisper of the same level of external objective scrutiny that the peer review of scientific hardware indicates. How did we let this problem develop? I think it was simply a matter of creeping normalcy.