Hat’s off to WINE magazine for publishing the judge’s scores for Thursday’s Chenin Challenge as it shines a precise statistical light onto the problem raised publicly by Remington Norman MW in November, that tasting panels don’t know what they’re doing.

Two judges, one, like Rem, a Master of Wine and the other the Proprietor of a Wine Judging Academy were clearly singing from different hymn sheets as on the 14 wines whose scores were published, they achieved a most remarkable negative correlation of -0.77. In plain English, the wines the Master loved the Proprietor hated and vice versa. Such consistent divergence of opinion makes including their scores in an averaging process with the other three judges worthless and indeed throws question marks at the whole rating methodology.

Differences of opinion at the Chenin Challenge

The Challenge winner (wine #2 above, the Rex Equus 2008) has the worst discrepancy between Master and Proprietor. To show how futile the process is, we can reconstruct one judge’s scores from the other by the simple formulas:

Master = 29.32 – 0.8*Proprietor
Proprietor = 28.25 – 0.74*Master

with the negative weights confirming the foolishness of the enterprise. If the two people in question were not powerful opinion formers it would be hilarious. As it is, it is simply sad and another small tragedy for Chenin and SA wine.

The “best” judge (with criterion highest correlation with the scoring mean) was the only Cape Wine Master in the line-up, which should please the Cape Wine Academy no end. Interestingly enough, the Proprietor (who is also a wine importer and retailer) correlates with published retail prices at 0.4 while the Master has a negative correlation of -0.4 with price. A glorious situation to be in as you prefer cheaper wines to more expensive ones.

The “best judge” correlated with price to the tune of an impressive correlation coefficient of 0.6 so in future the Chenin Challenge will be well advised to let Ginette de Fleuriot do the judging all on her own.