Why did judges score the same dish differently?
A field-study protocol and a worked hypothetical example for interpreting score differences.
Study protocol · Ahmad Issam Maadarani
Data status · The figures below are hypothetical. Field results await anonymised data cleared for publication.
Question and unit of analysis
Does a score gap recur for a particular judge, or does it appear with particular portions or serving times? The unit is one judge’s score for one identifiable portion under a stated criterion. Individual criteria are recorded before the overall score is calculated.
Session design and collection
Code portions drawn from the same batch with similar serving weights. Record serving order, waiting time and serving temperature where it can be measured. Vary sample order across judges where possible. Judges record scores independently and the original notes are retained before discussion.
Planned analysis
Calculate the range between the highest and lowest scores for each sample and criterion, then compare patterns across samples. Show missing scores as missing. Investigate notable gaps against judges’ notes and service records before proposing an explanation. Describing a gap and establishing its cause are separate tasks.
Reading the example and its limits
Sample A has a range of 8 − 6 = 2 points. Sample B has a range of 8 − 7 = 1 point. These figures describe dispersion in the example only. Real differences could relate to use of the scale, variation between portions or service conditions. Field findings will be reported after authorised data and recorded specialist review are available.
| Judge | A | B |
|---|---|---|
| 1 | 8 | 7 |
| 2 | 7 | 8 |
| 3 | 6 | 7 |
Preparing the first field session
- Agree on the criteria and scale before the session and define each criterion consistently. Record permission to use the data before publication.
- Start with an operational pilot of three judges and two coded samples. Use random judge and sample codes, and vary serving order where possible.
- Enter one row per judge, sample and criterion. Record the score and rationale before discussion. Record wait in minutes and temperature in Celsius where measured. Leave unavailable measurements blank.
- Check for duplicate and missing scores and calculate the score range for each sample and criterion. Use this pilot to test the workflow. Generalisable findings require a broader design.