Monitoring the agreement of sensory panelists

Tuesday, November 29, 2016

László Sipos, Márta Ladányi, Viktor Losó, Zoltán Kókai, Attila Gere


Keywords: consensus, sum of ranking differences (SRD) method, Page test, Cabilio-Peng multiple comparisons procedure


1.            Summary

It is an important theoretical question of sensometrics, as well as a practical issue of accredited laboratories, how to monitor and analyze the development of the agre­ement/consensus of the sensory evaluation group (panel), regarding the series of measurements in a given period. For the evaluation of the consensus of the individual judgements, the sum of ranking differences (SRD) method is a good evaluation alter­native. The difficulty in analyzing rank sum patterns lies in the fact that the pairwise significant difference method for rank sum data was developed only a few years ago. In the present work, a combination of methods is recommended for tracing the con­sensus of sensory panelists.


1.            Introduction and literature review


When monitoring performance, what is tested is whether the sensory judge or the sensory evaluation group (panel) is capable of recognizing, identifying and measuring the given property, as well as using and interpreting it the same way as other judges or evaluation groups (panels) [1]. Testing and improve­ment of trained and expert panelists, and the meas­urement of their performance can typically be real­ized through a standardized, multi-stage system that is based on feedback and realized under standard­ized conditions, preferably with some kind of soft­ware support.

Performance evaluation methods of sensory tests are classified in the literature in several ways. The most common classification is based on the number of people performing the tests (a single panel member, a full panel or panels), but it can also be based on the mathematical method used (single or multivariate statistical methods). Sensory assessors are classi­fied into three categories by the literature according to their training: naive / consumer panelists, trained panelists, expert panelists. The application of pan­elists with different training is necessary for different types of tasks, and performance evaluation methods are usually applied to tests performed by trained or expert panelists [2], [3], [4].

The solution of the quality management systems of testing laboratories accredited for sensory testing is the application of preventive and corrective actions, the task of which is the identification, prevention and elimination of non-conformities, i.e., of panelists evaluating differently from the group average. To do so, it is advisable to analyze the averages and medi­ans of raw data on a box-and-whisker plot and, fur­thermore, to perform the most important descriptive statistics (minimum, maximum, standard deviation, kurtosis, skewness, range), in order to obtain a pic­ture about the data structure and outliers. Graphical representation of the data can save time, and it also offers an effective way to investigate and evaluate the performance of the sensory panel.

 read more ...