MUSHRA stands for Multiple Stimuli with Hidden Reference and Anchor and is a methodology for conducting a codec listening test to evaluate the perceived quality of the output from lossy audio compression algorithms. It is defined by ITU-R recommendation BS.1534-3.[1] The MUSHRA methodology is recommended for assessing "intermediate audio quality". For very small audio impairments, Recommendation ITU-R BS.1116-3 (ABC/HR) is recommended instead.
The main advantage over the mean opinion score (MOS) methodology (which serves a similar purpose) is that MUSHRA requires fewer participants to obtain statistically significant results. This is because all codecs are presented at the same time, on the same samples, so that a paired t-test or a repeated measures analysis of variance can be used for statistical analysis. Also, the 0–100 scale used by MUSHRA makes it possible to rate very small differences.
In MUSHRA, the listener is presented with the reference (labeled as such), a certain number of test samples, a hidden version of the reference and one or more anchors. The recommendation specifies that a low-range and a mid-range anchor should be included in the test signals. These are typically a 7 kHz and a 3.5 kHz low-pass version of the reference. The purpose of the anchors is to calibrate the scale so that minor artifacts are not unduly penalized. This is particularly important when comparing or pooling results from different labs.
Both, MUSHRA and ITU BS.1116 tests[2] call for trained expert listeners who know what typical artifacts sound like and where they are likely to occur. Expert listeners also have a better internalization of the rating scale which leads to more repeatable results than with untrained listeners. Thus, with trained listeners, fewer listeners are needed to achieve statistically significant results.
It is assumed that preferences are similar for expert listeners and naive listeners and thus results of expert listeners are also predictive for consumers. In agreement with this assumption Schinkel-Bielefeld et al.[3] found no differences in the rank order between expert listeners and untrained listeners when using test signals containing only timbre and no spatial artifacts. However, Rumsey et al.[4] showed that for signals containing spatial artifacts, expert listeners weigh spatial artifacts slightly stronger than untrained listeners, who primarily focus on timbre artifacts.
In addition to this, it has been shown that expert listeners make more use of the option to listen to smaller sections of the signals under test repeatedly and perform more comparisons between the signals under test and the reference. In contrast to the naive listener who produces a preference rating, expert listeners therefore produce an audio quality rating, rating the differences between the signal under test and the uncompressed original, which is the actual goal of a MUSHRA-test.
The MUSHRA guideline mentions several possibilities to assess the reliability of a listener.
The easiest and most common is to disqualify listeners who rate the hidden reference below 90 MUSHRA points for more than 15 percent of all test items. The hidden reference should be rated with 100 MUSHRA points so this is obviously a mistake. While it can happen that the hidden reference and a high-quality signal are confused, a rating of lower than 90 should only be given when the listener is certain that the rated signal is different from the original reference.
The other possibility to assess a listener's performance is eGauge,[5] a framework based on the analysis of variance. It computes agreement, repeatability and discriminability, though only the latter two are recommended for pre or post screening. Agreement analyses how well a listener agrees with the rest of the listeners. Repeatability looks at the variance when rating the same test signal again in comparison to the variance of the other test signals and discriminability analyses if listeners can distinguish between test signals of different conditions. As eGauge requires listening to every test signal twice, it is more effort to apply this than to post screen listeners based on the ratings of the hidden reference. However, if a listener has proven a reliable listener using eGauge, he or she can also be considered a reliable listener for future listening tests, provided the character of the test does not change; A reliable listener for stereo listening test is not necessarily equally good in perceiving artifacts in 5.1 or 22.2 format test items.
It is important to choose critical test items; items that are difficult to encode and are likely to produce artifacts. At the same time, the test items should be ecologically valid; they should be representative of broadcast material and not some synthetic signals especially designed to be difficult to encode. A method to choose critical material is presented by Ekeroot et al. who propose a ranking by elimination procedure.[6] While this is a good way to choose the most critical test items, it does not ensure inclusion a variety of test items prone to different artifacts.
Ideally the character of a MUSHRA test item should have similar characteristics for the duration of that item. Otherwise, it can be difficult for the listener to decide on a rating if different parts of the items display different or stronger artifacts than others.[7] Often shorter items lead to less variability than longer ones, as they are more stationary.[8] However, even when trying to choose stationary items, ecologically valid stimuli will very often have sections that are slightly more critical than the rest of the signal. Thus, listeners who focus on different sections of the signal may evaluate it differently. In this case more critical listeners seem to be better at identifying the most critical regions of a stimulus than less critical listeners.[9]
While in ITU-T P.800 tests[10] which are commonly used to evaluate telephone quality codecs, the tested speech items should always be in the native language of the listeners, this is not necessary in MUSHRA tests. A study with Mandarin Chinese and German listeners found no significant difference between rating foreign language and native language test items. However, listeners needed more time and comparison opportunities when evaluating the foreign language items.[11] Such compensation is not possible in ITU-T P.800 ACR tests where items are heard only once and no comparison to the reference is possible. There, foreign language items are rated as being of lower quality when listeners' language proficiency is low.[12]