Home

Subjective Training and Evaluation Program

1. Enter a Session File typically by using the Browse button At this point the Session file is opened and parsed In the case of long Session files of multi channel signals this may take several seconds in which 5 Audio Research Labs STEP Subjective Training and Evaluation Program case a progress bar will appear at the bottom of the window Next enter values for Laboratory ID Listener ID and Session ID Select the desired Audio Device You can leave Transition at the default value Click the start button STEP will present the MUSHRA trial panel As with the Administrator Panel in the case of long Session files with multi channel signals reading the signals from disk into memory may take several seconds in which case a progress bar will appear at the bottom of the window Trial panel Trial 1 of 3 Signal_3 woo oo A 80 Excellent Good 60 Fair 40 Poor 20 Bad 0 El DE MES EE E EZ 100 100 100 100 100 100 fer Pee e Co E E gt I LOOP NEXT Position 4 gt 0 00 Stat 4 gt 000 Stop 4 gt 503 Click on the Ref button to play the reference signal or on any of the A B or C buttons to play the processed signals Click on the Loop button to play the signal continuously The horizontal Position slider indicates the instantaneous position in the signal waveform Grabbing and moving the Start slider alters the start point for waveform looping
2. A portion of an equivalent Session file that captures signal and system in the filename rather than using directory structures is shown below If we had wanted to make a truly double blind experimental setup we could have named every signal sigN_sysM wav where N is signal number and N is system number which would include the Hidden Reference all Systems under Test and all low pass anchors Line in Session file Description session mushra Session type Sigl Displayed signal name sigl_orig wav REF Open reference sigl_orig wav HR Hidden reference sigl_sysl wav Sysl System under test Sysl sigl_sys2 wav Sys2 System under test Sys2 sigl_sys3 wav Sys3 System under test Sys3 sigl_1p70 wav LP70 7 0 kHz lowpass anchor sigl_1p35 wav LP35 3 5 kHz lowpass anchor Sig2 Displayed signal name sig2_orig wav REF Open reference sig2_orig wav HR Hidden reference sig2_sysl wav Sysl System under test Sysl sig2_sys2 wav Sys2 System under test Sys2 sig2_sys3 wav Sys3 System under test Sys3 sig2_1p70 wav LP70 7 0 kHz lowpass anchor sig2_1p35 wav LP35 3 5 kHz lowpass anchor 5 3 2 Score file This Session file shown above results in a Score file for each listener where the filename is whatever text string was entered into the Session ID field field concatenated with the string in the Listener ID and with the suffix txt The first part of the Score file for listener SRQ is shown below The rows of the Score file consist o
3. 28 Audio Research Labs STEP Subjective Training and Evaluation Program o Drag Lab Listener Session and Sig down to the Filters or Report Filter box Ignore the File label it will not be used o Drag Sys down to the Rows or Row Labels box o Drag Score down to the Values box three times For each one use the drop down menu to select Value Field Settings Set the first to Average Set the second to StdDev Set the third to Count Excel 2000 These are the steps unique to Excel 2000 To create the Pivot Table do the following e Under the Data tab select PivotTable which will pop up the PivotTable Wizard In the Wizard do the following e Step of 3 o For question Where is the data that you want to analyze select Microsoft Excel list o For question What kind of report do you want to create select PivotTable o Select Next e Step 2 of 3 o For question Where is the data that you want to use click on the selection icon and select all the listener data in Scores worksheet and then click on the exit icon o Select Next e Step 3 of 3 o For question Where do you want to put the PivotTable report select New worksheet o Click on the Layout button e Layout o Drag Lab to the Page drop zone Drag Listener to the Page drop zone Drag Session to the Page drop zone Ignore the File item this will not be used Drag Sig to the
4. r Options V Randomize Trials Show Condition Name V Randomize Conditions Allow Previous M Only Score Item Playing Play Mono As Stereo V Approve Scores Session File This text box indicates the Session file The Browse button permits the administrator to easily select the desired Session file At this point the Session file is opened and parsed In the case of long Session files containing multi channel signals this may take several seconds in which case a progress bar will appear at the bottom of the Session Administration window The parsing of the Session file will produce a report contained in the file check t xt located in the same directory as the Session file The format of the check file is discussed in Section 3 4 3 Laboratory ID Listener ID and Session ID The administrator must enter values for Laboratory ID Listener ID in these text boxes The Laboratory ID is typically a short string indicating the test site Similarly the Listener ID is a short string indicating the listener STEP fills in the Session ID based on the Session filename and the channel mapping filename but this can be over written by the user by typing into the textbox The literal string Scores the Session ID the Laboratory ID and the Listener ID are concatenated separated by _ to provide a unique Score filename Audio Device This drop down box permits selection of the audio output device Tran
5. wav wav Wav wav wav Wav re file Session type Displayed signal name Open reference Hidden reference System under test Sysl Displayed signal name Open reference Hidden reference System under test Sys2 Displayed signal name Open reference Hidden reference System under test Sys3 Displayed signal name Open reference Hidden reference System under test Sysl Displayed signal name Open reference Hidden reference System under test Sys2 Displayed signal name Open reference Hidden reference System under test Sys3 A portion of the corresponding Score file for one listener is shown in the following table The Score file presents the signals and Systems under Test in the same order as the Session file Realize however that when Randomize Trials and Randomize Conditions are selected in the Administrator panel which they are by default each listener experiences a randomized presentation In addition the BS 1116 test methodology the Score file lists the filenames and scores associated with each of buttons A and B for each trial We will see that this format makes for easier data analysis Lab Listener Session ARL SRO bs1116 ARL SRQ bs1116 ARL SRO bs1116 ARL SRO bs1116 ARL SRO bs1116 ARL SRO bs1116 FileA FileB Sig SysA sigl_orig wav sigl_sysl wav Sigl HR sigl_orig wav sigl_sys2 wav Sigl HR sigl_orig wav sigl_sys3 wav Sigl HR sig2_orig wav sig2_sysl wav Sig2 HR sig2_orig wav sig2_sys2 wav Sig2
6. 6 1 Start up Behavior Double click on Session file mushra asi and the program will start First it presents the Welcome panel The user must click the OK button to close this window so that STEP can present the Administration panel Welcome panel STEP Subjective Testing and Evaluation Program Copyright c 2004 Audio Research Labs LLC All Rights Reserved Version 2 00 32 Channel configuration This is a demo version to purchase a complete version visit http www AudioResearchLabs com ASIO Technology by Steinberg Media Technologies GmbH 6 2 Limitations of Demonstration Version of STEP The demonstration version of STEP shows all the capabilities of the program but has the following limitations e It will present only the first 2 seconds of the signal files e It will only write out scores of 0 in the score file 44 Audio Research Labs
7. Sig1 Sys1 5 lab1 lis1 mos sig1_sys2 wav Sig1 Sys2 4 lab 1 lis1 mos sig1_sys3 wav Sig1 Sys3 3 lab 1 lis mos sig2_sys1 wav Sig2 Sys1 5 lab1 lis1 mos sig2_sys2 wav Sig2 Sys2 4 lab1 lis1 mos sig2_sys3 wav Sig2 Sys3 3 5 7 4 Excel Pivot Table To construct the Pivot Table for the systems under test proceed as in Section 6 2 4 but e Inthe Layout part of the Pivot Table o drop Lab Listener Session and Sig in the Report Filter drop zone o drop Sys in the Row drop zone o drop Score in the Data drop zone three times setting to Average StdDev and Count Construct the chart of system mean scores and 95 confidence intervals but e Click on the Y axis of the chart right click and select Format axis o Set the Minimum to be 1 and Maximum to be 5 o Set the Decimal places to be 0 e Set chart title and axis titles This should result in the following Pivot Table and chart 41 Audio Research Labs STEP Subjective Training and Evaluation Program Ca e q gt mos_scores xlsx Microsoft Excel ee Home Insert Page Layout Formulas Data Review View Developer Acrobat QuickBooks 5 A38 vO fe A a c D 0 G H 1 J K 1 Lab All y 2 Listener All y 3 Session All x 4 Sig All x 5 6 Values z Row Labels Average of Score StdDev of Score Count of Score cl Sys High Low Mean 8 Sys1 4 875 0 337831962 24 0 1 Sys1 5 0 4 7 4 9 q sys2 4 083333333 0 503
8. 1p70 sig2 sig2_1p70 sysl sig 2 sig 2_sysl sys2 sig2 sig2_sys2 sys3 sig 2 sig 2_sys3 orig sig3 sig3_orig orig sig 3 sig 3_orig 1p35 sig3 sig3_1p35 1p70 sig 3 sig 3_1p70 66 x in STEP Subjective Training and Evaluation Program XXXXXX OK sys1l sig3 sig3_sysl XXXXXX OK sys2 sig3 sig3_sys2 XXXXXX OK sys3 sig3 sig3_sys3 A check file for a more complicated test is shown here slightly re formatted to fit to the page Notice that in the first portion has the line Speaker configuration different which indicates that in this test different stimuli will play to different loudspeakers This can be confirmed by inspecting the matric of x s under the channel number indication Session Check Session configuration Session MUSHRA File format mono Speaker configuration different Open reference yes Channel Label File mono files CfP_loudspeaker_config txt Randomize Trials yes Randomize Conditions yes Only score item playing yes Play mono as stereo no 12 Trials 5 Conditions per Trial Diagnostic key Unexpected signal components USC Number of channels NC Sampling rate SR Audio word size WS Number of samples NS Channel Mask CM Channels Diagnostic File 11111111112222222222333 12345678901234567890123456789012 XXXXXXX XXXXXX XX XXXXXXXXX OK Orig CO_01_Church CO_01_Church XXXXXXX XXXXXX XX XXXXXXXXX OK Orig CO_01_C
9. 3 sigN_sysM wav Stimulus N processed by system under test M where Mis 1 2 or 3 The mono file format signal files are named as sigN_orig_ lt lab gt wav Reference or unprocessed test stimulus N where N is 1 2 or 3 and lt lab gt is a unique label that associated a mono signal file with an output loudspeaker or ASIO output channel sigN_sysM_ lt lab gt wav Stimulus N processed by system under test M where Mis 1 2 or 3 and N and lt lab gt are as above Note that if you have only a demonstration version of STEP the following behavior will be slightly different as outlined in ANNEX 1 Double click on mushra asi and the program will start First it presents the Welcome panel Welcome panel 4 Audio Research Labs STEP Subjective Training and Evaluation Program STEP Subjective Testing and Evaluation Program Copyright c 2004 Audio Research Labs LLC All Rights Reserved Version 2 00 32 Channel configuration Licensed to Audio Research Labs ASIO Technology by Steinberg Media Technologies GmbH After a slight delay it presents the Administration panel Administration panel Session File Browse Laboratory ID MUSHRA Test Methodolgy Listener ID Session ID Audio Device asio MADlface USB y Transition B5 1534 2 5ms y Options I Randomize Trials I Show Condition Name I Randomize Conditions M Allow Previous F Only Score Item Playing Play Mono As Stereo I Approve Scores
10. 53 lab1 lisl 1 test1 sig2_1p70 wav Sig2 LP70 34 lab1 lisl 1 test1 sig2_1p35 wav Sig2 LP35 20 Any time the subject exits STEP prior to completing a test actually after each completed trial STEP creates an interim score file in the same directory as the session file This file remembers all the subject s scores through the last completed trial In this way a subject can exit at test at any and re start exactly where he or she left off 3 4 5 Initialization File Each time you quit STEP while in a testing mode it creates a step ini file in the same directory as the Session file This file remembers all the options that the administrator may have selected The next time STEP is started in that directory default option settings are over ridden by the initialization file settings This insures consistency from one listener session to the next If one does not desire this behavior simply delete the step ini file prior to starting STEP 3 5 Administrator Panel The Administrator panel controls all parameters of the test presentation It is shown here and each of the settings made using this panel is described 15 Audio Research Labs STEP Subjective Training and Evaluation Program Session File fc STEP_Signal_Demo interleaved demo2 mushra asi Laboratory ID JARL MUSHRA Test Listener ID SRG Session ID mushra Audio Device Fast Track Pro ASIO v Transition BS 1534 2 5ms y
11. A portion of the training Session file is shown here Line in Session file Description session training Session type Sigl Displayed signal name sigl_sysl wav Sysl System under test Sysl sigl_sys2 wav Sys2 System under test Sys2 sigl_sys3 wav Sys3 System under test Sys3 Sig2 Displayed signal name sig2_sysl wav Sysl System under test Sysl sig2_sys2 wav Sys2 System under test Sys2 sig2_sys3 wav Sys3 System under test Sys3 For the second objective STEP can be used in the target test mode but with a set of training files The test administrator can walk the listeners through using the trial panel to listen to and score the test items Remember that the training items should be distinct from the test items 5 Score analysis using Excel 5 1 Overview It is straightforward to analyze the results produced by STEP using a Microsoft Excel Pivot Table An example of Session files concatenated Score files and Excel Pivot Table analysis for each methodology that STEP supports is included in the demo directory A step by step walk though of the analysis is given in this section MUSHRA is presented first as it is most straightforward Each other methodology is presented as differences relative to the MUSHRA analysis 5 2 Score file concatenation There is an additional executable in the STEP directory step_sc exe This supports easy assembly of individual score files into a single score file that is ready to open in Excel It is easiest to u
12. C button click Space Bar Alternately Play or Pause L Loop On Off The horizontal Position slider indicates the instantaneous position in the signal waveform Grabbing and moving the Start slider alters the start point for waveform looping and similarly moving the Stop slider alters the end point thus permitting a loop and zoom function that is particularly powerful for subjective evaluation Note that a subject can exit the test at any time by selecting the X button in the top right corner of the MUSHRA panel and the subject s interim results are saved in the listener s Score file Re entering the test will start exactly where the listener previously left off When all trials have been scored the Administrator window appears again 3 6 2 MUSHRA In each trial in the MUSHRA test methodology the listener must rate several conditions which are denoted by letters A B C etc underneath the vertical scroll bars The reference is indicated by the button REF to the left of the letter buttons The listener selects which of reference or conditions A B C etc he or she wishes to hear at a given instant by clicking on the REF or letter buttons When the listener is satisfied with the scoring he or she clicks on the NEXT button to proceed to the next trial The MUSHRA panel is shown here 18 Audio Research Labs STEP Subjective Training and Evaluation Program Trial 1 of 3 Signal_3 eH oOo oo 80 Ex
13. Sys3 1 labl Las ab7 sig2_sysl wav sig2_sys2 wav Sig2 Sysl Sys2 3 labl lisl ab7 sig2_sysl wav sig2_sys3 wav Sig2 Sysl Sys3 3 labl lisl ab7 sig2_sys2 wav sig2_sys3 wav Sig2 Sys2 Sys3 2 5 5 3 Excel data worksheet The scores txt file is opened with Excel and put in the Scores worksheet A portion of the worksheet is shown here Lab Listener Session FileA FileB Sig SysA SysA Score lab1 lis1 ab7 sig1_sys1 wav sig1_sys2 wav Sig1 Sys1 Sys2 2 lab1 lis1 ab7 sig1_sys1 wav sig1_sys3 wav Sig1 Sys1 Sys3 3 lab1 lis1 ab7 sig1_sys2 wav sig1_sys3 wav Sig1 Sys2 Sys3 1 lab1 lis1 ab7 sig2_sys1 wav sig2_sys2 wav Sig2 Sys1 Sys2 3 lab1 lis1 ab7 sig2_sys1 wav sig2_sys3 wav Sig2 Sys1 Sys3 3 lab1 lis1 ab7 sig2_sys2 wav sig2_sys3 wav Sig2 Sys2 Sys3 2 5 5 4 Excel Pivot Table Since a score is associated with a comparison an additional column must be added to the Scores worksheet Create a new column J with the label Sys In cell J2 insert the formula CONCATENATE G2 H2 and copy this formulas through to the last row of score data Now the new column shows both systems associated with Condition buttons A and B which will be used as a label in the analysis chart To construct the Pivot Table for the systems under test proceed as in Section 6 2 4 but e In the Layout part of the Pivot Table 36 Audio Research Labs STEP Subjective Training and Evaluation Program drop Lab Listener Session
14. and Sig in the Report Filter drop zone drop Sys in the Row drop zone drop Score in the Data drop zone three times setting to Average StdDev and Count Ignore the FileA FileB SysA and SysB items oOo0 0 Construct the chart of system mean scores and 95 confidence intervals but e Click on the Y axis of the chart right click and select Format axis o Set the Minimum to be 3 and Maximum to be 3 or Minimum to be 2 and Maximum to be 2 if this is a 5 level category comparison o Set the Decimal places to be 0 e Set chart title and axis titles This should result in the following Pivot Table and chart E3 W9 5 ab7_scores xlsx Microsoft Excel 5 Home Insert Page Layout Formulas Data Review View Developer Acrobat QuickBooks O A33 G fe A R PONA DONN lt O O N PO VO PO N a 1 Lab All iy 2 Listener All x 3 Session All lx 4 5 Values 6 Row Labels 7 Average of Score StdDev of Score Count of Score cl Sys High Low Mean 7 Sysi Sys2 2 166666667 0 868114732 24 04 Sys1 Sys2 2 5 1 8 2 2 8 Sysi Sys3 2 833333333 0 380693494 24 2 Sys1 Sys3 3 0 2 7 2 8 9 Sys2 Sys3 0 916666667 1 059805834 24 0 4 Sys2 Sys3 1 4 0 5 0 9 10 Grand Total 1 972222222 1 13797723 72 11 12 13 Test1 14 3 15 i 16 17 2 18 E 19 20 1 21 22 T 23 Sys1 Sys2 Sys1 Sys3 Sys2 Sys3 24 25 1 26 27 28 2 29 30 5 6 Comparative AB
15. and SysB The first portion of such a Score file is shown here Lab Listener labl lisl labl lisl labl lisl labl lis1 lab1 lis1 lab1 lis1 Sys1 Sys2 Sys1 Sys3 Sys2 Sys3 Sys1 Sys2 Sys1 Sys3 Session abx abx abx abx abx abx Session type System e System e System e Displayed sig System under System under Displayed sig nder System under Displayed sig nder test Sys2 System under test Sys3 Displayed sig nder System under Displayed sig System under System under nal name test Sysl test Sys2 nal name test Sysl test Sys3 nal name nal name test Sysl test Sys2 nal name test Sysl test Sys3 FileA FileB sigl_sysl wav sigl_sys2 sigl_sysl wav sigl_sys3 sigl_sys2 wav sigl_sys3 sig2_sysl wav sig2_sys2 sig2_sysl wav sig2_sys3 sig2_sys2 wav sig2_sys3 5 6 3 Excel data worksheet The scores txt file is opened with Excel and put in the Scores Sig wav Sigl wav Sigl wav Sigl wav Sig2 wav Sig2 wav Sig2 SysA Sysl Sysl Sys2 Sysl Sysl Sys2 SysB Sys2 Sys3 Sys3 Sys2 Sys3 Sys3 gt Ww W P wo PP EX Score PF OoOrP FF Fo worksheet A portion of the worksheet is shown here Lab Listener Session FileA FileB Sig SysA SysB X Score lab1 lis1 abx sig1_sys1 wav sig1_sys2 wav Sig1 Sys1 Sys2 B 0 lab1 lis1 abx sig1_sys1 wav sig1_sys3 wav Sig1 Sys1 Sys3 A 1 lab1 lis1 abx sig1_sys2 wav si
16. and copy this formula through to the last row of score data Randomized Session file If you wish to use a Session file that is not in systematic order for example due to double bind test requirements this section presents a procedure to un scramble the assignment of Hidden Reference and System under Test to Condition buttons A and B Immediately to the right of the score data create five new columns columns K L M N and O with headings HR Sys HR Score Sys Score and Diff Grade Enter the following formulas under the HR column K IF G2 HR G2 H2 under the Sys column L IF H2 HR G2 H2 under the HR Score column M IF G2 HR 2 J2 under the Sys Score column N IF H2 HR 12 J2 under the Diff Grade column O N2 M2 and copy these formulas through to the last row of score data In this way for each trial column HR always has the hidden reference name column Sys always has the System under Test name column HR Score always has the hidden reference score column Sys Score always has the System under Test score This makes calculation of the difference grade straightforward If you wish to do listener post screening adjust the column references in the next section accordingly Post Screening The BS 1116 test methodology assesses two items of information at each trial the first is whether the listener can identify the hidden reference which is done by giving it the score of 5 00 and the second is the listeners rating
17. open this file dialog box select Select program from a list and click OK e When you are presented with the Open with dialog box select the Always use the selected program to open box e Click on the browse button browse to the folder into which you loaded the STEP executable see Section 2 1 click on step exe and finally click the Open button 2 4 Run a test The STEP session file can accommodate signals in two formats interleaved WAV files or multiple mono WAV files The STEP demonstration signal set has both formats identified by directories inter interleaved format and mono multiple mono file format Under each of inter and mono are directories demol demo2 and demo5 that contain several example Session files and signal files Session files can make use of the following test methodologies Methodology Description bs1116 asi BS 1116 test methodology 1 mushra asi BS 1534 MUSHRA test methodology 2 ab5 asi P 800 Comparison Category Rating methodology with 5 rating grades 4 ab7 asi P 800 Comparison Category Rating methodology with 7 rating grades 4 Mos asi P 800 Absolute Category Rating Mean Opinion Score 4 abx asi ABX forced choice methodology training asi Demonstrates the STEP training mode playlist asi Demonstrates the STEP playlist mode The interleaved file format signal files are named as sigN_orig wav Reference or unprocessed test stimulus N where Nis 1 2 or
18. the slider associated with the condition currently being played out can be adjusted for scoring Approve Scores If selected then STEP will raise a dialog box with a confirmation checkbox that the listener must select in order to proceed to the next trial Show Signal Name If selected then the signal filename will appear at the bottom of the trial or test panel This is the default for training By default it is not selected for any test mode Note that if the Session file is not double blind selecting this may reveal the identity of the System under Test However if the purpose of the session is engineering evaluation and it is desired to know what condition a letter button corresponds to this option should be selected Allow previous If selected then the Prev button will appear on the test panel This is the default for training By default it is not selected for any test mode Play Mono as Stereo If selected then mono test items are played to both left and right output channels If not selected mono items are played to left channel only The following table indicates the options selected by default for each test methodology Option BS 1116 MUSHRA ABS AB7 MOS Training ABX Comparison Has Open Reference X X X Randomize Trials X X X X Randomize Conditions X X X X Only Score Item Playing Approve Scores X X X X Show Condition Name Allow previous X Play Mono as Stereo X X
19. 37837 24 1 93 Sys2 75 85 71 98 73 92 10 Sys3 65 83333333 4 061127137 24 1 71 Sys3 67 55 64 12 65 83 11 HR 100 0 24 0 00 HR 100 00 100 00 100 00 12 LP70 29 3 787852437 24 1 60 LP70 30 60 27 40 29 00 13 LP35 18 58333333 4 323008882 24 1 83 LP35 20 41 16 76 18 58 14 Grand Total 62 04166667 29 51587815 144 15 16 17 E Test1 19 100 20 21 90 El 5 4 BS 1116 The analysis of data from BS 1116 test methodology will be presented as differences relative to the MUSHRA analysis found in the previous section 5 4 1 Session file We use a Session file that captures signal and system in the filename and a portion of the Session file is shown below As with the MUSHRA Session file the BS 1116 Session file is constructed in a systematic way since by default STEP automatically randomizes the order of signal presentation and the assignment of Hidden Reference and System under Test to Condition buttons A and B 31 Audio Research Labs Line in Session file STEP Subjective Training and Evaluation Program Description session BS 1116 Sigl sigl_orig sigl_orig sigl_sysl Sigl Sigl_orig sigl_orig sigl_sys2 Sigl sigl_orig sigl_orig sigl_sys3 Sig2 Sig2_orig Sig2_orig sig2_sysl Sig2 Sig2_orig Sig2_orig sig2_sys2 Sig2 Sig2_orig Sig2_orig sig2_sys3 5 4 2 Sco wav wav Wav wav wav Wav wav wav Wav wav wav Wav
20. 610155 24 0 2 Sys2 4 3 3 9 4 1 10 Sys3 2 833333333 0 564659703 24 0 2 Sys3 3 1 2 6 2 8 11 Grand Total 3 930555556 0 968902223 72 6 eee ITU R Recommendation BS 1116 3 Methods for the subjective assessment of small impairments in audio systems including multichannel sound systems Geneva Switzerland 2015 Available at http www itu int rec R REC BS 1116 en ITU R Recommendation BS 1534 2 Method for the subjective assessment of intermediate quality level of coding systems MUIti Stimulus test with Hidden Reference and Anchor MUSHRA Geneva Switzerland 06 20014 Available at http www itu int rec R REC BS 1116 en ITU R Recommendation BS 1284 1 General Methods for the Subjective Assessment of Sound Quality Geneva Switzerland 2003 Available at http www itu int rec R REC BS 1116 en ITU T Recommendation P 800 08 96 Methods for objective and subjective assessment of quality Geneva Switzerland 1996 Available at http www itu int rec T REC P 800 199608 T en 42 Audio Research Labs STEP Subjective Training and Evaluation Program 5 WAVE or RIFF WAVE sound file format http www mmsp ece mcgill ca documents AudioFormats W AVE WAVE html 6 Steinberg ASIO developer s page http www steinberg net en company developers html 43 Audio Research Labs STEP Subjective Training and Evaluation Program ANNEX I Behavior of demo version of STEP
21. HR sig2_orig wav sig2_sys3 wav Sig2 HR 5 4 3 Excel data worksheet Concatenate all the listener Score files retaining only the first header line and open the resulting file using Excel The first portion of the resulting worksheet is shown here SysB Sysl Sys2 Sys3 Sysl Sys2 Sys3 ScoreA ee Oh Gt SP O a ScoreB 4 ES 2 Ow 0 Y ds ds Lab Listener Session FileA FileB Sig SysA SysB ScoreA ScoreB ARL SRQ bs1116 sig1_orig wav sig1_sys1 wav Sig1 HR Sys1 5 4 4 ARL SRQ bs1116 sig1_orig wav sig1_sys2 wav Sig1 HR Sys2 5 4 3 ARL SRQ bs1116 sig1_orig wav sig1_sys3 wav Sig1 HR Sys3 5 3 2 ARL SRQ bs1116 sig2_orig wav sig2_sys1 wav Sig2 HR Sys1 5 5 32 Audio Research Labs STEP Subjective Training and Evaluation Program ARL SRQ bs1116 sig2_orig wav sig2_sys2 wav Sig2 HR Sys2 al 3 9 al 3 5 ARL SRQ bs1116 sig2_orig wav sig2_sys3 wav Sig2 HR Sys3 5 4 4 Excel Pivot Table Systematic Session file The example Session file shown above is in systematic order If you are using a Session file in systematic order then you only need to do the following Immediately to the right of the score data create one new column K with heading Diff Grade This will calculate the difference between the System under Test score and the Hidden Reference score Enter the following formula in that column J2 12
22. Lab all z 2 Listener Al 2 3 Session An 2 4 Sig all iJ 5 PostScr Al 2 6 7 Values 8 Row Labels Average of Diff Grade StdDev of Diff Grade Count of Diff Grade cl Sys High Low Mean 9 Sysl 0 458333333 0 392225165 24 0 2 Sys1 0 3 0 6 0 5 10 Sys2 0 870833333 0 390628442 24 02 Sys2 0 7 1 0 0 9 11 Sys3 1 579166667 0 488101172 24 0 2 Sys3 1 4 1 8 1 6 12 Grand Total 0 969444444 0 627456051 72 13 14 15 Test 1 do 0 0 17 Sys1 Sys2 Sys3 ue 05 1 19 Zn 10 21 15 23 24 2 0 2 zd 25 27 28 3 0 29 a 3 5 31 a2 4 0 33 System 34 35 36 al 28 M 4r m Sheetl bs1116_scores Ready PJ 5 5 Comparative AB5 and AB7 5 5 1 Session file A portion of an example Session file for the 7 category A B Comparison forced choice test methodology with Reference is shown below There are two options in creating the A B Comparison Session file e Use 5 or 7 rating categories e Include open reference and show REF signal button Open Reference Option If the option of having an open reference and associated REF button is desired then include the has_open_reference tru line in the session file as shown below 5 or 7 rating categories If the session value is ABS then a 5 category rating scale is displayed Alternatively if the session value is AB7 then a 7 category rating scale is displayed Line in Session file Description session AB7 Session type has_open_refere
23. Page drop zone Drag Sys i e Condition or System under Test to the Row drop zone Drag Score to the Data drop zone three times where it will appear as Sum of Score Double click on the first Sum of Score and change to Average Double click on the first Sum of Score and change to StdDev Double click on the second Sum of Score and change to Count o Click on OK to exit Layout panel o Click on Finish to exit PivotTable wizard e Reformat o In the PivotTable sheet right click anywhere in the PivotTable and select Show PivotTable Toolbar o On the PivotTable Toolbar left click the Report Format icon Select the upper left format choice selection indicated by bold outline Click OK to exit Format Panel OO O Ov O 5 3 5 Construct Plot The final steps are for the most part the same in all versions of Excel e Construct a table as shown below where for each table row o Construct the 95 confidence CI interval for each of the Conditions using the following Excel formula Note that the Student t distribution is used This is always correct and converges to the Gaussian distribution as count become greater than 50 TINV 5 Count 1 StdDev SQRT Count where StdDev and Count are the PivotTable values for that condition o Condition is the label for the condition or System under Test 29 Audio Research Labs STEP Subjective Training and Evaluation Program o Mean is
24. STEP Subjective Training and Evaluation Program Subjective Training and Evaluation Program STEP Version 2 00 A computer controlled system for audio presentation and subjective evaluation Audio Research Labs info audioresearchlabs com January 2015 1 Audio Research Labs STEP Subjective Training and Evaluation Program Contents T Minimum Requirement isiin ee ceses tcc state cb ote TAN 3 2 QUES tdi A ted ae ete 3 2 1 Install STEP Soria 3 Ded Download STEP demo lesa 3 2S Run add cis 3 3 a Operation ti iii 7 3 1 Typicalsystem SOUP eins iia dad 7 3 2 Au dio Qutput Devices 8 3 3 signal File Formats ennir E lr ia 8 3 4 Session and SUppOrt Files peison ne E R E E S iss 8 3 5 Administrator Panels 15 3 6 Test Panels AS 18 A NON 25 SS AAA spas ierk Teese ES E Ee E EEEE TEE ESEE E EEk 25 5 1 OVEIVIE Wire ari ETE esha E A E E G ET 25 32 Score file CONCALEM ALTON eorna e rE E it 25 5 3 MUSHRA circa bs 26 SA BSI Ore tira er sek api E abe ained avec ales Rees aah eee eee 31 5 5 Comparative ABS and ABT vurcnconesoocscitncntbenrrenon cross ener E Lin EEEE EE TE EEEa Es 35 5 6 Comparative AB Xion iria ia iia 37 ML MOS ida 40 G6 O O 42 ANNEX I Behavior of demo version Of STEP 0 eccceecceessecesececseceseeecsaeceeneecsaeceneecsaeceseecaeeeeneecsaeeneneees 44 6 1 Start up Behaviors indi 44 6 2 Limitations of Demonstration Version of STEP csccesscecssceesceceeeeeencecseeeeneeceseeesaeecsaeeseaeec
25. X The ABX forced choice comparison is unique amongst the test methodologies in that the outcome is 1 correct identification or 0 incorrect identification Any statistic whose outcome is binary has the Binomial distribution which as we will see has implications when computing the 95 confidence interval on the mean scores An example of a statistic with a Binomial distribution is the outcome of a coin toss where P Heads and P Tails 1 0 Note that if O 0 5 the coin is not fair 37 Audio Research Labs 5 6 1 Session file STEP Subjective Training and Evaluation Program A portion of an example Session file for the ABX Comparison forced choice test methodology is shown below Note that there are only two systems lines per signal block STEP will randomly assign them to Condition buttons A and B and then randomly assign one of them to Condition button X Line in Session file Description session ABX Sigl sigl_sysl sigl_sys2 Sigl sigl_sysl sigl_sys3 Sigl sigl_sys2 sigl_sys3 Sig2 sig2_sysl sig2_sys2 Sig2 sig2_sysl sig2_sys3 Wav Wav Wav Wav Wav Wav Wav Wav Wav Wav 5 6 2 Score file The set of Score files must be concatenated retaining only the first header line in the file However no further editing is required Note that the ABX score file shows the identity of the systems assigned to the A and B buttons i e in the columns SysA
26. X X X Some test methodologies have additional checks on the scoring BS 1116 One of A or B must be scored at 5 0 MUSHRA One of the sliders score for A B etc must be scored at 100 ABX X must be sored as one of A or B Start This starts the test which causes the Administrator panel to be replaced by the appropriate trial or test panel Quit This exits STEP About This shows information about this version of STEP 17 Audio Research Labs STEP Subjective Training and Evaluation Program 3 6 Test Panels 3 6 1 Common Controls and Behavior For all test methodologies each invocation of the test panel represents a trial and the trial number and signal name is displayed in the upper left corner of the panel In each trial the listener must make one or more subjective judgments by setting the values on a set of vertical sliders The subjective rating scale is shown to the left of the set of vertical sliders Play is initiated by clicking on the Condition or systems under test buttons beneath each vertical slider and the presentation software supports seamless switching from one condition to another At any time the listener can click the pause icon button to pause or the play icon button to play the selected signal Selecting the alternate action Loop button causes the signal to play continuously Each of these has a keyboard shortcut as listed here Key Function R Ref button click A B C A B
27. and similarly moving the Stop slider alters the end point thus permitting a loop and zoom function that is particularly powerful for subjective evaluation Rate the processed signals by grabbing and moving the vertical sliders above the letter condition buttons When you are satisfied with the ratings click on the Next button to go on to the next trial Note that if 6 Audio Research Labs STEP Subjective Training and Evaluation Program Approve Scores was selected a confirmation window appears When the last trial is scored the MUSHA window is replaced by the Administrator window and the STEP is ready for the next listener Exit STEP by clicking on the Quit button in the Administrator window 3 Operation 3 1 Typical system setup 3 1 1 Workstation The STEP software is invaluable for use by a codec designer at his or her workstation For example it can be used in Evaluation mode to get a quick assessment of the quality impact of varying one or more coder parameters each parameter set is a different System under Test The workstation mode does not require any special configuration of equipment 3 1 2 Listening Room One typically uses STEP to conduct listening tests in an acoustically conditioned listening room or sound booth In this case Figure 1 shows a simple and effective configuration for use with a listening room Here the PC external to the room runs STEP and has a sound card that produces a multi c
28. art 4 gt 000 Stop 4 503 3 6 8 Playlist This mode provides a simple means of playing a set of files The session file is just a list of files to play such as Sigl_orig wav sigl_sysl wav sigl_sys2 wav sigl_sys3 wav It only supports sequential playing of files so there are no A B or C buttons only NEXT and PREV Trial 1 of 6 sigl_Ip35 way Position l gt f 000 Start 4 gt 0 00 Stop KI 503 24 Audio Research Labs STEP Subjective Training and Evaluation Program 4 Training Every listener should receive training prior to their first test session This has two functions first to familiarize the listener with the range and types of impairments that he or she will hear in the test and second to familiarize the listener with the trial panel to be used in the test For the first objective STEP can be used in training mode In training mode the signal name and system name are both visible on the trial panel see Section 3 2 1 and the correspondence between filenames in the session file and buttons on the user panel are not randomized i e REF is the first file A is the second file etc Using the training panel the test administrator and the listeners can step through representative test files and discuss what impairments should be apparent Remember the administrator can discuss what impairment might be heard but not what score it might receive
29. ceed to the next trial The A B Comparison panel is shown here 21 Audio Research Labs STEP Subjective Training and Evaluation Program Trial 1 of 9 Signal_2 AO BO E A gt 00 LOOP NEXT Position gt 0 00 Stat gt oo Stop 4 3 6 6 MOS The MOS test is a single stimulus Absolute Category Rating ACR test methodology The listener must make a single judgment of the subjective quality of signal A When the listener is satisfied with the scoring he or she clicks on the NEXT button to proceed to the next trial The MOS panel is shown here 22 Audio Research Labs STEP Subjective Training and Evaluation Program Trial 1 of 9 Excellent Good Fair Poor Bad gt an os Position 4 Start 4 Stop a 3 6 7 Training This mode can be used either for listener training or Engineering Evaluation It does not require or support scoring so the vertical sliders are not present Since there is no scoring it is not required to enter either a Laboratory or Listener ID Optionally the processed filename is shown at the bottom of the panel Also it allows browsing forward or back in the Session file so that both PREV and NEXT buttons are available The Training panel is shown here 23 Audio Research Labs STEP Subjective Training and Evaluation Program Trial 1 of 3 Signal_1 ea A EE gt oo LOOP PREV NEXT Position gt 0 00 St
30. cellent Good 60 Fair 40 Poor 20 Bad 0 Sie IES E E E 100 100 100 100 100 100 fr AE Ce Co E EF gt II LOOP NEXT Position 4 gt 000 Stat 4 gt 0 00 Stop a Bie 503 3 6 3 BS 1116 In each trial in the BS 1116 test methodology the listener must rate two conditions which are denoted by letters A and B underneath the vertical scroll bars One of them is a hidden reference The listener also has available the open reference which is indicated by the button REF to the left of the letter buttons The listener selects which of reference or conditions A or B he or she wishes to hear at a given instant by clicking on the REF or letter buttons Note that because the listener knows that one of A or B is a hidden reference the Check Scores option checks that one of the A or B scores has the value 5 0 implicitly indicating that it is the hidden reference When the listener is satisfied with the scoring he or she clicks on the NEXT button to proceed to the next trial The BS 1116 panel is shown here 19 Audio Research Labs STEP Subjective Training and Evaluation Program Trial 1 of 9 Signal_2 Imperceptible Perceptible but not annoying Slightly Annoying Annoying Very Annoying ea ES e gt 00 LOOP NEXT Position gt 0 00 Start 4 gt 0 00 Stop EN 5 03 3 6 4 A B Comparison The A B Comparison mode invokes a two comparison forced ch
31. ding order 1 1 2 2 10 Audio Research Labs STEP Subjective Training and Evaluation Program 3 3 4 4 And so on In the case of interleaved files the file pathname shown in the Session file is the full pathname Mono Files The mandatory mono channel label file for multiple mono files is signaled with the following Session header line mono_chn_label_file ITUR5 1 txt In this case the map file is also for a 5 1 channel signal ITU R 5 1 channel speaker configuration and the file map file contains 1 L 2 R 3 E 4 FE 5 Ls 6 Rs In the case of mono files the file pathname shown in the Session file is the base pathname The set of mono files that make up the signal are constructed by concatenating the base pathname _ the channel label and wav For example if one line in the body of the Session file is demo5 sigl sigl sigl_orig REF then this one line indicates that STEP should read the following set of files demo5 sigl sigl sigl_orig_L wav demo5 sigl sigl sigl_orig_R wav demo5 sigl sigl sigl_orig_C wav demo5 sigl sigl sigl_orig_LFE wav demo5 sigl sigl sigl_orig_Ls wav demo5 sigl sigl sigl_orig_Rs wav Channel label files can be extended to as many channels as are needed to support a test up to the STEP limit of 32 For example a test with 22 2 channel signals that is run in a listening room with 30 loudspeakers might use the f
32. dology as indicated in Section 3 5 Attribute Possible Values randomize_trials true false randomize_conditions true false only_score_item_playing true false play_mono_as_stereo true false The options Show Condition Name and Allow Previous are only accessible from the Administration Panel Options pane 3 4 1 2 Body The body section of the Session file consists of a simple structure associated with each test trial which is repeated for every trial in the test The structure begins with a line having as the first character followed a space or tab followed by the test item label Subsequent lines in this structure have the pathname relative to the directory containing the Session file of each stimuli or condition i e test item processed by system 9 Audio Research Labs STEP Subjective Training and Evaluation Program under test in the trial followed by space or tab followed by its associated system label which is written to the Score file Depending on the test methodology and options e g has_open_reference the original signal is the first of these files and will be associated with the REF button and subsequent files are associated with condition buttons A B C Note that STEP has a limit of 15 processed files plus the reference file for a total of 16 files although MUSHRA 2 recommends that there be no more than 12 processed files Here is an example structu
33. each signal file and whether each signal file agrees in format with the other signal files This is reported as one line per signal file with a blank line indicating a new trial in the session An example check file for signals in the interleaved file format is shown below If everything is correct then it indicates the number of channels in the interleaved files and the status OK Session Check Session configuration Session MUSHRA File format interleaved Speaker configuration same Open reference yes 3 Trials 6 Conditions per Trial Diagnostic key Unexpected signal components USC Number of channels NC Sampling rate SR Audio word size WS Number of samples NS Channel Mask C Channels Diagnostic File 6 channel OK sigl_orig wav 6 channel OK Sigl_orig wav 6 channel OK sigl_sysl wav 6 channel OK sigl_sys2 wav 6 channel OK sigl_sys3 wav 6 channel OK sigl_1p70 wav 6 channel OK sigl_1p35 wav 6 channel OK sig2_orig wav 12 Audio Research Labs STEP Subjective Training and Evaluation Program hannel hannel hannel hannel hannel hannel 6 channel 6 channel 6 channel 6 channel 6 channel 6 channel 6 channel OO OO OG NNNNRNRN O K OK OK OK OK OK OK sig2_orig Wav Wav Wav Wav Wav sig2_sysl sig2_sys2 sig2_sys3 sig2_1p70 sig2_1p35 sig3_orig sig3_orig wav sig3_sys2 sig3_sys3 sig3_1p70 sig3_1p35 sig3_sysl wav wav wav
34. ees 44 2 Audio Research Labs STEP Subjective Training and Evaluation Program 1 Minimum Requirements The minimum system requirements to support this product are Microsoft Windows 7 64 bit operating system Pentium 64 bit processor i e i5 or 17 4GB of memory 8 GB is recommended for 16 to 32 channel signals 1 MB of hard disk space Stereo or multi channel audio output device with ASIO drivers 2 Quick Start Assuming that your computer platform meets the minimum requirements and that you have your sound card installed get started by doing the following steps 2 1 Install STEP There is only one ARL STEP software program the demo is the fully functional product but it functions in demo mode when it finds no license file When you purchase the software you will receive your license file via email You should have already downloaded the self extracting zip archive Step64 exe Note as of this writing it is Version 2 00 We recommend that you unzip the files to the following folder which is the default location C Programs Files ARL Do this by double clicking on Step64 exe and clicking on the Unzip button The extraction process will create the folder C Programs Files ARL Step64 that will contains the 64 bit STEP executable and STEP Version 2 00 user s manual When you purchase the software you will receive a license file license txt Please copy that to the folder containing the Step64 executable Note
35. ener correctly identified the hidden reference 95 percent of the time Construct Pivot Table To construct the Pivot Table for the systems under test proceed as in Section 6 2 4 but e Select all data for the Pivot Table i e columns A through M or Q e Inthe Layout part of the Pivot Table o drop Lab Listener Session Sig and PostScr if present to the Report Filter drop zone o drop SysB in the Row drop zone the System under Test o drop Diff Grade in the Data drop zone three times setting to Average StdDev and Count o ignore the FileA and FileB items Construct the chart of system mean scores and 95 confidence intervals but e Click on the Y axis of the chart right click and select Format axis o Select Scale tab and designate Minimum to be 4 Maximum to be 0 o Select Number tab and set Decimal places to be 1 e Click on the X axis at the top of the chart right click and select Format axis o Select Scale tab and designate Minimum to be 4 Maximum to be 0 o Select Patterns tab and set Tick Mark Labels to be high This should result in the following Pivot Table and chart 34 Audio Research Labs STEP Subjective Training and Evaluation Program 09 do 7 bs1116_scores xlsx Microsoft Excel x Home Insert Page Layout Formulas Data Review View Developer Acrobat QuickBooks ax A37 vO fl B c D E F 6 H 1 J K L M N E 1
36. er section can specify a number of options but STEP assumes defaults for all but the test methodology such that the header can be as short as one line as shown above Options are specified using attribute value pairs i e using a header line attribute value The following tables list optional header lines their attribute text strings the possible values text strings and their default values The attributes and values are show as literal strings The exception is interleaved_chn_map_file and mono_chn_label_file that take a filename as value and are discussed in detail in Section 3 4 2 The attribute lock_admin_options_panel has the value true then all of the check box selections in the options panel in the Session Administration window are de activated Their values are the defaults for the test methodology or are over ridden by other lines in the session header This table indicates options that can only be set in the Session file header Attribute Possible Values Default file format interleaved interleaved mono interleaved_chn_map_file Ascending order mono_chn_label_file No default speaker_configuration same same different has_open reference true true false lock_admin_options_panel true false false This table indicates options that can override the options specified in the Administration Panel Options pane These options have default values that depend on the test metho
37. erleaved file format the mono file format can use any subset of ASIO output channels i e loudspeaker signal feeds For example if the mono channel label file specifies a mapping for 32 channels a stimuli may consist of an arbitrary and non contiguous subset of those 32 channels STEP automatically feeds the remaining ASIO output channels with the zero signal 3 4 Session and Support Files The format of the MUSHRA Session and Score files will be discussed in this section The difference between MUSHA Session and Score files of that of the other test methodologies is discussed in Section 5 3 4 1 Session File By default STEP Session files have extension asi although any extension or no extension will work The session file is a plain ASCII text file that can be created with any text editor It can have PC line terminations CR LF or Mac Linux line terminations LF A session file is composed of a header section followed by a body section 3 4 1 1 Header The simplest header section of a session file is just the line that specifies the test methodology Session MUSHRA STEP is not sensitive to case when parsing the Session header so any mix of upper and lower case will work 8 Audio Research Labs STEP Subjective Training and Evaluation Program The possible session header lines are Session BS 1116 Session MUSHRA Session AB5 Session AB7 Session ABX Session MOS Session Train The head
38. f tab separated fields with one score per row such that the Score files can be directly opened by Microsoft Excel The signal name string after the in the Session file appears in the Sig column of the Score file Similarly system name string after the file pathname in the Session file appears in the Sys column 27 Audio Research Labs STEP Subjective Training and Evaluation Program Score File Line Description Lab Listener Session File Sig Sys Scor Header lin ARL SRQ mushra Sigl_orig wav Sigl HR 100 Hidden reference ARL SRQ mushra Sigl_sysl wav Sigl Sysl 96 Sysl ARL SRQ mushra sigl_sys2 wav Sigl Sys2 66 Sys2 ARL SRQ mushra sigl_sys3 wav Sigl Sys3 98 Sys3 ARL SRQ mushra sigl_lp70 wav Sigl LP70 28 7 0 kHz anchor ARL SRQ mushra sigl_1p35 wav Sigl LP35 22 3 5 kHz anchor ARL SRQ mushra sig2_orig wav Sig2 HR 100 Hidden reference ARL SRQ mushra sig2_sysl wav Sig2 Sysl 86 Sysl ARL SRQ mushra sig2_sys2 wav Sig2 Sys2 74 Sys2 ARL SRO mushra Sig2_sys3 wav Sig2 Sys3 53 Sys3 ARL SRQ mushra sig2_1p70 wav Sig2 LP70 34 7 0 kHz anchor ARL SRQ mushra sig2_1p35 wav Sig2 LP35 10 3 5 kHz anchor 5 3 3 Excel data worksheet The set of Score files must be concatenated but with only one header line as the first line in the file The resulting file which should retain the extension txt should be opened by Excel and loaded into a worksheet named for example Scores The first portion of such a worksheet is sh
39. g1_sys3 wav Sig1 Sys2 Sys3 B 1 lab1 lis1 abx sig2_sys1 wav sig2_sys2 wav Sig2 Sys1 Sys2 B 1 lab1 lis1 abx sig2_sys1 wav sig2_sys3 wav Sig2 Sys1 Sys3 A 0 lab1 lis1 abx sig2_sys2 wav sig2_sys3 wav Sig2 Sys2 Sys3 B 1 5 6 4 Excel Pivot Table Since a score is associated with a comparison an additional column must be added to the Scores worksheet Create a new column K with the label Sys In cell K2 insert the formula 38 Audio Research Labs STEP Subjective Training and Evaluation Program IF G2 lt H2 CONCATENATE G2 H2 CONCATENATE H2 G2 and copy this formulas through to the last row of score data Since column data SysA and SysB show the identity of the A and B buttons rather than systematic order as with other tests the Excel formula must sort the identity to obtain the systematic ordering Now the new column shows both systems associated with Condition buttons A and B in sorted order which will be used as a label in the analysis chart To construct the Pivot Table for the systems under test proceed as in Section 6 2 4 but e Inthe Layout part of the Pivot Table o drop Lab Listener Session and Sig in the Report Filter drop zone o drop Sys in the Row drop zone o drop Score in the Data drop zone three times setting to Average StdDev and Count o Ignore the FileA FileB SysA SysB and X items As stated above scores of the ABX forced choice comparison have the Binomial distributi
40. hannel digital output using e g the MADI format This connects to MADI compatible multi channel D A and associated audio amplifiers which in turn connect to the speakers located inside the listening room A Keyboard Video Mouse KVM switch permits the PC console to be either the Test Administrator Console external to listening room or the listener console which would consist of only a presentation screen e g LCD and mouse both located inside the listening room After the test administrator sets up the test using the Administrator panel KVM control is switched to the listener for the duration of the test session AMP D A lt lt MADI Listener Screen KVM switch ____ PC y Admin Console Listening Room Figure 1 Suggested Configuration for Listening Room 7 Audio Research Labs STEP Subjective Training and Evaluation Program 3 2 Audio Output Device STEP only supports audio output hardware that has an ASIO 5 driver See www audioresearchlabs com step for a list of audio interface hardware that is known to work with STEP 3 3 Signal File Formats 3 3 1 Overview STEP Version 2 0 supports both interleaved file signal formats and multiple mono file signal formats as detailed in this section In addition STEP Version 2 0 supports up to 32 channels of signal playout Because this can entail a significant amount of data transfer a
41. hurch CO_01_Church XXXXXXX XXXXXX XK XXXXXXXXX OK LP35 CO_01_Church LP35_CO_01_Church XXX XX XX XX x OK Test2 Sys1 C0_01_Church Sys1_128_C0_01_Church XXXXXXX XXXXXX XX XXXXXXXXX OK Test2 Sys2 CO_01_Church Sys2_128_CO_01_Church XXXXXXX XXXXXX XX XXXXXXXXX OK Test2 Sys3 CO_01_Church Sys3_128_CO_01_Church XXX XX XXX XX XXXX OK Orig CO_04_Fountain CO_04_Fountain XXX XX XXX XX XXXX OK Orig CO_04_Fountain CO_04_Fountain XXX XX XXX XX XXXX OK LP35 CO_04_Fountain LP35_CO_04_ Fountain XXX xx X xXx x OK Test2 Sys1 C0_04_Fountain Sys1_128_C0_04_Fountain XXX XX XXX XX XXXX OK Test2 Sys2 C0_04_Fountain Sys2_128_C0_04_Fountain XXX XX XXX XX XXXX OK Test2 Sys3 CO_04_Fountain Sys3_128_CO_04_Fountain XXX XX X XX XX x x OK Orig CO_05_BarcoTR CO_05_BarcoTR XXX XX X XX XX x x OK Orig CO_05_BarcoTR CO_05_BarcoTR XXX xx X XX XX x x OK LP35 CO_05_BarcoTR LP35_CO_05_BarcoTR XXX XX XX XX x OK Test2 Sys1 CO_05_BarcoTR Sys1_128_CO_05_BarcoTR XXX XX X XX XX x x OK Test2 Sys2 CO_05_BarcoTR Sys2_128_CO_05_BarcoTR XXX XX X XX XX x x OK Test2 Sys3 CO_05_BarcoTR Sys3_128_CO_05_BarcoTR XXX XX XX XX OK Orig CO_08_Musik2 CO_08_Musik2 XXX XX XX XX OK Orig CO_08_Musik2 CO_08_Musik2 XXX XX XX XX OK LP35 C0_08_Musik2 LP35_C0_08_Musik2 XXX XX XX XX OK Test2 Sys1 C0_08_Musik2 Sys1_128_C0_08_Musik2 XXX XX XX XX OK Test2 Sys2 CO_08_Musik2 Sys2_128_CO_08_Musik2 XXX XX XX XX OK Test2 Sys3 C0_08_M
42. nce true Include Open Reference Sigl Displayed signal name sigl_orig wav REF Open Reference sigl_sysl wav ysl System under test Sysl sigl_sys2 wav Sys2 System under test Sys2 Sigl Displayed signal name sigl_orig wav REF Open Reference 35 Audio Research Labs STEP Subjective Training and Evaluation Program sigl_sysl wav Sysl System under test Sysl sigl_sys3 wav Sys3 System under test Sys3 Sigl Displayed signal name sigl_orig wav REF Open Reference sigl_sys2 wav Sys2 System under test Sys2 sigl_sys3 wav Sys3 System under test Sys3 Sig2 Displayed signal name sigl_orig wav REF Open Reference sig2_sysl wav Sysl System under test Sysl sig2_sys2 wav Sys2 System under test Sys2 Sig2 Displayed signal name sigl_orig wav REF Open Reference sig2_sysl wav Sysl System under Test Sysl sig2_sys3 wav Sys3 System under Test Sys3 Sig2 Displayed signal name sigl_orig wav REF Open Reference sig2_sys2 wav Sys2 System under Test Sys2 sig2_sys3 wav Sys3 System under Test Sys3 5 5 2 Score file The set of Score files must be concatenated retaining only the first header line in the file However no further editing is required The first portion of the Score file is shown here Lab Listener Session FileA FileB Sig SysA SysA Score labl lisl ab7 sigl_sysl wav sigl_sys2 wav Sigl Sysl Sys2 2 labl lisl ab7 sigl_sysl wav sigl_sys3 wav Sigl Sysl Sys3 3 lab1 lisl ab7 sigl_sys2 wav sigl_sys3 wav Sigl Sys2
43. of the System under Test in a given trial Failure to correctly identify the hidden reference may indicate an unacceptably low level of listener reliability and can be a reason to exclude a listener s scores from the data analysis If and how this is determined typically depend on the systems under test For example a miss identification of the hidden reference might not be counted against the listener if the System under Test for that particular trial receives a very high score e g greater than 4 90 The following is one way to use hidden reference identification to perform post screening on listener data Note that it assumes a Session file having systematic order Create enter headings PostScr and HR ID in columns L and M Note that rows 2 10 corresponds to the scores for the first listener Enter the following formulas In the first cell L2 under the PostScr column L IF SUM M2 M10 gt 0 95 COUNT M2 M10 1 0 in cell L3 L2 and copy this formula through row 10 33 Audio Research Labs STEP Subjective Training and Evaluation Program under the HR ID column M 1F 12 5 1 0 and copy this formula through row 10 Note that the above assumes that a listener score block consists of 9 scores You will have to change the 10 to whatever is appropriate for your test Now copy the cell block for the first listener L2 M10 to all subsequent listeners In this way the PostScr column for each trial will indicate whether that list
44. oice test methodology using either a 5 point or a 7 point response scale The listener must make a single judgment by adjusting the vertical slider which indicates the extent that condition A is or is not better than condition B The optional reference specified by has_open_reference true in the session header is selected by the button REF to the left of the letter buttons When the listener is satisfied with the scoring he or she clicks on the NEXT button to proceed to the next trial The A B Comparison Session file supports quite a bit of flexibility as described here 5 or 7 point scale You can invoke either a 5 point scale 2 or a 7 point scale 3 via the session value as follows session AB5 5 pont scale session AB7 7 pont scale The A B Comparison panel is shown here 20 Audio Research Labs STEP Subjective Training and Evaluation Program Trial 1 of 9 Signal_3 4 is much better than B 4 is better than B Ais the same as B B is better than A B is much better than A mn 0 EAEE Position 4 gt 0 00 Start 4 gt 0 00 Stop a 503 3 6 5 ABX Comparison The ABX Comparison mode invokes a two comparison forced choice test methodology The listener must make a single judgment by clicking on either the A or B radio buttons to indicate that X is the signal A or the signal B When the listener is satisfied with the scoring he or she clicks on the NEXT button to pro
45. ollowing channel label file 1 A 000_E 00 2 A 030_E 00 3 A 030_E 00 4 A 060_E 00 5 A 060_E 00 6 A 090_E 00 7 A 090_E 00 8 A 110_E 00 9 A 110_E 00 10 A 135_E 00 Te A 135_E 00 12 A 180_E 00 13 A 000_E 35 14 A 045_E 35 15 A 045_E 35 16 A 030_E 35 17 A 030_E 35 18 A 090_E 35 19 A 090_E 35 20 A 110_E 35 ZiL A 110_E 35 11 Audio Research Labs STEP Subjective Training and Evaluation Program 22 A 135_E 35 23 A 135_E 35 24 A 180_E 35 25 A 000_E 90 26 A 000_E 15 27 A 045_E 15 28 A 045_E 15 29 FEL 30 FE2 IMPORTANT Note about missing files from mono file signal set In the example above for 5 1 channel content STEP read a mono file for every label in the channel label file However this is not required and in the case of the 30 line channel label file above this is typically not the case STEP will attempt to open every file pathname constructed from the base filename and the appended channel label The files that are found are reported in the Check file check txt see the next section and if not found then that ASIO output channel is fed with the zero signal 3 4 3 Check File The file check txt is written as a result of parsing the Session file and is written to the same directory as the Session file The file reports e the header options specified in the Session file e the key for diagnostics for each stimulus file e The format of
46. on Let O be the fraction of correct identifications of X which is also equal to the average of the score Then the mean score is u 0 We define the following variables a n Zlp b 2n0 2 c n where n is the total number of responses Count of Score in the Pivot Table and Zo 95 1 96 Then the 95 confidence interval on the mean is given by the two roots of the quadratic equation b yvb 4ac 2a This is implemented in the example spreadsheet which is shown below Note that the standard deviation StdDev computed in the Pivot Table is not used in calculation of the 95 confidence interval although it would yield quite close results For comparison the standard deviation StdDev for the Binomial distribution statistic is computed According to the analysis Sys2 has a mean score that is different from Sys at the 95 level of significance i e the 95 CI does not encompass the 50 point while the mean scores of Sys3 and Sys1 are not different at the 95 level of significance i e the 95 CI does not encompass the 50 point Hence Sys3 achieves transparency that is subjects are not able to identify X when Sys3 is one of A or B with a probability greater than random chance 39 Audio Research Labs STEP Subjective Training and Evaluation Program E dd 9 0 abx_scores xlsx Microsoft Excel x Home Insert Page Layou
47. own below Lab Listener Session File Sig Sys Score ARL SRQ mushra sig1_orig wav Sigi HR 100 ARL SRQ mushra sig1_sys1 wav Sigi Sys1 96 ARL SRQ mushra_ sigi_sys2 wav Sig1 Sys2 66 ARL SRQ mushra sig1_sys3 wav Sig1 Sys3 98 ARL SRQ mushra sig1_lp70 wav Sigi LP70 28 ARL SRQ mushra_ sig1_lp35 wav_ Sig1 LP35 22 ARL SRQ mushra sig2 orig wav Sig2 HR 100 ARL SRQ mushra sig2_sys1 wav Sig2 Sys1 86 ARL SRQ mushra_ sig2 sys2 wav Sig2 Sys2 74 ARL SRQ mushra sig2_sys3 wav Sig2 Sys3 53 ARL SRQ mushra sig2 lp70 wav Sig2 LP70 34 ARL SRQ mushra sig2 Ip35 wav Sig2 LP35 10 5 3 4 Excel PivotTable Use the score worksheet data to create a Pivot Table in Excel as follows Recent versions of Excel Begin by doing the following e Concatenate all Score files retaining only the first header line Note that step_sc exe can be used to for this See Section 5 2 e Open the resulting file using Excel Rename worksheet Sheet 1 which should contain the scores to Scores To create the Pivot Table do the following e Under the Insert tab click on PivotTable This pops up a window permitting you to select the PivotTable data o You should select all the score data including the header row o For Where do you want PivotTable to be placed select New Worksheet o Click OK e Build the Report
48. re associated with one trial ina MUSHRA test using interleaved signal files Sigl test1 sigl_orig wav Ref test1 sigl_orig wav HR testl sigl_sysl wav Sysl testl sigl_sys2 wav Sys2 testl sigl_sys3 wav Sys3 test1 sigl_1p70 wav LP70 test1 sigl_1p35 wav LP35 The simplest MUSHA Session file using interleaved signal files is shown here session MUSHRA Sigl test1 sigl_orig wav Ref test1 sigl_orig wav HR testl sigl_sysl wav Sysl test1 sigl_sys2 wav Sys2 testl sigl_sys3 wav Sys3 test1 sigl_1p70 wav LP70 testl sigl_1p35 wav LP35 Sig2 test1 sig2_orig wav Ref test1 sig2_orig wav HR testl sig2_sysl wav Sysl testl sig2_sys2 wav Sys2 testl sig2_sys3 wav Sys3 test1 sig2_1p70 wav LP70 test1 sig2_1p35 wav LP35 3 4 2 Channel Map Files Interleaved Files The optional interleaved channel map file for interleaved files is signaled with the following Session header line interleaved_chn_map_file ITUR5 1 txt In this case the map file is for a 5 1 channel signal ITU R 5 1 channel speaker configuration and the map file contains aE 1 2 2 3 3 4 4 5 5 6 6 Each line in the file specifies two values the first is the ASIO output channel number the second is the channel number or order in the interleaved sample frame Note that channels are counted from 1 not 0 This is an optional line in the header and if unspecified the default mapping is just ascen
49. rence Sigl Displayed signal name sigl_orig wav REF Reference sigl_sysl wav Sysl A System Sigl Displayed signal name sigl_orig wav REF Reference sigl_sys2 wav Sys2 A System Sigl Displayed signal name sigl_orig wav REF Reference sigl_sys3 wav Sys3 A System Sig2 Displayed signal name 40 Audio Research Labs STEP Subjective Training and Evaluation Program sig2_orig wav REF Reference sig2_sysl wav Sysl A System Sig2 Displayed signal name sig2_orig wav REF Reference sig2_sys2 wav Sys2 A System Sig2 Displayed signal name sig2_orig wav REF Reference sig2_sys3 wav Sys3 A System 5 7 2 Score file The set of Score files must be concatenated retaining only the first header line in the file However no further editing is required The first portion of the Score file is shown here Lab Listener Session File Sig Sys Score labl lisl mos sigl_sysl wav Sigl Sysl 5 labl lis mos sigl_sys2 wav Sigl Sys2 4 lab lis mos sigl_sys3 wav Sigl Sys3 3 lab lis mos sig2_sysl wav Sig2 Sysl 5 labl lis mos sig2_sys2 wav Sig2 Sys2 4 labl lisl mos sig2_sys3 wav Sig2 Sys3 3 5 7 3 Excel data worksheet The scores txt file is opened with Excel and put in the Scores worksheet A portion of the worksheet is shown here Lab Listener Session File Sig Sys Score lab 1 lis1 mos sig1_sys1 wav
50. se 1f you create a shortcut to step_sc exe for example in your current working directory or on your desktop Double clicking on the shortcut brings up this panel 25 Audio Research Labs STEP Subjective Training and Evaluation Program The Browse button permits you to select a file that contains the pathname of all individual score files that you wish to include in the test Note that these pathnames are relative to the directory containing the Score Set File For example scores Lis_1 txt scores Lis_2 txt scores Lis_3 txt scores Lis_4 txt scores Lis_5 txt scores Lis_6 txt After selection the pathname of this file is shown in the Score Set File box The Score Result File box contains the default filename that will contain the concatenated scores You can edit this as appropriate Clicking on the OK button creates the concatenated score file 5 3 MUSHRA 5 3 1 Session file Assume that the STEP Session file uses in the clear naming In this example test items are named sig1 sig2 etc and systems under test are named sys1 sys2 etc but they could just as easily be named popl symphonyl1 where different signals are different music genres and AAC_Encl AAC_Enc2 where we are testing different MPEG Advanced Audio Coding encoders Note that the Session file contains signals and Systems under Test in a systematic order Sigl Sig2 Sig3 etc for each trial and HR Sysl Sys2 Sys3 LP70 and LP35 for each sy
51. sition This drop down box permits selection of the transition format when STEP switches from playing one stimulus to another Choices are Cross Fade Fade out the first stimulus while fading in the next Cross fade waveforms are a raised cosine The transition occurs in one ASIO buffer interval BS 1544 2 5ms Fade out of the first stimulus in 5 ms followed by 5 ms of silence followed by fade in of the next stimulus This is what is recommended in BS 1534 2 2 BS 1544 2 10ms Same as above but all intervals are 10 ms Options The Options panel provides the following options which are selected via check box 16 Audio Research Labs STEP Subjective Training and Evaluation Program Randomize Trials If selected then STEP will automatically randomize the order of presentation of trials with respect to the order indicated in the Session file This is the default for all test modes Regardless of whether or not this is selected the Score file still corresponds exactly to the Session file Randomize Conditions If selected then STEP will automatically randomize the order of conditions on the trial or test panel i e randomize the correspondence between the processed files in the Session file and the lettered buttons on the test panel This is the default for all test modes Regardless of whether or not this is selected the Score file still corresponds exactly to the Session file Only Score Item Playing If selected then only
52. stem within a trial Creating a Session file in this manner is straightforward and since STEP by default randomizes the presentation order of trials and the assignment of systems to Condition letters A B C etc there is no reason not to create a Session file in this manner In this example original i e unprocessed items are in subdirectory originals anchor files e g low pass filtered originals are in subdirectory anchors and processed files are located in a separate subdirectory for each System under Test e g coderl coder2 etc A portion of the Session file is shown here Line in Session file Description session mushra Session type Sigl Displayed signal name originals sigl_orig wav Ref Open reference originals sigl_orig wav HR Hidden reference coderl sigl wav Sysl System under test Sysl coder2 sigl wav Sys2 System under test Sys2 coder3 sigl wav Sys3 System under test Sys3 anchors sigl_lp70 wav LP70 7 0 kHz lowpass anchor anchors sigl_1p35 wav LP35 3 5 kHz lowpass anchor Sig2 Displayed signal name 26 Audio Research Labs STEP Subjective Training and Evaluation Program originals sig2_orig wav Ref Open reference originals sig2_orig wav HR Hidden reference coderl sig2 wav Sysl System under test Sysl coder2 sig2 wav Sys2 System under test Sys2 coder3 sig2 wav Sys3 System under test Sys3 anchors sig2_1p70 wav LP70 7 0 kHz lowpass anchor anchors sig2_1p35 wav LP35 3 5 kHz lowpass anchor
53. t Formulas Data Review View Developer Acrobat QuickBooks ox A36 f B y UNI gt NU SU LY OA UY J 2 NN FA Y UN A o E 1 Lab all x 2 Listener Al a 3 Session Al amp 4 5 Values Binomial Distribution 6 Row Labels Average of Score StdDev of Score Count of Score StdDev 2 05 2 7 Sys1 Sys2 0 8 0 405095747 40 0 400 1 96 43 8416 67 8416 25 6 Sys1 Sys2 0 90 0 65 0 80 8 Sysi Sys3 0 6 0 496138938 40 0 490 196 43 8416 51 8416 14 4 Sys1 Sys3 0 74 0 45 0 60 9 Sys2 Sys3 0 95 0 220721428 40 0 218 1 96 43 8416 79 8416 36 1 Sys2 Sys3 0 99 0 83 0 95 10 Grand Total 0 783333333 0 41370093 120 11 12 13 Test 1 14 1 00 i 16 0 90 17 18 0 80 F 19 0 70 20 21 0 60 22 23 0 50 24 a 25 0 40 22 0 30 27 28 0 20 29 30 0 10 31 32 0 00 r 33 Sys1 Sys2 Sys1 Sys3 Sys2 Sys3 34 System 35 s M 4 gt m Sheeti abx_scores 3 Ready 0J 5 7 MOS 5 7 1 Session file A portion of an example Session file for the Absolute Category Rating ACR or Mean Opinion Score MOS test methodology with Reference is shown below The MOS test is possible with and without an open reference Open Reference Option If the option of having an open reference and associated REF button is desired then include the has_open_reference true line in the session file as shown below Line in Session file Description session MOS Session type has_open_reference tru Include Open Refe
54. t the start of each Trial all stimuli are read from disk and buffered into memory During the Trial real time playout only occurs from memory buffers which minimizes the likelihood of a real time play dropout STEP only supports reading WAV or Extensible WAV file formats The WAV file contents can be any sampling frequency and can be signed integer values of 16 20 or 24 bit word length 3 3 2 Interleaved File Format In this case a stimulus is a single WAV file in which the channels are interleaved More precisely each sample of each audio channel is assembled in an ordered manner to form a sample frame and the WAV file contents is a series of sample frames The association the series of integer values in the sample frame with the loudspeaker signals is specified in the channel map file see Section 3 4 2 While an interleaved channel map file may specify the association or mapping for 32 channels a test can have fewer channels in the interleaved stimuli Specifically if the test stimuli have N interleaved channels those must be the first N channels of the interleaved channel map file STEP automatically feeds the remaining ASIO output channels with the zero signal 3 3 3 Mono File Format In this case a stimulus is a collection of single channel WAV files where each file is a single audio channel The association of audio signal channels to loudspeaker signals is specified in the mono channel label file see Section 3 4 2 Unlike the int
55. that a Version 2 license is required for Step Version 2 i e Step64 exe 2 2 Download STEP demo files We strongly recommend that you download the STEP demonstration signal files since this manual will use examples from the demo files The files are available in a zip archive from the STEP download page Unpack the files at some location in your documents file structure for example C ARL It will create the folder STEP_Signal_Demo containing the following directory structure analysis score files and data analysis spreadsheets setup channel identification signals 5 1 channels and 30 channels interleaved demo1 signal and Session files for interleaved file format mono signals interleaved demo2 signal and Session files for interleaved file format stereo signals interleaved demo5 signal and Session files for interleaved file format 5 1 channel signals mono demo2 signal and Session files for mono file format stereo signals mono demo5 signal and Session files for mono file format 5 1 channel signals 2 3 Associate Session files with the STEP executable STEP Session files control the presentation The Session file extension is asi and it is convenient but not necessary to associate this extension with the STEP program To do this e double click on an example Session file e g in the demo2 directory 3 Audio Research Labs STEP Subjective Training and Evaluation Program e When you are presented with the Windows cannot
56. the PivotTable Average value for that condition o Upper is the Mean plus the 95 confidence interval value o Lower is the Mean minus the 95 confidence interval value Cl Sys High Low Mean 4 03 Syst 85 90 77 85 81 88 4 33 Sys2 80 13 71 47 75 80 4 68 Sys3 68 75 59 40 64 08 2 20 HR 98 35 93 95 96 15 3 87 LP7O 33 24 25 51 29 38 4 24 LP35 22 21 13 74 17 98 e Plot the result using the Stock chart type o Select Layout tab under Chart Tools and fill in Title e g Test 1 X axis as Systems and Y axis as Score o Delete the legend as it has no meaning in this context e Click on the Y axis of the chart o Right click Format axis o Select Scale tab and designate Minimum to be 0 Maximum to be 100 o Select Number tab and set Decimal places to be O The resulting Pivot Table and 95 CI plot are shown here 30 Audio Research Labs STEP Subjective Training and Evaluation Program r Ca ae 9 mushra_scores xlsx Microsoft Excel el gt Home Insert Page Layout Formulas Data Review View Developer Acrobat QuickBooks x K Cuesta x al A B G D E F G H m PA E 1 Lab AI 2 Listener All y 3 Session All y 4 Sig All x 5 6 Values 7 Row Labels Average of Score StdDev of Score Count of Score cl Sys High Low Mean 8 Sys 84 91666667 4 942422102 24 2 09 Sys1 87 00 82 83 84 92 9 Sys2 73 91666667 4 5770
57. usik2 Sys3_128_C0_08_Musik2 XXXXXXX XXXXXX XX XXXXXXXXX OK Orig CO_11_Betty3 CO_11_Betty3 XXXXXXX XXXXXX XX XXXXXXXXX OK Orig CO_11_Betty3 CO_11_Betty3 XXXXXXX XXXXXX XX XXXXXXXXX OK LP35 CO_11_Betty3 LP35_CO_11_Betty3 XXXXXXX XXXXXX XX XXXXXXXXX OK Test2 Sys1 CO_11_Betty3 Sys1_128_CO_11_Betty3 XXXXXXX XXXXXX XX XXXXXXXXX OK Test2 Sys2 CO_11_Betty3 Sys2_128_CO_11_Betty3 XXXXXXX XXXXXX XX XXXXXXXXX OK Test2 Sys3 CO_11_Betty3 Sys3_128_CO_11_Betty3 3 4 4 Score File STEP Score files have extension txt The format of the MUSHRA score files is shown below The file is tab separated ASCII text with a header line followed by one line for each signal line in the associated 14 Audio Research Labs STEP Subjective Training and Evaluation Program Session file Note that the Sig column has the signal or test item label from the session file and the Sys column has the system label from the session file Lab Listener Session File Sig Sys Score lab1 lisl 1 test1 sigl_orig wav Sigl HR 97 lab1 lisl 1 testl sigl_sysl wav Sigl Sysl 96 tabi lisl 1 testl sigl_sys2 wav Sigl Sys2 66 lab1 lisl 1 testl sigl_sys3 wav Sigl Sys3 98 lab1 lisl 1 test1 sigl_1p70 wav Sigl LP70 28 tabi lisl 1 test1 sigl_1p35 wav Sigl LP35 22 lab1 lisl 1 test1 sig2_orig wav Sig2 HR 100 lab1 lisl 1 testl sig2_sysl wav Sig2 Sysl 86 lab1 lisl 1 testl sig2_sys2 wav Sig2 Sys2 74 labi lisl 1 testl sig2_sys3 wav Sig2 Sys3
58. wav wav wav wav An example check file for signals in the mono file format is shown below Under the heading Channels Diagnostic are two lines that read vertically indicate the ASIO output channel number If everything is correct then it indicates the mono channels in the stimulus signal set by means of an that column and the status OK Session Check Session configuration Session MUSHRA File format mono Open reference yes Randomize Trials yes Randomize Conditions yes Play mono as stereo no 3 Trials 6 Conditions per Trial Diagnostic key Number of channels NC Sampling rate SR Audio word size WS Number of samples NS Channel Mask CM XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX XXXXXX Speaker configuration same Channel Label File mono files Only score item playing yes Unexpected signal components ITURS 1 txt Channels Diagnostic 11111111112222222222333 12345678901234567890123456789012 13 OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK Audio Research Labs File orig sig orig sig 1 sig 1 sig 1_orig 1_orig 1p35 sig1 sigl_1p35 1p70 sig 1 sig 1_1p70 sys1 sigl sigl_sysl sys2 sig sys3 sig orig sig orig sig 1p35 sig 1 sig 1 sig 2 sig 1_sys2 1_sys3 2_orig 2 sig2_orig 2 sig 2 1p35

Subjective Training and Evaluation Program

Contents

Download Pdf Manuals

Related Search

Related Contents