Understanding the Components of Inter-Rater Reliability (IRR)
If the airline has not implemented some form of IRR training, IRR should be presented as a group process. This should begin with an overview of the IRR process, followed by the critical nature of crew assessment, the IRR measures, the assessment form or worksheet, and rating scales and examples of the criteria for each point on the scale. The module should explain that IRR training needs to take into account key organizational elements. This should include the individual instructor or rater, the pilots being assessed, the instruments and scales being used, the specific values and implications of the rating scale, the types of analyses used on the data, and the organizational use of the results.
Consideration must also be given to organizational and regulatory environment. Airline philosophy and policy must support, or may have to be revised to support, developing a high level of instructor standardization. IRR training was developed to provide a more reliable technical and CRM assessment, and consideration needs to be given to the entire system and organization when planning for IRR training.
IRR training may include the use of one or more measures to determine the key elements of rater reliability. The main IRR measures and their functions are:
. .
. . .
Systematic Differences: Shows whether a rater is substantially or significantly above or below the group. Congruency: Shows the general relationship between an individual's ratings and group ratings by comparing the score distribution of individual rates with that of the entire group. Consistency: Shows the degree to which rater scores correlate or shift in unison. Sensitivity: Shows the degree to which raters can discriminate small differences in crew performance. Agreement Index: Shows the degree to which raters have the same rating for a specific item or performance.
Additional IRR information may be found at the end of the first page of this part and in Appendix D.

Working with IRR Data UnderAQP
With the advent of AQP and its requirement for a more systematic LOFT/LOE design and assessment process, the aviation industry needed a more robust form of reliability training. IRR training has been developed to meet the data collection requirements under AQP that reduce or eliminate the subjective nature of CRM rating while providing airlines with consistent data that can be compared over time. Consistently reliable data are very important because they allow organizations to conduct historical analyses of crew performance.
IRR training has been developed to provide airlines with a more precise, longer-term form of crew performance assessment. The IRR approach compares ratings to organizational standards based on the development of benchmark data. This allows airlines to maintain a systematic rating of crew performance over time, even as instructor/evaluators are replaced, new LOFT/LOEs are developed, and new CRM procedures are added to the SOP.
　
中國航空網 www.k6050.com
航空翻譯 www.aviation.cn
本文鏈接地址：Developing Advanced Crew Resource Management (ACRM) Training: A Training Manual(48)