Suen, H. K., Lee, P.S. (1985). Impact of the use of a percentage agreement on behavioural observation: a reassessment. Journal of Psychopathology and Behavioral Assessment, 7, 221-234. Behaviouralists have developed a sophisticated method for assessing changes in behaviour that depend on accurate measurement of behaviour. Direct observation of behaviour is traditionally one of the carriers of behavioural measurement. Therefore, researchers need to address psychometric properties, such as the .B the Interobserver Agreement, of observational measures to ensure a reliable and valid measurement. Of the many indices of the Interobserver agreement, the percentage of the agreement is the most popular. Its use persists despite repeated reminders and empirical evidence that suggests that it is not the most psychometric statistic that determines interobserver agreement because of its inability to take into account chance.
Cohens Kappa (1960) has long been proposed as a more psychometric statistic for the evaluation of the Interobserver agreement. Kappa is described and calculation methods are presented. Cohen`s Kappa should not be considered a gold standard for an agreement on time and movement studies. Shrout, P. E., Spitzer, R. L., Fleiss, J. L. (1987).
Comment: Quantification of compliance in the resumed psychiatric diagnosis. Archives of General Psychiatry, 44, 172-178. Fiss, J. L. (1971). Measure of the scale rated correspondence between many advisors. Psychological Bulletin, 76, 378-382. Hartmann, D. P. (1977, spring).
Reflections in the choice of the reliability estimates of inter-observers. Journal of Applied Behavior Analysis, 10, 103-116. Langenbucher, J., Labouvie, E., Morgenstern, J. (1996). Methodological evolution: measurement of the diagnostic agreement. Journal of Consulting and Clinical Psychology, 64, 1285-1289. A procedure to improve the credibility of the data by comparing the independent observations of two or more people from the same event. The IOA is calculated by calculating the number of agreements between independent observers and divided by the total number of agreements plus disagreements. The coefficient is then multiplied by 100 to calculate the percentage (%) Consent. Berk, R.
A. (1979). Generalization of behavioural observations: a clarification of the Interobserver agreement and the reliability of the inter-observer. American Journal of Mental Deficiency, 83, 460-472. The Inter-Observer Agreement (IOA) is an important aspect of data quality in clinical work time and movement trials. To date, these studies have used simple and ad hoc approaches to the evaluation of IOA, often with minimal reports on methodological details. The most important methodological questions are how task time-stamping intervals are aligned, which rarely have start and end times, and how the IOA is evaluated for several nominal variables. We present a combination of methods that addresses both problems at the same time and provides a more appropriate measure for the evaluation of IOA for time and movement studies. The problem of orientation is addressed by converting task-level data into small time slots and then indexing data from different observers over time. A method for multivariate nominal data, the Iota score, is then applied to time data.
We illustrate our approach by comparing iota scores to the average of the united cohen-kappa scores, based on these measurements based on existing data from an observational study conducted by emergency physicians.