CRET is a non-profit organization with the purpose of conducting research and development on new educational tests that will be demanded in the future as well as on the basic design of these.

Custom Search

Presentation Report on the International Society for Bayesian Analysis World Meeting (ISBA2012)

- Assessing Prior Distributions for the Item Parameters in the Two-Parameter Logistic IRT Model -

In order to develop a test based on the Item Response Theory (IRT), it is necessary to collect response data from field testing and estimate the properties (item parameters) of each test item, such as discrimination and difficulty. Although estimating item parameters with a high degree of accuracy requires a vast amount of data, it is not infrequent that various restrictions and costs hinder the collection of sufficient data during the field testing. In such cases, it is possible to quantitatively integrate available existing information into the analysis to supplement data to be obtained and to ensure a certain level of estimation accuracy. There are three ways to achieve this: a) refer to item parameter estimates of similar existing test items, b) estimate item parameters based on an item's characteristics such as its content or the skills needed to answer questions, or c) incorporate opinions of experts such as item writers, editors or content specialists. This research focuses on c).

The Bayesian estimation method is a technique of using prior information for parameter estimation in a systematic manner. In this method, prior information on item parameters is expressed in the form of a probability distribution (called a prior distribution). When this is combined with information from data (called likelihood), the posterior distribution of item parameters is obtained. Based on this posterior distribution, the parameter estimates are determined. Therefore, within the context of this research, it is necessary to express expert opinions as a prior distribution. This process is called a "pobability assessment," and is quite common in decision making in economics and other disciplines.

However, many item experts, including item writers, lack knowledge in the IRT, making it difficult for them to assess their prior distributions about item parameters. It is considered easier for experts to work with proportions of correct responses to obtain the prior distributions. The prior distributions are then transformed to those on item parameters. It is, nonetheless, important to note that the proportion of correct responses for an item in the IRT depends on the ability of test takers. Thus it is necessary to have experts imagine two groups (e.g. students from a specific school) whose average ability is already known. The experts are then asked about the probability distribution of a correct answer in each group for the following two questions.

"What is the probability that the percent of correct responses is less than 80%?"

"What is the probability that the percent of correct responses is less than 50%?"

Their response (in the form of a probability estimate) to each question for the two imagined groups is fit to a beta distribution. When this is converted based on statistical theories, a prior distribution of item parameters becomes available. I used a beta distribution because it is mathematically easy to work with the two-parameter logistic IRT model, and because an existing method is available to fit the beta distribution to the probability estimates data as in this study.

In the future, it will be necessary to develop a program that will implement the Bayesian estimation method using the prior distribution obtained in this method, improve the algorithm for fitting the beta distribution, and check the improvement of estimation accuracy using actual data.

(Kentaro Kato, Former CRET Researcher)

2011-07-21

Other Researcher

2011-07-21

Other Researcher

2011-03-08

Other Researcher

2010-09-18

Other Researcher

2010-08-29

Other Researcher

This laboratory conducts research on test evaluation and analysis. We also perform joint research and exchange programs with overseas testing research institutes.

This laboratory conducts research and development into testing approaches that measure communication skills, teamwork skills, and social skills, etc.

Dr. Atsushi Aikawa

Professor,

Faculty of Human Sciences,

University of Tsukuba

Ph.D. in Psychology

This laboratory conducts research on the foundation of computer-based testing, and basic research on media and recognition, as well as applied and practical research

that utilize such knowledge.

Dr. Kanji Akahori

Professor Emeritus of

Tokyo Institute of

Technology