Istation's Indicators of Progress Advanced Reading (ISIP-AR)

Research and Development

Phase 1

A large pool of items spanning the reading abilities of students in grades 4 to 8 was developed and programmed which allowed the measurement of growth in critical areas of reading. The size of the item pool must be sufficiently large to allow students to be assessed monthly. The aims of Phase 1 were achieved by: (a) developing a large item pool across each of the 4 domains of reading, including: word analysis, fluency, vocabulary, and comprehension; (b) field testing item delivery formats with middle grade students to ensure that students in this age group understand and can complete each task independently, as well ensure they find the tasks age appropriate, motivating, and engaging; (c) conducting focus groups with middle grade students to glean preferences about the look and feel of the assessment (i.e., incentives, graphics, and theme); and (d) programming all ISIP-AR items incorporating high quality graphics and animation in an engaging game like theme.

Phase 2

ISIP-AR The ISIP-AR study was completed as a CAT-driven CPM reading assessment tool with proven technical adequacy. To achieve this we: (a) conducted Item Response Theory (IRT) research on our pool of items by administering the entire pool of items via computer administration to a large number of middle grade students in order to determine item parameters, as well as determine which items do not perform as intended and need to be pulled from the item pool; (b) used the data collected from our IRT work to create a CAT algorithm that leads to the most stable θ (theta) score in the least number of items; (c) programmed ISIP-AR items using this CAT algorithm, which allowed us to create an assessment that tailored itself to the performance abilities of each child, as well as showed individual growth in performance over time; and (d) conducted an evaluation of the reliability, concurrent, and predicative validity of ISIP-AR to other well-established measures of reading. Finally, while conducting this reliability and validity work, we also conducted focus groups with teachers to assist us in designing the most meaningful and easy to interpret teacher reports.

ISIP CAT At the completion of the project, we completed programming of all ISIP-AR items subtests, assessed the psychometric qualities of our computer-administered measures by comparing them to already establish examiner-administered measures, and completed programming for data display and reporting information back to teachers and schools. Regarding measures of reliability, the data from the current study suggest consistently high levels of internal consistency, both in the subtest ability scores as well in the overall reading ability scores. In addition, ISIP-AR produced stable scores over time.

Technical Report

This report includes information and results regarding reliability and concurrent validity of ISIP-AR for grades 4 to 8, as well as information regarding Item Response Theory (IRT) calibration and the Computer Adaptive Test (CAT) algorithm of ISIP-AR. Click here to view the report.