§ 462.11 What must an application contain?

(a) Application content and format. In order for the Secretary to determine whether a standardized test is suitable for measuring the gains of participants in an adult education program required to report under the NRS, a test publisher must—

(1) Include with its application information listed in paragraphs (b) through (i) of this section, and, if applicable, the information listed in paragraph (j) of this section;

(2) Provide evidence that it holds a registered copyright of a test or is licensed by the copyright holder to sell or distribute a test.

(3)

(i) Arrange the information in its application in the order it is presented in paragraphs (b) through (j) of this section; or

(ii) Include a table of contents in its application that identifies the location of the information required in paragraphs (b) through (j) of this section.

(4) Submit to the Secretary four copies of its application.

(b) General information.

(1) A statement, in the technical manual for the test, of the intended purpose of the test and how the test will allow examinees to demonstrate the skills that are associated with the NRS educational functioning levels in the Guidelines.

(2) The name, address, e-mail address, and telephone and fax numbers of a contact person to whom the Secretary may address inquiries.

(3) A summary of the precise editions, forms, levels, and, if applicable, sub-tests and abbreviated tests that the test publisher is requesting that the Secretary review and determine to be suitable for use in the NRS.

(c) Development. Documentation of how the test was developed, including a description of—

(1) The nature of samples of examinees administered the test during pilot or field testing, such as—

(i) The number of examinees administered each item;

(ii) How similar the sample or samples of examinees used to develop and evaluate the test were to the adult education population of interest to the NRS; and

(iii) The steps, if any, taken to ensure that the examinees were motivated while responding to the test; and

(2) The steps taken to ensure the quality of test items or tasks, such as—

(i) The extent to which items or tasks on the test were reviewed for fairness and sensitivity; and

(ii) The extent to which items or tasks on the test were screened for the adequacy of their psychometric properties.

(3) The procedures used to assign items to—

(i) Forms, for tests that are constructed prior to being administered to examinees; or

(ii) Examinees, for adaptive tests in which items are selected in real time.

(d) Maintenance. Documentation of how the test is maintained, including a description of—

(1) How frequently, if ever, new forms of the test are developed;

(2) The steps taken to ensure the comparability of scores across forms of the test;

(3) The steps taken to maintain the security of the test;

(4) A history of the test's use, including the number of times the test has been administered; and

(5) For a computerized adaptive test, the procedures used to—

(i) Select subsets of items for administration;

(ii) Determine the starting point and termination conditions;

(iii) Score the test; and

(iv) Control for item exposure.

(e) Match of content to the NRS educational functioning levels (content validity). Documentation of the extent to which the items or tasks on the test cover the skills in the NRS educational functioning levels in the Guidelines, including—

(1) Whether the items or tasks on the test require the types and levels of skills used to describe the NRS educational functioning levels;

(2) Whether the items or tasks measure skills that are not associated with the NRS educational functioning levels;

(3) Whether aspects of a particular NRS educational functioning level are not covered by any of the items or tasks;

(4) The procedures used to establish the content validity of the test;

(5) The number of subject-matter experts who provided judgments linking the items or tasks to the NRS educational functioning levels and their qualifications for doing so, particularly their familiarity with adult education and the NRS educational functioning levels; and

(6) The extent to which the judgments of the subject matter experts agree.

(f) Match of scores to NRS educational functioning levels. Documentation of the adequacy of the procedure used to translate the performance of an examinee on a particular test to an estimate of the examinee's standing with respect to the NRS educational functioning levels in the Guidelines, including—

(1) The standard-setting procedures used to establish cut scores for transforming raw or scale scores on the test into estimates of an examinee's NRS educational functioning level;

(2) If judgment-based procedures were used—

(i) The number of subject-matter experts who provided judgments, and their qualifications; and

(ii) Evidence of the extent to which the judgments of subject-matter experts agree;

(3) The standard error of each cut score, and how it was established; and

(4) The extent to which the cut scores might be expected to differ if they had been established by a different (though similar) panel of experts.

(g) Reliability. Documentation of the degree of consistency in performance across different forms of the test in the absence of any external interventions, including—

(1) The correlation between raw (or scale) scores across alternate forms of the test or, in the case of computerized adaptive tests, across alternate administrations of the test;

(2) The consistency with which examinees are classified into the same NRS educational functioning levels across forms of the test. Information regarding classification consistency should be reported for each NRS educational functioning level that the test is being considered for use in measuring;

(3) The adequacy of the research design leading to the estimates of the reliability of the test, including—

(i) The size of the sample(s);

(ii) The similarity between the sample(s) used in the data collection and the adult education population; and

(iii) The steps taken to ensure the motivation of the examinees; and

(4) Any other information explaining the methodology and procedures used to measure the reliability of the test.

(h) Construct validity. Documentation of the appropriateness of a given test for measuring educational gain for the NRS, i.e., documentation that the test measures what it is intended to measure, including—

(1) The extent to which the raw or scale scores and the educational functioning classifications associated with the test correlate (or agree) with scores or classifications associated with other tests designed or intended to assess educational gain in the same adult education population as the NRS;

(2) The extent to which the raw or scale scores are related to other relevant variables, such as teacher evaluation, hours of instruction, or other measures that may be related to test performance;

(3) The adequacy of the research designs associated with these sources of evidence (see paragraph (g)(3) of this section); and

(4) Other evidence demonstrating that the test measures gains in educational functioning resulting from adult education and not from other construct-irrelevant variables, such as practice effects.

(i) Other information.

(1) A description of the manner in which test administration time was determined, and an analysis of the speededness of the test.

(2) Additional guidance on the interpretation of scores resulting from any modifications of the tests for an individual with a disability.

(3) The manual provided to test administrators containing procedures and instructions for test security and administration.

(4) A description of the training or certification required of test administrators and scorers by the test publisher.

(5) A description of retesting (e.g., re-administration of a test because of problems in the original administration such as the test taker becomes ill during the test and cannot finish, there are external interruptions during testing, or there are administration errors) procedures and the analysis upon which the criteria for retesting are based.

(6) Such other evidence as the Secretary may determine is necessary to establish the test's compliance with the criteria and requirements the Secretary uses to determine the suitability of tests as provided in § 462.13.

(j) Previous tests.

(1) For a test used to measure educational gain in the NRS before the effective date of these regulations that is submitted to the Secretary for review under this part, the test publisher must provide documentation of periodic review of the content and specifications of the test to ensure that the test continues to reflect NRS educational functioning levels.

(2) For a test first published five years or more before the date it is submitted to the Secretary for review under this part, the test publisher must provide documentation of periodic review of the content and specifications of the test to ensure that the test continues to reflect NRS educational functioning levels.

(3) For a test that has not changed in the seven years since the Secretary determined, under § 462.13, that it was suitable for use in the NRS that is again being submitted to the Secretary for review under this part, the test publisher must provide updated data supporting the validity of the test for use in classifying adult learners with respect to the NRS educational functioning levels and the measurement of educational gain as defined in § 462.43 of this part.

(4) If a test has been substantially revised—for example by changing its mode of administration, administration procedures, structure, number of items, content specifications, item types, forms, sub-tests, or number of hours between pre- and post-testing from the most recent edition reviewed by the Secretary under this part—the test publisher must provide an analysis of the revisions, including the reasons for the revisions, the implications of the revisions for the comparability of scores on the current test to scores on the previous test, and results from validity, reliability, and equating or standard-setting studies undertaken subsequent to the revisions.

[73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55552, Aug. 19, 2016]