Examination Development & Test Validation Process

The Certification Institute’s certification examinations are developed through the input of technical experts, also referred to as "Subject Matter Experts" or SMEs. These SMEs are primarily made up of members of the AACE International’s Technical Board and the Institute’s Certification Board . A copy of this process can be downloaded by clicking here.

Content Development

In accordance with acceptable industry standards, all examinations must start with a job and task analysis of a particular field or occupation.

The job and task analysis identifies the core skills and knowledge in order to be considered a professional practitioner in that field or occupation and establishes the emphasis of core subjects or competencies to be tested on certification examinations.

For the Institute, this analysis is called a Recommended Practice (RP). RPs provide technical reference information, vetted by a rigorous review process, regarding specific competency areas, all of which are under the oversight of the Technical Board.

This first step in exam development establishes Content Validity.

Construct Development

Once the job and task analysis/RP has been completed, SMEs of the Technical Board and Certification Board work cooperatively to develop the structure of the examination. The structure of a test is also considered the table of test specifications and is used to determine the content and emphasis of the test. This aspect is often referred to as Construct Validity.

Developing the table of test specifications maintains a degree of control over the content of the test.

The structure or table of test specifications identifies the number of questions/items to be applied to a given domain or topic of the test. Doing so maintains the content relationship of the test while individual questions, relating to the section/category may be changed or randomly selected from a test item bank.

Item Development

Items are simply questions on a test. The Institute’s exams are comprised of a variety of question types, such as:

• Multiple-choice Question:  All examinations contain multiple-choice questions. These questions are a single question with a problem or incomplete statement (called the stem) and four answers one is correct and three are plausible distractors.
• Scenario Multiple-choice Question: Only Professional and Expertise examinations contain scenario multiple-choice questions. These questions contain one scenario with 5-7 multiple-choice questions related to the scenario.
• Short-Essay Question: Only Professional and Expertise examinations contain short-essay questions, which is to test the candidate’s proficiency in written communication. These questions provide a complex scenario outlining a combination of problems or issues and a defined role for the candidate in addition to those the candidate is addressing in the written response.

SMEs from the Technical Board and Certification Board are involved with the initial development of test items. Items are driven by the table of test specifications and generated in many different ways. The goal is to always develop more items than what the test requires so that if items are abandoned for any reason there are sufficient reserve items to maintain the length of the examination.

Beta Testing

Beta testing is an important step in the development of an examination to ensure it is technically accurate and relevant to the specified job or occupation.

Beta testers consist of individuals who are believed to possess the required minimum skills and abilities for the occupation. Beta testers are either selected by known experienced people within the occupation or by meeting a set of minimum eligibility requirements to demonstrate an understanding of the content being tested.

Beta testers are instrumental in validating that the examination name, test categories and items are conducive to the field or occupation. They are asked to respond and comment on each part of the examination, in addition to pointing out words or phrases that may have an impact on protected groups to eliminate bias.

Item Analysis

Item analysis is evaluating responses to individual exam questions to assess exam quality, effectiveness, and fairness.

Examination quality can be demonstrated in the following ways:

• Item Difficulty: If every candidate gets an answer right or wrong, the question can be deemed too easy or too hard. This will decrease an examination’s reliability. Monitoring an item’s difficulty level will help to maintain an acceptable level of examination reliability.
• Item Discrimination: It is useful to determine the varying degrees of knowledge candidates have of the items being tested. Desirable discrimination can be shown by comparing those who have a high overall score and a higher rate of correct answers on an item to those who scored lower overall.
• Item Distractors: Distractors play a significant role in multiple-choice questions when they are disguised effectively. They become ineffective whey they are obviously incorrect. An effective distractor will attract test takers with a lower overall score than those with a higher overall score.

Final Examination Format

The item analysis of the beta-examination will reveal items that are not working as expected. The identified items will either be eliminated or rewritten. The examination will be reformatted to the correct number of total questions and each topic category is reviewed for the correct number of questions according to the table of test specifications. Any spelling or formatting issues are also corrected.

Test Delivery

Delivery of the examination will be provided through a high-stakes computer-based testing provider at a testing center or through online proctoring.

Continuing Analysis

Examination results are monitored on a periodic basis, but no less than annually, by the Certification Board. If score anomalies occur, the entire examination or individual items on the examination are analyzed for a deeper review by the Certification Committees for the respective certification type.