Assessment Design
WGU uses domains, competencies, objectives, and evidence statements to define expectations for student performance.
Domains (e.g., Nursing Leadership and Management) are generally broad topics that act as umbrellas for all of the associated competencies and objectives. Domains are primarily organized around the subject matter, skills of a profession, and ability to perform specified actions.
Competencies (e.g., “The learner integrates leadership and management roles and functions to meet current and anticipated demands”) provide standards for what the student should be able to demonstrate .
Topics group similar content in order to direct students to applicable content in their courses for remediation.
Objectives (“Develop a plan for adapting specified management functions to meet given changes in nursing practice”) provide further insight into the competencies. Objectives are generally more specific in their requirements than competencies. Objectives are the instructional framework used to introduce students to various components of subject matter. They provide the “building blocks” and intellectual foundation necessary for a student to achieve the competency.
Evidence statements are a description of what the student must do to demonstrate competency. Each evidence statement is supported by one or more objectives and communicates how a student can appropriately demonstrate competency. Evidence statements are the foundation for assessment item and task creation.
Figure 1: Courses and Assessments at WGU Are Designed around Competencies, Objectives, and Evidence Statements
All assessments are associated with specific domains, competencies, objectives, and evidence statements within a WGU program or course. Domains, competencies, objectives, and evidence statements provide standards to measure the students against (e.g., if students are asked to apply their knowledge of leadership in nursing, the domains, competencies, objectives, and evidence statements associated with that assessment will all relate to nursing leadership).
Each element of the assessments aligns to specific objectives and competencies. Objective assessments generally assess lower-level cognitive skills such as basic recall of knowledge while performance-based assessments generally assess higher-level cognitive skills by asking students to apply their learning in an approximation of real-world situations. Objective assessments are written to align to the evidence statement level, while performance assessments are written to align to the competency level.
A typical objective assessment at WGU contains approximately 60–90 test items. This length is necessary to ensure that the exam is both rigorous and reliable. High-stakes exams measure student mastery of specific competencies, consist of at least two different forms, have a cut score (minimum passing score) specific to each exam, and are delivered through online delivery systems in a proctored environment.
Cognitive Demand
Depending on the evidence statements and competencies of a given exam, you will find test items that involve different aspects of levels of the cognitive process. A test question about a hammer, for example, could require you to (1) define what a hammer is, (2) use a hammer, or (3) explain what type of hammer is appropriate for a given task. The thought process that you must use to answer test items at each of these three levels is different. Measurement experts call these different levels of thought process cognitive demand: the type of cognitive activity a test taker must use to answer a test item.
Cognitive demand refers to the expected mental complexity involved in performing a particular skill or task across the knowledge domain, or extent of a given subject area. Cognitive demand is not absolute. Rather, cognitive demand depends on the nature of the task and the instructional history of the test taker. Thus, no item or objective has an absolute cognitive demand, although for a given population of individuals, such as those who engage in a particular course, we can make some assumptions about the test takers. Accordingly, WGU assigns a cognitive level to each evidence statement to assist item developers with targeting items at the appropriate level.
WGU targets four levels of cognitive demand (as adapted from Bloom’s taxonomy) across the knowledge domain for its assessments:
- Level 1: Remember/Summarize
- to recognize and retrieve relevant knowledge or details from long-term memory
- to interpret and summarize information using simple examples or classifications as well as detect similarities and differences between objects, events, ideas, and situations
- Level 2: Apply
- to use information provided to solve problems by means of known skills, techniques or methods with familiar and unfamiliar tasks
- to create something using existing materials with little to no creative thought involved
- to carry out a procedure using clues to provide guidance or understanding of the problem and which technique or method would work best
- to apply an understanding in an active role involving making decisions as situations emerge
- to understand a cause-and-effect model and how a change in one part of the system affects a change in another
- Level 3: Analyze
- to distinguish the parts of a whole structure in terms of their relevance or importance; and determine how the parts fit into the overall structure or whole
- to identify the elements of a communication or situation and recognizing how they fit together into a coherent structure
- to ascertain the point of view, biases, values, or intention underlying communications
- Level 4: Evaluate/Create
- to test for internal inconsistencies or fallacies in an operation or a product drawing a conclusion and using data to support a hypothesis
- to use critical thinking to judge a product or operation based on criteria or standards
- to evaluate, judge, and then change something (or go through a cycle of these changes) to fit to a new condition or situation
- to represent the problem and arrive at alternatives or hypotheses that meet certain criteria
- to devise a solution method that meets a problem’s criteria; developing a plan for solving the problem (but not carrying out the steps to create the solution)
- to carry out a plan for solving a given problem that meets certain specifications (may not include originality or uniqueness)
The cognitive demand assigned to a particular evidence statement can be found in the test specification document. Figure 2 is an example of where to find the cognitive level for a particular evidence statement.
Figure 2: These Evidence Statements Have Been Assigned a Cognitive Level of 2
Task Models
A task model is a sample item or item framework for a given evidence statement. A task model provides context for item development, including the competency, the evidence statement to which items should be written, contextual information, and scope. WGU provides a task model with each evidence statement to clarify the expected item framework and cognitive demand. A task model should be used as a point of reference for item creation.
The following task models are for the evidence statements in Figure 2:
130.2.1-06
Which term describes the position of the lungs relative to the ribs?
A. Deep
B. Dorsal
C. Superficial
D. Lateral
130.2.1-07
Place the anatomical structural levels in order of simplest to most complex.
A. Cell
B. Tissue
C. Organ
D. Organ system
130.2.1-07
What is the correct order, from most complex to simplest, of anatomical structural levels?
A. Tissue, organ system, organ, cell
B. Organ system, organ, tissue, cell
C. Cell, tissue, organ, organ system
D. Organ system, organ, cell, tissue
Item Numbering
For information about item numbering, check the AD Knowledge Base (ADKB).
Terminology
alignment: a measure of how well an item conforms to an evidence statement
distractors: incorrect answer options
formative assessment: assessments created as part of the course development; include lesson quizzes, knowledge checks, learning checks, section tests (See the Curriculum page for more information about formative item development.)
given statement: Given statements provide necessary information that a student needs in order to answer the questions. They set up a scenario or situation prior to asking the question. This information should not be instructional.
instructions: tells students exactly how many answers to select (e.g., "Choose 2 answers.") or indicates the method of response
item: a question, problem, or task which is part of an assessment
objective assessment: an assessment where students need to select an answer from the given options
reliability: the measure of how consistent an assessment or item is in measuring the same thing each time
stem: the question or prompting portion of an item
summative assessment: an assessment created to test a student's competency at the end of a course; also called the high-stakes assessment
validity: the degree to which an assessment or item accurately measures what it claims to measure
General Item Standards
Topics such as cluing, trick items, positive phrasing, and plausibility are covered in more depth on the Standards and Practices Guidance platform: https://guidance.wgu.edu/standards/article-categories/assessment-and-feedback/.
General Item Standards
All-inclusive options: WGU does not use answer options that build upon other answer options (e.g., A and B; A, B, and C). This includes answer options where the answer is all listed answer options ("all of the above").
Answer option randomization: Answer options should be shuffled in the delivery system; however, ensure the correct option does not always fall in the same place (e.g., the correct answer should not always be option A). This does not apply to CPLT items.
Clarity: The given (if included), stem, and answer options should be clearly, concisely, and simply stated. Whenever possible, minimize the amount of time it takes to read the item. The item should only contain information that is relevant to answering the question, not instructional information. There should not be distracting or confusing information.
Cohesion: All the elements of an item (i.e., given, stem, prompts, and answer options) should clearly relate to each other. All answer options should grammatically, logically, and directly answer the stem.
Color in images: An image should make sense without any use of color. Color should not be used as the only method for conveying information, especially in charts and graphs.
Correct answer: An item's correct answer(s) should be 100% correct and the only correct answer(s).
Clues: The stem and answer options should not provide a student with clues as to the right answer, including the following types of clues. You can avoid clueing with the following guidelines:
• Answer options should be of similar complexity, detail, and length. There should not be a single option that stands out from the others.
• Answer options should avoid the use of words such as always, never, and completely to indicate an incorrect answer.
• Key terms that appear in the stem should not be included in the answer options, unless all answer options include those same terms.
Focused stem: The stem should contain the central idea of the item, not the answer options. The student should be able to understand what is being asked without reading the answer options
Negative questions: The item should be positively worded unless specified in the evidence statement. Items should not ask students to identify which of the answer options is not correct. These items ask students to identify what they do not know instead of asking them to apply the knowledge they should have gained.
Nonanswers: WGU does not use nonanswers (e.g., all of the above, none of the above, both A and C), including options that contradict the question. Rare exceptions may apply.
Originality and Quotations: Whenever possible, OA content should be original and unique. If it is not possible to write an original passage or if a quote is necessary to test evidence statements that need to reference another source, such as literature, a quotation may be used. Quotations should be taken from the public domain or from a learning resource that WGU has already adopted as part of the course. Do not use the quotation in the exact context as used in the learning resource. For example, if a passage appears in a section of the book that teaches the concept of inference, the passage should not be used in an item that asks about inference; however, it may be acceptable to use this passage in an item about differing points of view or some other topic.
All quotations must be accompanied by a full citation. See the Quotation formatting entry for information on how to format quotations. Quotations are not exempt from the fairness and bias guidelines.
Parallelism: Answer options should be parallel in nature. This means they should all start with the same part of speech and be in the same tense. For example, each option could start with a noun or a present-tense verb.
Plausible distractors: Distractors should be incorrect but still plausible. Distractors should be designed so that a student who does not have competence in the area could consider them to be correct answers. Distractors must be completely believable and should not use humor. Great distractors often come from common mistakes or misconceptions about the material. Distractors should not be incorrect only because they are irrelevant to the scenario or question stem.
Pronouns: Pronouns should have clear referents and be used appropriately, both in the question stem and the answer options.
Question form: Stems for multiple choice, multiple select, and MCEO items should be complete sentences, written in question form (i.e., starting with who, what, which, how, when, why, etc.). However, if it is clearer to place an introductory phrase before the question word, do so. Matching items and order objects items should be phrased as imperative statements.
Readability: The item should test the student’s knowledge of the subject, not reading comprehension, except when comprehension is what is being assessed. A novice should be able to read the item and understand what is being asked. Outside of content-specific language, items should be simple and clear in word choice and sentence structure.
Repeated words: If a term or synonymous terms are repeated in every answer option, they should be incorporated into the question stem or omitted when appropriate. Ultimately, the answer options should be clear and straightforward, so if the item is clearer with repeated terms in the answer options, then they should be retained.
Second person: Stems, givens, and answer options should generally be written to avoid second person.
Sensitive content: The item should not cause an emotional reaction to the material and thereby interfere with a student’s ability to take the assessment. This includes avoiding the use of humor in the given, stem, or answer options. Whenever possible, avoid using individual names or gender-specific pronouns. If names or pronouns are necessary for clarity or to avoid awkward phrasing, use an equal number for each gender across all items. Brand names should also be avoided unless necessary for the content.
Single idea: Items should represent one idea or concept.
Should: Generally use should rather than would or could. Using would and could can often lead to subjectivity in items, but they may be appropriate in rare cases.
Example:
A teacher wants to assess how well students have learned the material for a unit on Asian geography.
Which type of assessment should this teacher use?
Reasoning:
If this stem used would or could instead of should, students could argue that they wouldn't know what the teacher would use, so any answer could be considered correct.
Exceptions:
A firm begins a strategic business initiative by launching a project that is expected to result in a more efficient and tailored service to its customers.
Which factor would reflect a more efficient and tailored service through the new project?
a. Liaison with contractors
b. Improvement in service
c. Feedback from customers
d. Decrease in customer costs
Subjectivity: Items should not require students to form an opinion to answer the question. Subjective items may also rely on students to make assumptions.
Terminal punctuation in answer options: Use a period or question mark at the end of answer options only if the option is a complete sentence, including imperative sentences (e.g., commands). Sometimes an option seems like a complete imperative sentence but is not; only an implied "you" as the subject makes a sentence imperative, but since we typically avoid second person in item stems, these will not frequently occur.
Time sensitivity: Whenever possible, avoid using specific dates and years in order to prevent assessments from becoming outdated. However, in some situations it may be beneficial to include specific years to create an authentic situation, to reference a specific event, or to make an item more clear and straightforward (e.g., accounting or history assessments).
Trick items: Items should not attempt to trick test takers.
What, which, and of the following: What and which suggest different things. Both words are acceptable for use in items, but each word has its own use.
• What precedes a verb (e.g., What is the fastest animal in the jungle?).
• Which precedes a noun (e.g., Which dog is wagging its tail?).
• Avoid using of the following (e.g., Which of the following is…) when referring to answer options, but it is appropriate to use the phrase when referencing additional information given in the question itself.
• Avoid using Which statement or Which option. Answer options should represent a homogeneous class or set. Use a more descriptive noun (e.g., Which animal, Which design) or recast the question.
Formatting
Abbreviations: Abbreviations, such as initialisms and acronyms, should be spelled out on first use within the item with the abbreviation included in parentheses after the term. After the first use in each item, the abbreviated form may be used. Some abbreviations may stay in their abbreviated form throughout an assessment if the abbreviation is common knowledge or is part of the required knowledge for the evidence statement or competency.
Blank spaces: To indicate a blank space within an item, use four underscore marks.
Example:
4 + ____ = 49
Bulleted lists: Bulleted lists within the given statements of items should use automatic bullets. Ensure that any list is visually clear and readable.
Capitalization: The first letter in answer options should be capitalized to maintain consistency. If the word is a hyphenated compound, only the first word is capitalized (e.g., Whole-life insurance). Exceptions may be made for industry-specific standards. Do not capitalize answer options for drop-down lists in the middle or end of a sentence unless the term would otherwise be capitalized (e.g., proper noun).
Dialogue: Answer options that could be considered verbal responses from an identified person from the given or stem should be put in quotation marks.
Example:
A nurse practitioner (NP) is working with a patient who explains a frustrating situation The patient states, "I am feeling angry."
Which response should the NP provide?
a. "Try to express your thoughts in a journal."
b. "What causes you to experience the anger?"
c. "Imagine yourself in the situation again."
d. "Have you had similar feelings of anger?"
Text in Images: WGU's preferred font for assessments is Verdana, the exception being Courier New for IT code. Use Verdana for any font within images.
Given statements: Given statements are separated from the stem by a hard return. Use consistent terminology between the given statement and the question stem. In items that have given statements, stems that refer to information in the given statement should use the word this instead of the.
Example:
Given: John has trouble getting up in the morning.
Stem: What would help John solve this problem?
Images: If one or more images are needed to accompany a stem or given, they should be placed after the stem as it makes sense to do so. The reasoning is this may allow students using screen readers to be able to know what the item is asking for before delving into the alternate text. (See the "Table formatting" entry for a similar rule.)
Example:
Which mode of inheritance is possible for this trait, given the following diagram?
[Image referenced in the stem]
Additionally, it is useful for an introduction or reference to the image to be made in the stem. It is advised to put this reference at the end of the stem so it may begin with a question word (e.g., what).
Introductory phrase in items with passages: Language or literature items that have a given passage of text of four or more sentences or five or more lines should start with "Read the following passage:" (This can be adapted according to the item, such as using paragraph or poem instead of passage or providing the title or author name) Short passages, literature, poems, etc. do not need to start with the introductory phrase.
Example:
Read the following passage:
Exercise is beneficial to a person's health and emotional well-being. It is an activity that raises the heart rate and causes exhaustion. It can also lower cholesterol, blood pressure, and weight. Exercise was an integral part of ancient Greek culture. Physical activity produces endorphins that contribute to a feeling of happiness. There are many forms of exercise, such as cleaning the house or climbing stairs.
Which two details support the main idea as expressed in the first sentence of the given paragraph?
Choose 2 answers.
A. Exercise can lower cholesterol, blood pressure, and weight.
B. [ . . . ]
C. [ . . . ]
D. [ . . . ]
Quotation formatting: Put quotation marks around quotes and include reference information below the quote.
Table formatting: If one or more tables are needed to accompany a stem or given, they should be placed after the stem as it makes sense to do so (see "Images" entry). Additionally, it is useful for an introduction or reference to the table to be made in the stem. It is advised to put this reference at the end of the stem so it may begin with a question word (e.g., what).
Example:
What is the profit margin, given the following table?
[Table referenced in the stem]
Ensure that tables in the given, stem, and answer options are formatted uniformly. When multiple tables are used, the column widths should align between the different tables in order to make the tables easier to read. Table and column headings should be in bold. All text should be left justified unless the industry standard is otherwise. However, monetary amounts given should be aligned by the decimal point. For ADA compliance, tables should be coded in HTML, not included as images.
Numbers and Mathematics
Bold for emphasis: When a certain term needs to be emphasized for clarity, bold text should be used for that term rather than using capitals or italics (e.g., How much will the third contractor receive?).
Comma usage: An exception to WGU's rule of using commas in numbers higher than 1,000 is in a long string of numbers. In situations where a series of numbers are together, do not use commas to separate the thousands place.
Example:
A. 395, 1185, 3555
B. 375, 1195, 3585
C. 425, 1205, 36150
D. 405, 1215, 3645
Decimals after whole numbers: When the answer options include both decimals and whole numbers, a decimal and a zero should not be used after the whole numbers.
Example:
a. 0.23
b. 2.3
c. 23
d. 230
Exception:
Adding a decimal and a 0 after a whole number may be appropriate if it makes the answer options look more cohesive.
a. 40.2
b. 32.3
c. 23.1
d. 23.0
Equations and expressions in question stems: The question stem should be given first, followed by the equation or expression. Equations and expressions should be separated from the question stem with a hard return.
Example:
What is the value of the following expression?
(2 + 3) × 7(–8 – 16)
Graphs: Graphs can be treated like givens or like equations or expressions and may be placed either before or after the question stem.
Ordering: Answer options that begin with numbers or are only numbers should be arranged in ascending order. Answer options should not be randomized in the delivery system.
Ranges of numbers: If answer options are written as ranges of numbers, they should not overlap and should be relatively equal in size (e.g., 51–55, 56–60).
Rounding: The requirement to round an answer can be placed in either the question stem or the instructions, depending on the complexity of the question stem. A very simple question stem can easily accommodate the rounding instructions, while a longer question stem should include the rounding requirement with the instructions.
In the question stem, there should be a comma before "rounded to the nearest. . . "
Example:
What is the area of the circle, rounded to the nearest hundredth?
In the instructions, there should be a line break between stem and the instructions.
Example:
Sand is being poured onto a table. The width of the pile of sand changes with respect to time and is described by the function d(t) = t^3, 1 ≤ t ≤ 3.
At what time, t, does the instantaneous rate of change and the average rate of change of the width of the pile equal each other?
Round to the nearest hundredth.
Stems: Phrase the stem in the form of a question if it is not awkward or inappropriate for the content to do so. Some math stems are more appropriate in command form and should not be changed to question form. This format may be less awkward and clearer than pulling the equation into the given and phrasing the stem as a question.
Example:
Predict the next term in the sequence 3, 5, 8, 12 . . .
Issue:
The sequence should be separated as a given statement like so:
What is the next term in the given sequence?
3, 5, 8, 12 . . .
Example:
Compute 98 + (−54) – (–36.5) + 44.
Item Types
Item types differ based on the platform and the type of assessment (i.e., formative assessments may use different item types than summative assessments). The following image represents an overview of the available items for each of the major platforms. An interactive version of the following item type crosswalk is also available.
The following item types are a sample of all available item types. The item types in this section may be used in summative assessments. Note Fill-in-the-blank, true/false, and yes/no item types should not be used in summative assessments and have not been included in this section.
- Multiple choice:
- 4–5 answer options
- 1 correct answer
- Stems should typically start with a question word and be in the form of a complete interrogative statement.
Example:
Which sentence uses serial commas correctly?
a. I need eggs, butter and milk.
b. We have, three cats, a dog, and a parrot.
c. Last night was a good, fun time.
d. The first three presidents were George Washington, John Adams, and Thomas Jefferson.
- Multiple selection:
- 4–7 answer options
- 2 or more correct answers
- Include at least the same number of incorrect answers as correct answers.
- Stems should typically start with a question word and be in the form of a complete interrogative statement.
- Include the number of correct answer options in the stem when it is clear and not awkward to do so (e.g., "Which two factors affect . . . ").
- Instructions are "Choose [#] answers." with "#" being the number of correct answers.
Example:
What are two functions of fat in the human body?
Choose 2 answers.
a. Fat helps maintain a strong immune system.
b. Fat enables the transport of vitamins A, D, E, and K.
c. Fat helps maintain fluid and electrolyte balance.
d. Fat provides a concentrated source of energy.
Exception: The Leavitt School of Health may not require this in an effort to better prepare students for the NCLEX. Check with the developer to see if your exam is an exception to the rule. If so, use the instructions of "Choose all that apply."
- Multiple choice exhibit option (MCEO):
- 4–5 answer options
- 1 correct answer
- Each answer option should be about three sentences.
- Stems should typically start with a question word.
- Matching:
- Answer options should be less than 80 characters.
- Distractors are optional.
- Question stems should be in the form of a complete imperative statement, usually beginning with either match or classify.
- Longer text should be placed as the prompts (on the left) so that students can see the prompts better in the delivery system.
- The stem wording lists the prompts first and the answer options second (to correspond with the order of the prompts and options)
- Include the first instruction "Answer options may be used more than once or not at all." In order to avoid confusion, this instruction can be omitted when students are required to put things in sequence.
- The second instruction says, "Select your answer from the pull-down list."
- Answer options should be less than 80 characters.
The two types of matching items are matching and classification:
Matching: Has 3–6 prompts and 2 or more answer options; typically starts with Match.
Classification: Has 3–6 prompts and at least 2 answer options; typically starts with Classify.
Matching Example:
Match the cellular characteristics to their associated phases of meiosis.
Answer options may be used more than once or not at all.
Select your answer from the pull-down list.
1. Chromosomal crossover occurs. A. Prophase I
2. Homologous chromosomes line up at the equator. B. Anaphase I
3. Chromosomes migrate to polar ends of the cell. C. Telophase I
4. Nuclear membranes form in daughter cells. D. Metaphase I
Classification Example:
Classify each statement as being either a description or an explanation.
Answer options may be used more than once or not at all.
Select your answer from the pull-down list.
1. Heat is removed from wet skin when evaporation, a cooling process, occurs. A. Description
2. More swimming pools are open in the summer than in the winter. B. Explanation
3. The sky is cornflower blue today with scattered cumulus clouds.
4. The cinder cone volcano was formed from volcanic ash collecting around the vent.
- Ordering:
- 4–6 prompts (on the left) and an equal number of answer options (on the right, in a pull-down list)
- Distractors are not used. Each prompt must have exactly one correct response
- Numbers must start at 1 and cannot skip numbering (e.g., if there are six prompts, each number 1 through 6 should be used once)
- The stem may be worded similarly to the following: "Place [objects] in order from first (1) to last (#)."
- Include the instruction, "Select your answer from the pull-down list."
Example:
Place the stages of protein synthesis in order of occurrence from first (1) to last (5).
Select your answer from the pull-down list.
(Prompt) (Correct Match)
Two DNA strands unzip 1
Creates mRNA 2
Material travels to ribosome 3
tRNA carries amino acid to ribosome 4
Creates protein chain 5
Course Planning Tools (CPLTs)
Course planning tools (CPLTs) include items used to determine students' levels of readiness for courses. CPLT items are sometimes developed with the high-stakes items but are sometimes developed separately. They should follow similar style guidelines as other items except for in the following ways:
Final answer option: The last answer option for each CPLT item should be "I need to learn this."
Randomization: Answer options in CPLT items should not be randomized. For this reason, please ensure the correct option does not always fall in the same place (e.g., the correct answer should not always be option A).
Standard items: Each CPLT will have two standard items. The course descriptions provided in these items should not be edited since they have been pulled from elsewhere. However, typos or major errors in the course description should be flagged to ensure any changes can also be made in other instances of the course description. These two standard items use the following templates:
Given this course description:
[Course description.]
How confident are you that you can succeed in this course?
a. I don't know how I can get through this.
b. I'm concerned that I may not succeed. I will need a lot of support to get through this.
c. It will take a lot of effort, but I think I can do it if I am able to stay focused.
d. It doesn't look harder than any other course. I'm sure I can handle it just fine.
e. It seems easy to me. I'll be done with it in no time.
Given this course description:
[Course description.]
How much experience do you have in this area?
a. This is all new to me.
b. I've heard of some of these things before, but I'm not sure what they all are.
c. I know what these things are, but I need to learn more about what to do with them.
d. I have seen a lot of this and done some of it occasionally.
e. I have done this myself many times.
Summative Assessment Standards
Summative (i.e., high-stakes) items should follow all guidelines in the General Item Standards, Formatting, and Numbers and Mathematics sections of this page.
Standards
Image file type: Provide images in .svg format if possible. If not, then .jpg, .jpeg, or .png files are acceptable.
Image size: Images will be displayed to users initially at 50% of the image resolution (pixels wide by pixels tall). For example, an image that is 200 px by 80 px will be initially displayed at 100 px by 40 px. This makes it possible for users to zoom into the displayed image to 200% without losing any image clarity. Because images will be displayed initially at 50% zoom, all images must be created at twice the resolution. For instance, if normal text content is authored using a font size of 11 pt, then the font size of text in an image should be 22 pt. Any diagram or picture that is part of an image should also be twice the normal size so that it will still look good when it is scaled to its initial 50%.
Necessary information: The item should contain all information necessary to answer the question. This includes tables, images, or other reference materials.
Standard setting: the process of establishing the minimum passing score for each assessment based on the difficulty of the items. Beta testing is currently used to determine the cut scores.
Specificity: An item’s content should not be specifically tied to the learning resource. An industry expert should be able to answer the question without engaging in the learning resource.
Terms and Definitions
1. CLONES. Items that are so similar, we wouldn’t want them on the same form OR across static forms (they are essentially the same item). In other words, we have little to no use for these items for bank building until we have adaptive assessment forms. Items are categorized as clones if they meet all of the following criteria:
a. Elicit the same cognitive response process1 and assess the same concept within an evidence statement
b. Only superficial variation in item stem features (e.g., synonymous words or phrasing; variation in only a subset of surface features) and same correct answer
i. Note: Distractors do not have to be the same across items for them to be considered clones.
2. ENEMIES. Items that are similar enough that we wouldn’t allow them to be on the same form. However, we would allow them to be on different operational forms. Items are categorized as enemies if they meet the following criteria:
a. Elicit the same cognitive response process 1 and assess the same concept within an evidence statement, and either:
b. Inverse items (i.e., the content in the stem versus response options switch places) because the item stem structure and the cognitive load may vary between items despite highly similar content, or
c. Clueing and clanging, or
d. Variation in all surface features of the item given or stem (to prevent memorization of item content), same item stem structure, and same OR different correct answer, or
e. Variation in most surface features of the item given and/or stem, different item stem structure, and different correct answer
3. UNIQUE. Items that are allowed on the same form. Items are categorized as unique if they meet the following criteria:
a. Elicit the same cognitive response process and/or assesses the same concept in an evidence statement, but there is variation in:
i.All surface features in the given and/or item stem,
ii.Item stem structure,
iii.Correct answer,
iv. Plus, or minus:
1. At least one deep feature of the item given or stem (i.e., variation in cognitive response process by changing # of steps or length of time required to answer item)
b. Elicit different cognitive response processes and/or assesses different concepts, and same or different item stem structures
i. Within an evidence statement
ii. Across evidence statements
Key Concepts
1. Cognitive response process
The cognitive response process is a crucial aspect of construct validity that details how information in the item stem is processed by the student, and how the student’s relevant knowledge is accessed, manipulated, and potentially revised to generate a response to the assessment item (e.g., Embretson, 2023). The cognitive response process is often related to difficulty, where more extensive and time-consuming cognitive components of the response processes can increase the difficulty of an item. Examples of cognitive components of the response process include steps required to solve a math problem or perform an analysis or evaluation of non-quantitative information (e.g., comparison of concepts or scenarios; diagnosing an illness; determining the best course of action in a scenario), memory retrieval, sequence of schema or mental model activation and manipulation, working memory capacity (e.g., cognitive load or mental effort), information synthesis.
2. Surface features
Surface features of an item stem that, when varied across items, do not affect the response process or change the mental effort required to answer an item. Examples include generic or randomly chosen names, places, dates, and numbers in an item stem.
3. Deep features
Deep features of an item stem that, when varied across items, directly impact the response process, resulting in an increase or decrease in the mental effort required to answer an item. Examples include the number of steps required to complete an analysis or evaluation, variation in cognitive load (length of item, number of features the student must process), variation in the mental effort associated with a certain configuration of item stem features such as rare or conflicting symptoms in a diagnosis (greater mental effort), whole number values versus fractions in a math problem, where the latter is more time consuming in a calculation performed mentally or by hand.
4. Clueing
When the content of two items overlaps such that knowing the answer to one item gives away the answer to the other item (see examples 2 and 6, below).
5. Clanging
When the surface features or structure of item stems are so similar between items that test-takers might perceive them to be the same item or the similarity might engender confusion amongst test-takers (see “enemy” in examples 1, 2, and 3, below).
