Learning analytics have come a long way in Learning Management Systems (see our handy dandy blog about learning analytics here), but what if tests and quizzes could adapt as they were being taken to motivate test-takers and provide your organisation with detailed data as the quiz happens without having to wait for analytics after? Computer adaptive testing is what you are looking for.
Adaptive testing leverages advanced technology to provide an in-depth, customised evaluation of every student and to track student growth and performance consistently and continuously throughout a test-taker’s training.
By dynamically adapting based on student response patterns, adaptive assessments can extract large amounts of information from a limited number of test items. This allows the assessments to more accurately and more efficiently pinpoint students’ needs compared to traditional fixed-form tests.
Fixed form vs adaptive exam design
There are two basic test designs: Fixed Form, or static testing, and Adaptive. In a fixed-form test, every test taker receives the same items or questions and can be delivered on paper or computer. Since pen-and-paper exams have been the predominant testing method, this is traditionally how it has been done. In static testing, different versions of the test can be created to combat cheating, however, they cannot change once they have been delivered. Adaptive tests, on the other hand, are dynamic and use student response patterns to tweak the questions on the fly.
What is adaptive testing?
Computer adaptive testing or CAT, is a form of testing that varies the difficulty, level, and order of questions that get asked depending on your performance within the test. This is in contrast to fixed form testing, where the questions are predefined, and nothing changes during the exam.
The idea behind an adaptive test is to drill down into the level of proficiency to the greatest granularity and assessment reliability. Computer adaptive testing provides easier questions if you’re struggling with a particular type of question. If you’re excelling in a subject, the test will automatically adapt to provide harder questions.
Adaptive testing has many benefits to learning and testing:
Saves time and money
Computer adaptive testing saves time by reducing test length. Imagine you need to administer an exam for training purposes. Your top students will quickly fly through the most straightforward questions and only really take their time with the more difficult questions during the last portion of the exam. The middle-ground students won’t need the exam to deviate too much and won’t have much use for the easier items or the difficult ones. And the students with difficulties will be given an easier exam without stumbling over questions and taking a ton of time to answer them, and then be given additional materials to help them skill up to par with everyone else.
Saving time is great, but imagine the cost savings. If you were using a test centre that charges hourly, the reduction in exam times means large money savings passed on to you. Corporate training just got more streamlined as well. With adaptive testing, your business could use the analytics during the test to evaluate the trainee’s performance and knowledge level and have dedicated training plans ready automatically by the end of the exam, targeted at reinforcing specific skills the trainee struggled with during the exam. Qualifying and requalifying employees is now an automated process that takes hours, not weeks.
More precise, fairer exams
Computerised adaptive tests will make exams more accurate, in general. It designs algorithms to get more accurate scores without wasting examinee time.
Traditional static exams test middle students well, favouring them over the bottom students that will struggle and fail, and the top students who sail through without being challenged. With adaptive testing, middle students will perform how they always do. Still, bottom students can take a test more suited to their understanding (and then be given additional help and training after the exam). Top students will be given more complex questions that make for a demanding test of their language skills, thus evening the playing field academically and assuring assessment reliability.
Although each test-taker may see different items, they are scored on a percentile basis that accounts for the difficulty of questions. This shows instructors each student’s ability level. Likewise, it gives employers a fair comparison of each candidate’s qualifications.
Reducing cheating
Since all students are getting an assessment tailored to them, there is better test security than everyone seeing the same exam items. Adaptive test algorithms are very flexible and can have potentially millions of variations, making them much more secure than a traditional exam with X number of questions being administered to a group of people.
Better overall experience
Adaptive assessments will tend to be less frustrating for examinees of all ranges of ability. Since each test-taker is challenged appropriately during an adaptive test, the overall experience is more favourable than a traditional exam. Low performers are not discouraged or intimidated, while high performers won’t get bored and might even enjoy receiving more complex items.
A better test-taking experience will encourage all levels of test-takers to try harder than they might with a conventional test.
We know that not all technology like this is going to be all sunshine and roses – there will be some disadvantages you’ll want to keep an eye on:
Stress
Exams are already stressful enough, and certain aspects of computer adaptive testing can add to that. Middle-of-the-road students will be fine, but imagine if you were taking an exam and suddenly the questions started to get easier and more accessible. You’d assume you were failing, and the stress of underperforming might cause test anxiety to increase. Similarly, if you were on the upper echelon of testing performance but just barely, and the algorithm continued to feed you harder and harder questions, you’d probably find yourself struggling towards the end of the exam to complete it. The continual stress of not knowing what difficulty is coming next might snowball in on itself and not accurately measure performance – which is the whole point of computerised adaptive testing to begin with.
Cost
While it saves money over the long term, the initial costs can be expensive.
CAT is best suited to larger organisations
Adaptive testing isn’t for everyone. If you’re a smaller organisation with limited item banks, you may not need it. The algorithm selects the correct type of questions for the test-taker, all the questions go through a pre-testing phase. This phase requires a fairly large test group of up to 1,000 test-takers.
Redoing questions isn’t possible
The nature of computer adaptive testing does not allow students to review their past questions, or skip questions and go back. From start to finish, there are no detours allowed so as to not trip up the algorithm.
Students could “fool the system”
Not the biggest concern, but students could “trick” the algorithm by purposely answering questions in a certain way to receive an easier exam. Students that Christmas Tree answer exams (randomly fill in multiple choice answers) will also not get proper exams and therefore data collection will be inaccurate.
Pinpointing efficacy in formative versus summative testing
Computer adaptive testing isn’t ideal in every exam situation, but it’s wonderful as a focus tool for teachers and trainers. More accurate feedback can be provided immediately after the test through competency-based ability statements rather than a grade. This indicates to the candidate areas they have done well on, and areas they may need to improve. This kind of feedback is more useful in formative assessment, demonstrating to learners that there are areas to progress to, or constructive guidance. Trainers can also see their program’s performance as a whole, indicating areas they may need to focus their teaching on.
Summative assessment is vastly improved due to formative assessment and computer adaptive tests, but the nature of a summative assessment is that there’s a strict bar to pass, and whoever does has proven their level of competency required. Computer adaptive testing counters this by giving struggling students easier questions and advanced students harder ones, meaning while it’s excellent as a learning tool, summative assessment should still stick to equal-difficulty test forms and assessment reliability.
Now more than ever, organisations need to appreciate that each individual learns and tests differently. It is vital for educational institutions, businesses and awarding bodies to meet everyone’s needs, which is why adaptive testing can be so instrumental in helping teachers, in turn, help their students.
As for human resource solutions, CAT can help expedite the hiring process so as not to waste the employer or the candidate’s time. Of course, adaptive tests are also a great way to develop employees and for a company’s overall growth. Computer adaptive testing can be incredible in educational contexts, where you’re attempting to measure specific skills to see what needs to be taught next. It can allow for personalisation of education and going over previous material to give the student a deeper understanding.
