Assessment data Should Shape Decisions, Not Describe Them
These days, we speak about data constantly. We are presented with a myriad of dashboards, reports and metrics, before immediately launching into conversations about data-driven decisions and data-powered transformation.
We collect data, we slice it, and circulate it in pretty reports that look increasingly sophisticated.
But do we actually conduct any meaningful analysis of that data?
Or are we content to use it simply to explain the past, and reassure ourselves that everything we are doing is fine – without seeking to use it to improve the future?
Asking Only Easy Questions Means That We Only See What’s on the Surface
It is relatively easy to sit in meeting rooms and ask ourselves, what was the pass rate? Which items performed poorly? Or how did cohort D, compare with cohorts A, B and C?
It is much more challenging to ask whether we are measuring the right things, whether candidate underperformance is actually a symptom of poor assessment design, or whether the consistent overperformance of some delivery centres is because they are genuinely better, or because they have found ways to game the system?
But it is only when we ask ourselves those challenging questions that our data becomes strategically valuable.
Data should be gathered to help us sharpen our judgment, expose weaknesses and challenge our assumptions. It should be a tool to stimulate change.
If we are not analysing and using our data to make decisions, inform redesigns or trigger interventions, are we simply creating colourful graphs and charts to reassure ourselves that everything is under control?
Knowing What Happened is Not Enough
John F Kennedy once said that “Change is the law of life. And those who look only to the past or the present are certain to miss the future”.
As a sector, we spend a lot of time looking at the past.
We excel at producing information, almost every meeting is littered with dashboards, trend lines, item analysis, cohort comparisons and performance summaries. And these are all useful – in theory.
But there is a huge difference between collating data and using it, and too often we are guilty of confusing visibility with action.
Assessment data can be invaluable when it helps us to decide what to do next, but it offers much less if we are using it only to validate what has already happened.
World class assessment organisations don’t simply collect data, they use it to lead them towards more challenging questions:
- Data might reveal that a particular question is performing poorly, and that is useful, but is it also flagging that candidates are consistently struggling with an entire area of the curriculum?
- Perhaps it tells us that one training provider is achieving markedly better outcomes than its peers. But is that a teaching issue, a content issue, a support issue or has that centre simply learnt how to teach the candidates to game the system?
- Data might reveal that in areas of the course where candidates receive detailed developmental feedback, their performance improves significantly more than in areas where only a score is provided. If so, what does that tell us about the effectiveness of the feedback we provide elsewhere?
- Or perhaps candidates who are awarded a particular accommodation are performing less well than anticipated, which might raise questions about whether the accommodation is actually addressing the barriers effectively.
Data only unleashes its true power when it takes us beyond description and into judgement.
Measuring Ambition
As a sector, we claim to be passionate about fairness, accessibility and candidate experience, yet we collect remarkably little evidence that allows us to assess whether those areas are actually improving. We measure pass rates, reliability, item performance.
But do we measure accessibility barriers, support interactions, accommodation effectiveness, candidate friction or navigational issues?
If we are honest, are we measuring what is easy to count and to defend rather than what really matters?
The most valuable data makes us challenge what we are doing.
Yes, that might interrupt the story we would prefer to tell, but it forces us to confront assumptions that have become embedded in our processes, our assessments and sometimes our culture.
The Answers Nobody Asked For
Good data does not only validate decisions – occasionally, it should invalidate them.
Perhaps the data will tell us that an assessment is less robust than we expected, or that a long-standing teaching approach is not actually producing the outcomes that we assumed it was.
It might even suggest that the underlying performance issues are not candidate weaknesses but organisational ones.
None of these things are easy to hear.
But if the vast quantities of dashboards, charts and infographics that we are presented with never unsettles us, we can be fairly confident that we are not using them correctly.
The organisations that get the most value from their data are not the ones who can create the most attractive dashboards and reports, they are the ones who are not afraid to analyse what they have, and are willing to let the data change their minds. They treat data as a decision-making tool, and not a reporting ritual.
They use it to improve content, strengthen delivery, refine feedback and raise standards. They use it to make a difference.
And if our data is not making a difference, we have to ask ourselves why we are wasting our time collecting it, collating it and presenting it in the first place.
We often have more data than we know what to do with.
But.
Are we using it to change things for the better?
Is it influencing our assessment design?
Is it improving our candidate outcomes?
Is it challenging long-held assumptions?
Is it helping us make better decisions?
If the answer is no, then our dashboards are not decision-making tools.
They are decorations.
And decoration has never improved an assessment programme.
If you would like to talk about turning your assessment data into better decisions, we would be glad to. Speak to our team.
