The introduction of e-assessment by the National Council for the Training of Journalists was a huge step for the organisation and for the journalism industry as a whole. Faced with growing challenges around the authenticity and security of their paper based exams and demands from a fast changing digital media industry, the NCTJ needed to make a bold decision.
The project commenced in January 2016 with first live exams needing to be launched in November 2016. Since then:
- The Essential Journalism exam is delivered in a secure, lock-down browser-environment that complies with the regulators’ general conditions and its own rigorous exam policies and procedures.
- The Cirrus platform provides an innovative and demanding test of students’ ability to produce work to a deadline and reflects the journalistic skills needed to operate in a multimedia newsroom.
- Test centre administrators not only felt that the Cirrus platform was user-friendly, but that they were very well-supported by the Cirrus team, every step of the way.
- Students’ satisfaction rates have increased with an overwhelmingly positive feedback post exam
- NCTJ’s environmental footprint has decreased.
Background
After exclusively administering paper-based exams, NCTJ chose to revolutionise the journalism industry’s examination system, which had been in place for over 65 years. Undaunted by a challenge, they reached out to Cirrus Assessment to deliver a new e-assessment programme – including innovative video and editing within 12 months.
A Journalism Awarding Body’s approach to assessment
The National Council for the Training of Journalists (NCTJ) is the industry’s charity delivering the premier training scheme for UK journalists. This is provided through a world-class education, and a training and qualification system that develops current and future journalists for the demands of a fast-changing multimedia industry. The NCTJ is regulated by Ofqual, Qualifications Wales and CCEA.
Challenge
Until 2016, NCTJ’s exams were delivered using traditional pen and paper methods. This presented a number of challenges.
- NCTJ qualifications assess the skills required by modern, multimedia journalists. It was increasingly clear that paper based assessments could not effectively mirror modern working practices and the digital environments that modern day journalists operate in.
- The logistics associated with paper exams were increasingly complex. Distribution of papers, secure delivery, marking and moderation were also becoming costly.
- These difficulties, in turn, presented a risk to the confidentiality and security of exams.
- There was also potential for longer than desired timelines for the release of results and feedback to unsuccessful students.
Moving the Essential Journalism exam online
In 2015, the NCTJ conducted a comprehensive review of the content and structure of the Diploma in Journalism, their ‘gold standard’ qualification for trainee journalists. This informed the development of the Essential Journalism exam and their decision to modernise their assessment processes. A thorough e-assessment market review was undertaken and the Cirrus Platform was selected for the project.
The Essential Journalism exam is a mandatory assessment in the Diploma in Journalism, designed to assess core journalism skills including reporting using digital platforms, social media, data and analytics.
Need to speed up the implementation
With only 11 months from the project commencing in January to first rollout of live exams in November, it was recognised that the move to e-assessment would present a new set of challenges. The NCTJ and Cirrus Assessment worked together to identify these and ensure that solutions were devised to make the transition as seamless as possible in order to meet the deadline.
“The role of the modern journalist is digital so when our essential journalism syllabus was introduced it made perfect sense to use an innovative digital assessment platform.”
Joanne Butcher – Chief Executive, NCTJ
Solution
Implementing the e-assessment platform across the NCTJ organisation
It was recognized that for the rollout to be successful, multiple stakeholders would need to be involved in the process and commit to working together to make this project a success.
The stakeholders included:
- Internal committees and staff responsible for exam design and development
- External markers and moderators, currently unfamiliar with e-assessment
- Training providers and centres who were apprehensive about using technology for high-stakes exams
A detailed implementation plan was devised to get all stakeholders onboard, with the following activities planned to assist with transition:
- Training programmes for staff including initial training and refresher workshops throughout the year. These would focus on each delivery phase (item creation, assessment creation, pilot delivery, marking and moderation).
- Seminars at locations across the country with all centres invited. The NCTJ and Cirrus Assessment would together outline plans, provide updates, demonstrate functionality and answer questions and concerns in open forums.
- Pilot exam sessions were scheduled with participating centres across the country. In these, a trial version of the Essential Journalism exam was delivered under exam conditions to students due to take the live exams in November 2016. Members of the NCTJ exams team and Cirrus Assessment staff were present to take feedback from both students and centre administrators. Feedback was given, from both viewpoints on exam format and user experience.
- Mock exams were made available, in the Cirrus Platform, to all centres in October 2016 to aid their students’ preparation for the live exam. This also enabled centres to thoroughly test processes and procedures ahead of live delivery.
Live exam launch November 2016
The first NCTJ exams delivered using e-assessment, took place as scheduled, with representatives from both the NCTJ exams team and Cirrus Assessment in attendance. Detailed planning and close collaboration between the organisations ahead of this meant they were delivered efficiently, securely and effectively using the Cirrus Platform.
The Essential Journalism exam is delivered in a secure, lock-down browser-environment that supports the NCTJ in complying with the regulators’ general conditions and its own rigorous exam policies and procedures.
Candidates have the first 30 minutes to read a press release, identify key facts from the story, watch and listen to a simulated video press conference and produce a 140-160 word online news story, headline and social media snippet. This is an innovative and demanding test of their ability to produce work to a deadline and reflects the journalistic skills needed to operate in a multimedia newsroom.
Results
On submission, candidate scripts were immediately accessible in the Cirrus Platform to the NCTJ exams team. The sophisticated marking workflow assigned scripts to external markers and moderators. Assessors were able to view candidate responses and associated mark schemes within the UI. Markers were able to use onscreen tools to annotate responses which could then be viewed by moderators.
“We are grateful to the fantastic team at Cirrus Assessment for working with us to design a new form of online assessment to test digital journalism skills. The tight deadlines we set and use of technology have made this a challenging but rewarding project.”
Joanne Butcher – Chief Executive, NCTJ
The NCTJ prepared a feedback report using system data for unsuccessful students. This included feedback on questions where they did not score well. It also included comments from markers and guidance on where to focus future study. Tutors were then able to use this data to inform future planning.
Delivering exams anytime, anywhere
For NCTJ, the benefits of digital exams are numerous:
- Candidates gave detailed feedback after each exam. This was overwhelmingly positive. They found the user interface extremely intuitive and simple to use and enjoyed using video and audio elements within the assessment.
- Additional options are also available to students with visual and auditory impairments. Accessibility can be further enhanced by adding remote invigilation for students with difficulties accessing test centres.
- Test centre administrators found the Cirrus Platform very easy to work with and felt well supported by both the NCTJ and Cirrus Assessment. They had access to Cirrus Assessment’s customer success portal where they could find detailed guidance on using the system. Specific guidelines for tutors, trainers, invigilators and students were also provided.
Ultra-fast implementation
Perhaps the most unique aspect of the project was not only the enormous task of making the switch from paper to digital, but the condensed timeframe in which it was done within. This meant that absolutely everyone on both teams needed to come together as one united force in order to make the digital shift happen smoothly. This transformation would not have been at all possible had it not been for the communication, support, hard work, and open-mindedness of these two companies.
Achievements
- The Essential Journalism exam is delivered in a secure, lock-down browser-environment that complies with the regulators’ general conditions and its own rigorous exam policies and procedures.
- The Cirrus platform provides an innovative and demanding test of students’ ability to produce work to a deadline and reflects the journalistic skills needed to operate in a multimedia newsroom.
- Test centre administrators not only felt that the Cirrus platform was user-friendly, but that they were very well-supported by the Cirrus team, every step of the way.
- Students’ satisfaction rates have increased with an overwhelmingly positive feedback post exam
- NCTJ’s environmental footprint has decreased.
