E-ATP Vienna: Navigating Digital Equity, Fair Access, and AI in E-Assessment

Share via:

As we’re settling back into the swing of things here in Utrecht, we would love to take the time to reflect on the wonderful E-ATP conference. We couldn’t have asked for a better backdrop for E-ATP: the city of Vienna, with its rich history, imperial charm, and musical legacy, has always been a beacon for culture, learning, and imagination. So, pull up a chair, sip on a comforting cup of Viennese coffee, and let’s discuss our key highlights from the E-ATP conference!

This year’s theme was E-ATP – The European Home of Assessment, and the topics this year to focus on were: transforming, candidates, innovation, personalisation, stories, and purpose. Reflecting on these themes from the E-ATP, Cirrus Assessment sees a horizon that is not just bright, but truly redefining for the e-assessment industry. The journey from traditional paper to dynamic digital forms of assessment heralds a significant shift not only in methodology but in the essence of what it means to evaluate knowledge. As an e-assessment company, we recognise that our candidates are evolving, their values are shifting, and their expectations are more diverse than ever. Digital equity, fair access, and ensuring no candidate is left behind due to digital poverty are more than just buzzwords and boxes to tick for us; they represent the very ethos we aim to embrace. 

We are on the brink of an era where technology, like artificial intelligence and machine learning, offers us tools that are not just disruptive but also profoundly enabling. It empowers us to weave personalisation into the assessment fabric, tailoring each test to align with the diverse needs of industry and employment. Each story shared, whether it’s a lesson learned, a refugee empowered, or a groundbreaking innovation, is a testament to the passion and resilience of this community. It reminds us of our collective purpose: to create authentic assessment programs that uphold the highest standards and best practices. We are not just looking to secure our assessments from external threats but to secure a brighter future for every candidate who entrusts us with their aspirations. As we ponder the myriad possibilities laid out at the conference, Cirrus Assessment is more committed than ever to championing innovation, ensuring fairness, and embracing the future with unwavering optimism.

E-ATP Vienna – Day 1

Cirrus touched down in Vienna on September 19th and after admiring the gorgeous Saint Stephen’s Cathedral and having the world famous Figlmuller schnitzel, we settled into our hotel to get a good night’s rest to wake up refreshed and hit the conference hard.

September 20th, Day 1 began with the aroma of freshly brewed coffee at the Marriott, Stadtpark as we busied ourselves with setting up our booth. The ring of the bell marked the beginning of E-ATP, but before delving into the richness of the day’s first keynote speaker, we must bid adieu and good luck to Lauren Scheib, whose commitment to the ATP has left an indelible mark. Here’s to new beginnings, Lauren, and a future filled with joy! 

Tearing up the Certificates: The Measure Is Not the Target

The opening keynote speaker was German Bennci from Code Your Future, who introduced us to their transformative approach. With an ethos that prioritises individual skill-building over paper qualifications, Code Your Future stands as a beacon for immigrants and refugees. By addressing fundamental needs—think Maslow’s foundational rungs—they offer a pathway for newcomers to not only survive but thrive, finding camaraderie, embracing their new environment, and reaching their own self-actualisation. At Code Your Future, traditional grading has been relegated to the annals of history. Instead,they provide students with solutions, teach them to self-review, and base graduation on the practicality of these solutions.

Key takeaways:

  • Code Your Future transcends traditional educational norms, emphasising real-world skills over certifications, especially for those navigating the challenges of immigration.
  • By eliminating grading, and equipping students with solutions and self-review techniques, the learning process becomes more holistic, focusing on genuine understanding and application.
  • The graduation criteria are comprehensive, requiring learners to not only construct solutions but demonstrate their viability, recognise potential shortcomings, and chart the path forward.

The Intersection of Authenticity and Innovation: VR and AI in Medical Assessment

In a captivating presentation, Jeremy Carter delved into the  potential of Virtual Reality (VR) and Artificial Intelligence (AI) in medical assessment, shedding light on their growing application across multiple continents. Recognising the inherent challenges posed by clinical assessments—such as cost, standardisation issues, and logistical constraints—he underscored the compelling case for technological interventions. Offering insights from organisations in both the UK and the US, Carter showcased how VR and AI can foster enhanced reliability in evaluations, provide invaluable feedback to examinees, expedite result delivery, ensure equitable access, and curtail associated costs. The session wasn’t just theoretical: attendees were immersed in VR patient scenarios and navigated clinical decisions aided by AI within an internal medicine context. Through a global lens, Carter encapsulated the novel ways in which these technologies are being employed in Australia, the UK, and the US, cautioning about potential pitfalls and casting a vision for their future role in medical assessment.

Key Takeaways/Learning Objectives:

  • VR and AI present promising solutions to the longstanding challenges of clinical assessments, including cost, standardisation, and logistics, with demonstrated potential to enhance reliability and equity.
  • Hands-on VR patient encounters and AI-powered clinical decision-making tools offer a more authentic, immersive, and real-time assessment experience.
  • As the technology’s footprint expands globally, there’s a need to be cognisant of potential pitfalls while remaining optimistic about its novel applications in the domain of medical evaluations.

The End of Test Centres?

Our very own Vegard Sivertsen, CEO Cirrus Assessment, and Martijn van der Jagt, Account Manager Cirrus Assessment took the stage to talk about LOFT 3.0 – our blueprint technology that combines the power of large item banks, equivalent test forms, and parameterisation to deliver calibrated, fair, and randomised exams anytime, anywhere in the world. And who better to share a virtual stage with than Norman Grotzbach of Galileo Global Education/AKAD University? 

This pioneering German university is revolutionising the examination landscape by leveraging advanced blueprint technology to craft balanced, tailor-made, and legally sound exams. This institution, offering over 90 online degree programs suited for varied demographics such as working professionals, families, overseas residents, and individuals with disabilities, shuns traditional semester structures, placing the decision of exam timing squarely in the students’ hands. Through the innovative use of blueprint technology, AKAD creates well-balanced assessments grounded in specific criteria, employing taxonomies, outlined learning objectives, and core topics to determine item difficulty. Crucially, exams come with integrated difficulty tiers and credit points, with each item’s worth intrinsically tied to the time it takes to answer. By melding taxonomy with response time, a unique metric for gauging difficulty and credit value emerges, allowing the institution to churn out unbiased, legally defensible, and randomised exams at a moment’s notice, granting unmatched flexibility to its global student base.

Key takeaways:

  • The German university’s use of advanced blueprint technology ensures the creation of balanced, customisable exams that are legally defensible.
  • The unique integration of taxonomies, learning objectives, and response times allows for a novel way to determine item difficulty and credit points, ensuring fair evaluations.
  • The ability to generate randomised exams on demand empowers students worldwide with unprecedented flexibility in their education journey.

After working long and hard at E-ATP, we thoroughly enjoyed happy hour with our colleagues, and finished off the day with a traditional Viennese dinner at Gasthaus Zur Oper with our fabulous partners Sojin Lee of Blees.ai and Ian Browne and his team at Chartered Accountants Ireland.

E-ATP Day 2

Day 2 kicked off bright and early with an interesting debate: ‘The Promise and Pitfalls of AI in European Assessment: An Executive Debate’ given by Neil Wilkinson, John Kleeman, Ruben Garcia, Alina von Davier, and Paul Edelblut. AI has proven, yet again, to be the hot topic we just can’t stop talking about, and for good reason. The debate over generative AI in European assessments triggers a blend of emotions – anticipation, excitement, fear, apprehension, curiosity. As top industry voices debate AI’s potential, Cirrus is reminded of the age-old tension between man and machine. AI, with its cold efficiency and relentless logic, promises accuracy, but at what cost? There’s a certain art to assessments, a human touch, an educator’s intuition that an algorithm might overlook. In this tech-driven world, where AI’s limitations are still a pressing concern, can we trust it with our education? And what does it mean for the diligent worker, the assessor, the educator? Are they to be replaced, or can they coexist? Then there’s the elephant in the room—the intellectual property rights and the security implications. As technology races ahead, are we unwittingly opening Pandora’s box? We don’t have all of the answers, but a balance must be struck between AI making us more efficient and augmenting education, but it shouldn’t overshadow it.  

Defeating the Marking Time Bandit: AI-Automated Essay Marking

In a great session, Sojin Lee, CEO and founder of Blees.ai, and Vegard Sivertsen, CEO and founder of Cirrus Assessment explored the ground-breaking potential of AI-automated essay marking, addressing a longstanding issue in education: the time-intensive and often biased nature of essay marking. While acknowledging the intrinsic value of essays in assessing students’ depth of understanding, Sojin presented a Blees.ai x Cirrus Assessment x Chartered Accountants Ireland solution—AI-automated marking on an end-to-end e-assessment platform—touted to not only significantly reduce marking time and costs but also expedite result delivery, thereby enhancing learner satisfaction. 

The technological marvel wasn’t without its concerns, however. Many attendees raised valid questions about potential biases embedded within AI, a matter Sojin candidly addressed by admitting that, since AI learns from human inputs, biases remain a challenge. Nonetheless, the session shed light on the project’s rigorous commitment to AI ethics, emphasising research-driven efforts to bolster fairness and reduce such biases. It became evident that, when appropriately calibrated, AI has the potential to bring objectivity and consistency to essay marking, challenging the inherent inconsistencies and prejudices that may inadvertently creep into human evaluations.

Key Takeaways/Learning Objectives:

  • AI-automated essay marking can significantly reduce the time and cost associated with traditional marking methods, offering swifter feedback and enhancing student satisfaction.
  • While AI presents an opportunity for more objective and consistent evaluations, concerns about biases—stemming from human-driven training data—are genuine and need to be rigorously addressed.
  • Ethical considerations in AI essay marking are paramount; ongoing research and conscientious implementation can ensure fairness, inclusivity, and reduced bias in assessments.

OpenAI and GPT, Supporting SMEs in Rapid Item Writing

If you remember back to our LOFT 3.0 presentation, there were questions and concerns surrounding creating large enough item banks in order to accommodate an endless stream of randomised exams, and Vegard mentioned AI/ChatGPT creating item banks as well as parameterisation as a solution. This insightful presentation by Patrick Coates explains just this! 2023’s burgeoning enthusiasm for OpenAI models like GPT was critically examined. 

Drawing attention to the integration of GPT into a content authoring system, Patrick showcased how this synergy empowers Subject Matter Experts  to craft superior-quality test items with remarkable efficiency. By utilising Test Blueprint parameters, such as job roles and topics, and SME-driven brief item descriptions, the platform ensures distinctness in content creation. Furthermore, the system’s adept handling of GPT’s tuning parameters, like ‘temperature,’ streamlines the content generation process. This approach allows GPT to construct the foundational elements of test items, which the SME can subsequently refine. With the aid of natural language processing, this platform offers SMEs the dual advantage of rapid test item creation and enhanced accuracy. Patrick’s preliminary findings indicate that GPT-assisted content creation has the potential to elevate productivity levels by sevenfold, while diversifying the nature of test items. Items produced with AI assistance hold their ground during peer reviews, showing no signs of compromise in quality.

Key Takeaways/Learning Objectives:

  • Integrating GPT with content authoring systems can significantly expedite the test item development process while ensuring high-quality output, controlled by Subject Matter Experts.
  • The fusion of Test Blueprint parameters and GPT’s tuning capabilities ensures diverse and distinct content generation, preventing overlap in item creation.
  • Preliminary studies suggest that AI-assisted content creation may lead to a sevenfold boost in productivity without compromising the integrity of the test items during peer reviews.

A Dutch Tradition: Borrel!

The end of Day 2 drew to a close, and what better way to send it off than a borreltje! Let’s dive into a quintessential Dutch tradition: the “borrel.” Imagine it’s the end of a long workday, and rather than heading straight to the pub for a pint, the Dutch gather around for a “borrel.” It’s a laid-back, cozy gathering, often in someone’s home or a bar (or at the Cirrus E-ATP booth ;)), accompanied by drinks (typically beer, wine, or something stronger like jenever) and a spread of delightful Dutch nibbles (or, “borrelhapjes’) like bitterballen (deep-fried ragu), kaas (cheese), kaastengels (cheese sticks), or patatjes met (fries with mayonnaise). Although the we couldn’t take fried snacks with us on the plane, we did have a borrel at our booth with drinks! Think of it as the Dutch equivalent of “happy hour,” but with a touch more gezelligheid (a Dutch word that’s a bit tricky to translate but embodies warmth, conviviality, and togetherness). So, next time you find yourself visiting Cirrus Assessment in the Netherlands, don’t just stop at the tulips and windmills; come to our office for a borrel! Cheers, or as the Dutch say, “Proost!”

We closed out the night at Strand Bar Hermann, sat on the banks of the Danube River in Vienna, a laid-back oasis after the fast-paced pulse of the conference. It was the perfect urban escape to catch the sunset and relax with our colleagues before the last day.

E-ATP Day 3

Unpacking Myths and Misunderstandings about the Importance of Diversity, Equity, and Inclusion for European Companies

In an engaging session on diversity, equity, and inclusion (DEI), participants confronted prevailing myths and misconceptions surrounding DEI in European corporate cultures. The conversation highlighted alarming statistics: while research underscores the tangible benefits of DEI to organisational success, over half of surveyed European executives view these initiatives as mere distractions. The session pivoted around dispelling notions that check-the-box approaches or technological solutions alone can address DEI complexities. Crucially, it emphasised that a mere demographic mix without actual equity and inclusion falls short of genuine diversity. The discussion also shone a light on the pitfalls of government-mandated quotas, emphasising that true DEI strategies are rooted in the very ethos of company culture, not in obligatory compliances.

Key Takeaways:

  • DEI is not just about meeting quotas or checking boxes; it’s about integrating these principles into the heart of an organisation’s culture.
  • True diversity extends beyond mere demographics and requires both equity and inclusion to make an impact.
  • While assessments can be invaluable tools for sourcing diverse talent, they must be executed with genuine commitment, ensuring no group faces discrimination.

Closing Keynote – Oded Vanunu: The Upside Down: Unraveling the Underworld of Cybercrime Syndicates

Cybercrime, much like any modern enterprise, has evolved into a sophisticated operation marked by its innovation, expertise, and recruitment of specialised talent. This deep dive session drew compelling parallels between the world of underground cyber operations and traditional mafia syndicates, a comparison that illuminated the intricate web of roles and structures in both realms. Much like the mafia, which operates through a network of capos, soldiers, and associates to carry out its illicit activities, cybercrime syndicates have developed a hierarchy that includes strategists, hackers, and foot soldiers. Both entities are driven by profitability, albeit through illegal means, and maintain a clandestine presence, often hiding in plain sight.

Major cyber-incidents were dissected, providing invaluable insights into the modus operandi of these virtual criminals. From a cybersecurity perspective, understanding this underworld is vital. Just as authorities have historically battled the mafia with intelligence-led policing and witness protection, today’s organisations need to adopt proactive cyber defenses, continuous monitoring, and regular employee training. In essence, to fend off this digital mafia, businesses must remain ever-vigilant, adaptive, and agile, ensuring that their virtual walls are as fortified as their physical ones.

Key Takeaways:

  • Cybercrime operations bear a striking resemblance to traditional mafia syndicates, with a defined hierarchy and an intricate network dedicated to illicit profitability.
  • An in-depth understanding of major cyber-incidents helps in decoding the strategies and vulnerabilities exploited by cyber criminals.
  • To protect against these evolving threats, organisations must combine proactive cyber defenses with continuous education and awareness campaigns.

And that’s a wrap! After breaking down the stand and making our way to the Vienna airport we said goodbye to another great conference!  We are so proud to have participated, not only with a booth but with two presentations.  A very special thanks to our amazing partners: Ian Browne & crew at Chartered Accountants Ireland, Sojin Lee at Blees.AI, and Norman Grotzbach with AKAD University and Galileo Global Education

And to everyone we had the opportunity to meet, reconnect with, and chat to – a big thanks to you all! Andriani Papapericleous and Muhammad Tayyb Rahman – British Council, Catherine Christmas – ICAEW, Ellen Bakkers and Sofie Van Damme – FOD BOSA, Karolina Cisek-Ndlova and Graham Morris – Comms Multilingual, William G Harris – Association of Test Publishers,, Nadav Blum – National Inst. for Testing & Evaluation, Denis Saunders – Future Strategies Limited, Yordanka Kavalova – Oxford University Press, Beatrice Segura-Harvey – Ecctis, Tim Burnett – Education Tech Consulting, Lucy Midgley-Robbins and Elizabeth Smith – Driver and Vehicle Standards Agency, Jeremy Carter and Emily Johnstone – Sentira XR, Chelsea Dowd – Questionmark, 

Share via:
Topics
Cristina Gilbert
Cristina Gilbert
Copywriter and digital content enthusiast, Cristina is motivated by the fast-paced world of e-assessment and the opportunities online exams give students to thrive.
Would you like to receive Cirrus news directly in your inbox?
More posts in Company News
 

Curious about all things e-assessment?

As Cirrus looks to the future, we are excited to bring you the latest news, trends, and useful information about the industry.

 

Subscribe to the monthly Cirrus Examiner to join our ever-growing community of people passionate about the unbridled potential of EdTech.