Navigating the pandemic: digital assessment at Brunel

Context

Brunel University London is a multidisciplinary research-intensive university, and with over 17,000 students and 2,500 staff from all over the world, it is one of the most diverse institutions in the UK. The past two years have been 'interesting'; like others, Brunel had to act quickly to mitigate the impact of the Covid-19 pandemic and national lockdowns on learning, teaching and assessment, and ensure that students were able to continue or complete their studies as planned.

Achieving this was made easier by already having a digital assessment platform; Brunel has been using WISEflow since 2015/16 across various departments. However, it was still a big task to move almost all exams online in spring 2020.

As a new 'normal' is being established, Professor Mariann Rand-Weaver, Vice-Provost Education, and Robyn Fitzharris, Executive Officer to the Vice-Provost, reflect on the adaptability of digital assessment, and how exams data was used to monitor equality and inform the response to student welfare issues.

Initial response

‘When the first UK lockdown was announced in March 2020, we were a month away from the first exam on 20 April. Although we had been using WISEflow for a number of years, only 20% of students had experienced a digital exam. However, all staff and students were familiar with WISEflow from their assignment submissions, and we took the decision that the remote exams would be in open-book format. This was not the time to introduce our students to a lockdown browser! Instead, we changed question types and ran exams of similar duration to that planned in order to negate the fact that students had access to notes, books and various web browsers.’

'Concerns were raised before and during the process by both staff and students. These misgivings related primarily to the availability of equipment – laptops, Wi-Fi and quiet spaces – and so it became clear we had to do more than transferring exams online.’

Daily monitoring of exams

‘As most of our exams kept the original duration, we carefully thought about how we would mitigate against students missing the upload cut-off time and flooding us with panicked calls. We still got some of those, but by setting a 'hidden' time allowance in WISEflow, we avoided students being timed out if there was an issue at the point of submission.’

'While we were concerned about the impact of technical issues and considered that a hard cut-off point would have been unfairly detrimental to students, we didn't want students to get an unfair advantage. By monitoring every exam, we knew that only about 10% of submissions were late, mostly submitted within a couple of minutes of the deadline and therefore not of concern. Students who submitted later were followed up using the WISEflow exam log to understand whether their claims of technical problems were real. The pragmatic approach that Brunel took served us well, and being able to use the digital footprint has been key to ensuring that no one was taking advantage of the system.’

Assessing student engagement using WISEflow pi

‘Daily monitoring was also used to determine how many students were doing exams - having rightly given students the option to defer their exams to later in the year, we were nevertheless concerned that if large numbers did this it would have a large operational knock-on effect. WISEflow API data allowed us to track exam engagement daily, flagging particular patterns by different student groups. Combining this with wider student information, we compared those who took exams with those who didn't and asked whether they were unable to access assessments (digital poverty?), and if we needed to be proactive in reaching out to support them.’

‘Whilst we have ways of measuring levels of disadvantage, such as household income, we still struggle to find reliable data on an evolving basis. But we know that lots of our students are from deprived areas on London – 25%-30% come from some of the most underprivileged areas in the UK.’

Data-informed support

‘We were pleased to find that a high proportion of students (~87%) were able to access and successfully complete exams from home in May 2020. However, engagement data by student characteristics highlighted that disabled students, mature students and those from deprived areas were less likely to sit exams in the Spring term. We proactively reached out to these students, which helped inform our additional support put in place for the August (and subsequent) exam period such as bookable quiet spaces, increases to hardship funds and more loanable laptops.’

‘An analysis of degree qualifications only saw small fluctuations in award outcomes for these groups, which we feel validates our approach to how we supported students through a very difficult time. We will be continuing to monitor engagement data in the future to ensure our assessment methods do not inherently disadvantage particular groups of students.’

Precipitating change

‘Some staff used the opportunity of the enforced remote, open-book exam format to rethink assessment. For example, the traditional three-hour essay used in Politics and History has been replaced with an extended exam (or shortened assignment) of 24-48 hours, allowing more expansive questions to be set. Other innovations include students being given six to seven hours to research and integrate information from a range of sources to address the questions.'

‘What both these examples have in common is that they represent authentic assessments; students may in future be given a day to write reports or provide some analysis. These new formats are therefore closer to real-life experiences than being locked in a room for three hours with a pen and paper! Being forced to do things online and remotely has opened our minds up to what is possible and recognition that doing things differently can be better. And students are positive too - they take digital assessments in their stride and reported enjoying working on their laptops – essentially, what’s the big deal?’