Navigating the pandemic: digital assessment at Brunel

Context 

Brunel University London is a multidisciplinary research-intensive university, and with over 17,000 students and 2,500 staff from all over the world, it is one of the most diverse institutions in the UK. The past two years have been 'interesting'; like others, Brunel had to act quickly to mitigate the impact of the Covid-19 pandemic and national lockdowns on learning, teaching and assessment, and ensure that students were able to continue or complete their studies as planned.  

Achieving this was made easier by already having a digital assessment platform; Brunel has been using WISEflow since 2015/16 across various departments. However, it was still a big task to move almost all exams online in spring 2020. 

As a new 'normal' is being established, Professor Mariann Rand-Weaver, Vice-Provost Education, and Robyn Fitzharris, Executive Officer to the Vice-Provost, reflect on the adaptability of digital assessment, and how exams data was used to monitor equality and inform the response to student welfare issues.  

Initial Response 

‘When the first UK lockdown was announced in March 2020, we were a month away from the first exam on 20 April. Although we had been using WISEflow for a number of years, only 20% of students had experienced a digital exam. However, all staff and students were familiar with WISEflow from their assignment submissions, and we took the decision that the remote exams would be in open-book format. This was not the time to introduce our students to a lockdown browser! Instead, we changed question types and ran exams of similar duration to that planned in order to negate the fact that students had access to notes, books and various web browsers.’  

'Concerns were raised before and during the process by both staff and students. These misgivings related primarily to the availability of equipment – laptops, Wi-Fi and quiet spaces – and so it became clear we had to do more than transferring exams online.’ 

‘We advertised a laptop loan scheme, to try and ease the disadvantaging of students across the University; however, we thought that more was necessary to safeguard students. We therefore took the decision to allow students to defer exams, without the need for approval. This gave students flexibility and reduced the burden of the impending exams amid constantly changing and challenging external circumstances.’ 

Daily Monitoring of Exams 

‘As most of our exams kept the original duration, we carefully thought about how we would mitigate against students missing the upload cut-off time and flooding us with panicked calls. We still got some of those, but by setting a 'hidden' time allowance in WISEflow, we avoided students being timed out if there was an issue at the point of submission.’  

'While we were concerned about the impact of technical issues and considered that a hard cut-off point would have been unfairly detrimental to students, we didn't want students to get an unfair advantage. By monitoring every exam, we knew that only about 10% of submissions were late, mostly submitted within a couple of minutes of the deadline and therefore not of concern. Students who submitted later were followed up using the WISEflow exam log to understand whether their claims of technical problems were real. The pragmatic approach that Brunel took served us well, and being able to use the digital footprint has been key to ensuring that no one was taking advantage of the system.’ 

Assessing Student Engagement Using WISEflow API 

‘Daily monitoring was also used to determine how many students were doing exams - having rightly given students the option to defer their exams to later in the year, we were nevertheless concerned that if large numbers did this it would have a large operational knock-on effect. WISEflow API data allowed us to track exam engagement daily, flagging particular patterns by different student groups.  Combining this with wider student information, we compared those who took exams with those who didn't and asked whether they were unable to access assessments (digital poverty?), and if we needed to be proactive in reaching out to support them.’ 

‘Whilst we have ways of measuring levels of disadvantage, such as household income, we still struggle to find reliable data on an evolving basis. But we know that lots of our students are from deprived areas on London – 25%-30% come from some of the most underprivileged areas in the UK.’  

Data-Informed Support  

‘We were pleased to find that a high proportion of students (~87%) were able to access and successfully complete exams from home in May 2020. However, engagement data by student characteristics highlighted that disabled students, mature students and those from deprived areas were less likely to sit exams in the Spring term. We proactively reached out to these students, which helped inform our additional support put in place for the August (and subsequent) exam period such as bookable quiet spaces, increases to hardship funds and more loanable laptops.’  

‘An analysis of degree qualifications only saw small fluctuations in award outcomes for these groups, which we feel validates our approach to how we supported students through a very difficult time. We will be continuing to monitor engagement data in the future to ensure our assessment methods do not inherently disadvantage particular groups of students.’ 

Precipitating Change 

“Being forced to do things online and remotely has opened our minds up to what is possible and recognition that doing things differently can be better”

‘Some staff used the opportunity of the enforced remote, open-book exam format to rethink assessment. For example, the traditional three-hour essay used in Politics and History has been replaced with an extended exam (or shortened assignment) of 24-48 hours, allowing more expansive questions to be set. Other innovations include students being given six to seven hours to research and integrate information from a range of sources to address the questions.'  

‘What both these examples have in common is that they represent authentic assessments; students may in future be given a day to write reports or provide some analysis. These new formats are therefore closer to real-life experiences than being locked in a room for three hours with a pen and paper! Being forced to do things online and remotely has opened our minds up to what is possible and recognition that doing things differently can be better. And students are positive too - they take digital assessments in their stride and reported enjoying working on their laptops – essentially, what’s the big deal?’ 

Looking to the future 

‘We consider ourselves fortunate that we had a digital assessment platform when the pandemic hit – WISEflow allowed us to continue assessing students in a time-limited way where appropriate; to run lock-down exams for students that were familiar with those; and monitor what was happening in almost real-time. Through this experience, we have shifted as an institution - the exams in 2020/21 were almost non-events - and although there has been a return to some pen-and-paper exams in the latest exam period (May 2022), the majority of disciplines are using WISEflow - remotely or on campus.' 

‘Our experiences over the past two years have highlighted the many advantages of digital exams. We 'jumped' in when we first used WISEflow, and that's what we had to do again in May 2020 – to me, this has been advantageous, as adoption of WISEflow exams would otherwise have been slow. Credit should go to our staff who have risen to the challenge! The main disadvantage has been the resulting student misconduct (plagiarism and collusion), and although we have made use of the WISEflow Facial Comparison software, this, in our minds, is not a solution. Nor do we think proctoring is an avenue we want to go down, and going forward, we are likely to use invigilated on-campus exams where access to resources have to be controlled.'  

“‘Who could have imagined when we started our relationship with UNIwise in 2015 and planned our future direction that it would be significant in coping during a pandemic five years later?”

Concluding Comment 

‘Who could have imagined when we started our relationship with UNIwise in 2015 and planned our future direction that it would be significant in coping during a pandemic five years later? Many higher education institutions had to use their VLEs to submit exams, as this was their only option, whereas we had a dedicated assessment platform that also allowed us to monitor and mitigate the impact on students.’  

‘Exams are ordinarily short, high-stakes activities. The information we gathered during the digital assessment process identified issues related to those specific exams; however, the data shows a need to think more broadly – we have found that presentations, for instance, result in the same kinds of concerns for students: problems with Wi-Fi and a lack of quiet spaces, for example.'  

‘The data helped Brunel to think about what kind of support is needed for students in regard to all teaching and learning, not just specific assessments. The experience has really shone a light in that sense – we must always be thinking about how assessment can be better and equitable for all our students.’