“Online examinations help foster the skills candidates need in the digital age”: The European Patent Office on running the European Qualifying Exam on WISEflow
The EPO headquarters. All images courtesy of the European Patent Organisation.
The European Patent Office (EPO) is an intergovernmental agency that examines European patent applications, enabling inventors, researchers and companies from around the world to obtain protection for their inventions in up to 44 countries. The European Patent Academy, which runs the European Qualifying Exam, is the external education and training arm of the EPO.
The EPO began using WISEflow in March 2021.
Context
In order to be a European patent attorney, it is necessary to pass the European qualifying examination (EQE). This requires at least three years of preparation, including at least two years of relevant work experience followed by a pre-examination that assesses candidates’ ability to answer questions about a whole range of legal issues and the drafting of claims. Up until the pandemic hit, the EQE was held once a year in examination centres across Europe; thousands of handwritten answer papers were scanned and marked twice.
In 2020, the EQE was cancelled due to COVID-19. As the pandemic continued, it became clear that a physical exam would not be possible in 2021 either. This year, there were almost 3,800 candidates waiting to sit the EQE, including a backlog of almost 2,000 candidates from 2020. Postponing until 2022 would have meant as many as 5,500 candidates waiting to sit the exam.
This partly explains how the March 2021 EQE ended up taking place on WISEflow during the course of a single week, with 130 digital exam pages in each of the EPO’s three official languages. Five exams were split into 17 “flows”; 3 733 candidates sat 24 hours of exams in just the five days. There were 80 online invigilators and a rapid deployment team of 15 for first- and second-level support.
What follows is an interview with Xavier Seuba, Director of the Patent Academy and EQE, and Simon White, Director of IT Cooperation.
From left to right, Xavier Seuba and Simon White of the EPO
Xavier Seuba (XS): In order to guarantee the quality of the patent system and of the patents granted by the EPO, we must ensure that the professionals representing parties in front of the EPO have the highest standards of knowledge and expertise. The EQE is the entry ticket to the European patent system. Having a good EQE means we preserve the highest professional standards and, therefore, that the system at the EPO runs smoothly and efficiently. The EQE helps uphold quality throughout the member states of the European Patent Organisation: professional standards are the same in Portugal as in Poland, and the EQE is the instrument that makes this possible.
We actually needed several decades to move to computer-based exams. During the past 40 years though, we’ve provided a sophisticated exam with levels of security, infrastructure, design, and the necessary checks and balances to match – at all times, throughout the EQE cycle. We presently have nearly 400 people involved in the management, drafting and marking of EQE papers. Previously, candidates had to travel to places where the exam took place. And as the EQE is an open-book examination, they travelled across Europe with suitcases full of notes and reference books. Imagine the amount of paper generated by three years of preparation!
Simon White (SW): The exams in March 2020 were cancelled relatively close to the dates on which they were due to take place. You can imagine the administrative implications. The exam centres were organised within each member state, so the contracts between the Office and the governments in those members states had been concluded 18 months earlier, and the dates announced 12 months earlier – then in late February, we had to say, sorry we’re not doing next month’s exams. So, our first thought was that this couldn’t go on forever and we’d run the rescheduled exams in September. But then we saw an opportunity to digitise this whole activity.
At the EPO, we already do e-learning, so the office had been upskilling and delivering online learning for staff; with new management in place since 2018, there was an openness to change.
XS: We’re now an almost fully digitalised organisation. The past year has shown us that all of us – 6 400 people – can work from home and business can go on as usual.
SW: But the EQE had always seemed so analogue.
XS: We had run a pilot in the recent past, which involved providing a limited number of computers where the exams were offered to candidates, but it was a very limited digitalisation effort.
“We chose UNIwise because we were looking for a partner who we could grow with.”
SW: And then 2020 arrived. The way we did things with the EQE – and thought we’d always be doing things – was upended by the crisis. Crisis brings agility, change at a pace that was previously unheard of. We changed more than we expected this year, certainly more than we would have thought possible.
XS: Once we’d decided to move online, we did a market study, analysing nine companies and shortlisting three. We assessed how existing systems matched what we needed. Not only that: we also assessed how flexible the system could be. We needed a lot of flexibility, because we could not change the exams. The exam papers were ready and we could slightly adapt them, but substantive changes were impossible; the format could only be changed to a very limited extent. We were also looking for security and an exam environment where the candidate felt comfortable. Some systems were too invasive, establishing very high levels of control over candidates. But our intuition said, if we do that, candidates will feel that they are overly supervised, this may make them nervous, impact their performance and they may also feel we don’t trust them. We needed to avoid all that.
We chose UNIwise based on the features of WISEflow, but also because of the flexibility to adapt the system to the features of the EQE. We were also aligned on the principle of trusting people. Security measures are, of course, necessary, but in principle, we have to trust people to be honest.
SW: We’re less risk-averse than we used to be. And, ultimately, there was no low-risk approach here. We were looking for a partner who we thought we could grow with. Everything the European Patent Academy is doing is changing, and there’s a lot of new stuff we need to do that the Office has never really managed to do before. We need people to come with us on that journey. We found UNIwise and thought, we wonder if they want to grow, too? And they did.
The alignment between EPO and UNIwise was instinctive. We all knew this was a good match.
XS: Responsiveness was really important for us: to know that, when a situation had to be addressed, the provider would be there. Our perception is that this responsiveness has something to do with the fact that UNIwise thinks about the end user, the candidate. We needed to make sure is that we provide all the conditions that allow the candidate to show us that she has acquired the knowledge we expect of her. The fact that you have higher and professional education in your DNA also makes a difference because you understand what people go through during an exam, that we’re going to have to handle personal moments.
SW: The ability we felt we had to explore issues and arrive at decisions, to get it done, was really important.
XS: In the last round of the platform selection process, we asked UNIwise whether we would be able to adapt the system regarding a number of features we deemed would improve the experience of candidates, while also increasing invigilation functions by adding audio proctoring capabilities.
SW: We were open about the pressure we were facing. And they knew that the timelines were very tight - the improvements we needed would take until January at the earliest, and the exams were in March. UNIwise was willing to take a risk.
XS: Previously, the EQE papers were specially transported to examination centres. The exams were kept in a safe and an extremely limited number of people knew the text of the final exams. We had to replicate that level of physical security in the digital environment.
SW: After we signed the contract, the first thing we had to do was to assemble some IT resources ourselves, which we hadn’t necessarily foreseen. Our team and the EQE secretariat both had to learn all about WISEflow, and UNIwise worked with them so that we learned a lot about each other to get the exam done. The most important stages for me concerned the preparation of the mock exams; they were vital. We did three mock exams: the first two were chances for people to go into WISEflow and test, and the third was a time-boxed event very like the real thing with all of the people in play. Not everyone came fully on board with the practices, but those who did certainly reaped the benefits.
“By moving our exams online, we are fostering the skills candidates need in the digital age.”
XS: We had our mocks and a period of testing the entire system. And in the meantime, we had to create or adjust information for candidates: a user guide, a tutorial, extensive FAQs. We did three online seminars to explain to candidates how the system would work, with each seminar booked by 1,000 people.
The week of the exams started extremely well with the pre-exam on Monday. I remember talking to Simon in the afternoon and saying, this is going really well.
SW: It was one of those “Are you sure? It can’t be this easy” moments.
XS: Unfortunately, the next day, there was a problem with two of the three language versions of one paper, which were not uploaded on time. This was limited to one third of one paper and impacted candidates differently. While some did not even notice, for some the situation was really problematic. The splitting of the paper into three parts, each of them in three languages, allowed the Board in charge of marking the exams to address the incident by awarding all candidates the same compensation mark in that specific part of the paper.
Every day after the examinations concluded, we met with all those involved in the management, around 90 people, assessing what measures could be implemented to do things even better the day after. Seen from behind the scenes, together with the service provided to candidates at a very difficult moment in the pandemic, the motivation and engagement of the team was the best aspect of the EQE 2021. Even in the difficult moments, we knew that we were doing something unprecedented and providing a service that candidates needed and welcomed. The motivation of the team was enormous, and we are also now very grateful for the feedback provided.
SW: A number of candidates have said very clearly that they appreciated having the chance to do the exam. People appreciated not having to travel, not having to carry around so much paper, the very fact that there was an exam at all, which had by no means been guaranteed.
The exam is a major gateway, with huge financial ramifications for successful candidates. The cohort is a combination of lawyers and engineers, and while a few people were a little bit peeved after the first two days, on the whole, the feedback was pretty good.
XS: Candidates made a big effort to adapt to the new setting and started following our instructions as of December 2020, when we first opened the platform for testing. In the survey we sent to candidates after the EQE, 97% said they welcomed the opportunity to sit the EQE in 2021. Candidates appreciated being able to sit the exams in a place of their choosing (76%) and using a computer to type their answers (79%). Candidates also appreciated being able to save on travel time and costs, limit the risk to their health and avoid issues with travel logistics. There was a feeling among some candidates that there wasn’t enough time, but this seems to apply to almost all examinations, including the EQE. Candidates have also helped us by indicating which aspects should improve, from the layout of the examination portal to the possibility to annotate papers, amongst others.
SW: The thing is, this project is not just about saving the 2021 EQE. It’s about systemic change. We’re not in the event organisation business. We’re supposed to examine patents. There’s a lot of things at play here though that aren’t part of our core business, and by outsourcing these things to people whose business it is, like UNIwise, we can deliver a better service.
We need to be able to deliver a broader spread of assessment types and do so more flexibly. The days of a seven-year apprenticeship culminating in a single test with a hammer and piece of metal are long gone. We need to deliver fit-for-purpose assessment, using modular technology. WISEflow is great, it’s secure and it works in the cloud.
XS: By moving the exam online, we are helping to foster the skills that candidates need in the digital age. Patent attorneys will work with a computer from 8 a.m. till 7 p.m., and the fact that they can test their knowledge and display their skills on a computer better matches that reality than what was done in the past. In the past, they’d come with scissors and glue, cut things out and stick them on the exam paper.
There is also a question that goes beyond 2021, which is, how do we assess the competences of candidates willing to become the next generation of European patent attorneys?
SW: Exactly. It’s not about whether you are good at exams, it’s about whether you can do this job. And that’s something you’ve got to prove over a period of more than three hours.
