A DIGITAL APPROACH TO THE TEF

To ensure quality in teaching and help students with their future employment prospects, a digital approach is key. This is especially true for the assessment procedure in HEIs, as this educational key practice is notoriously underrepresented in digitisation initiatives within education, even though HEIs risk a misalignment in their teaching efforts – and thereby effectively lowering the effect of teaching – because of it.

The third year of the TEF (Teaching Excellence and Student Outcomes Framework) assessment was set in motion this January. According to the Office for Students (OfS), the policy – instituted by the Department of Education in 2016 – is put in place to help students and ensure high-quality teaching: “As well as helping prospective students to choose where to study, the TEF also encourages providers to work with their students to identify, pursue and maintain excellence.”

And while the TEF has inspired a great deal of controversy in higher education in the UK and garnered ill will from some HEIs (Higher Education Institutions) due to fear of damage to their reputation, the government assessment of UK-based universities has continued with broad participation.

It has been debated what the impact of the TEF will be on students choosing their educational provider. While still unclear, it initially seems that international students are being swayed by the gold standards awarded, as noted by The Guardian. Especially bronze-awarded institutions with less pronounced international profiles are at risk of not being chosen by international students.

A Brief Intro: What Is the TEF?

The TEF assessment is initially measured by six metrics, resulting in a rating of either gold, silver or bronze, valid for up to 3 years, while a ’provisional’ rating is available for educational providers with insufficient data. The panel responsible for awarding the ratings is also provided with contextual data and a written submission from the educational provider to further inform the panel in their assessment process.

The focal point of the assessment is the student learning experiences and outcomes provided by the individual educational institutions, which is measured by metrics from different sources.

• The NSS (National Student Survey) provides the first three metrics, based on student perspectives in 3 different categories: Quality of Teaching, Assessment and Feedback and Academic Support.

• The fourth metric is measured by retention rates or ‘non-continuation’ of students and is sourced from data from HESA (Higher Education Statistics Agency) and the ILR (Individualised Learner Record).

• The final two metrics are sourced from the DLHE (Destination of Leavers from Higher Education survey) and regard the employment of graduates and the complexity of their work (whether it is at graduate-level or not)

In the 2017-2018 academic year, the TEF will even include a subject-level assessment as a pilot project.

Student Views on TEF

The ‘student experience’ is an important matter in higher education and not only for the students. The competition among educational institutions is rapidly increasing, making more and more student-oriented criteria within higher education deciding factors when students choose their future HEI.

And while universities have already put a large amount of care into the student experience, the focus might become even greater as the TEF continues. And with good reason.

During the summer of 2017, a consortium of more than 20 student unions came together and funded a research project on student views on the term “teaching excellence”, to be worked out by European research institute Trendence. Almost 9000 students from more than 120 UK-based educational institutions were consulted, making it “the UK’s largest research project to date on students’ views of the TEF metrics”.

While the students seem to disagree with the importance lent to certain metrics of the TEF, placing graduate employment as the least important of seven factors of excellent teaching, 84% of the consulted students indicated a strong support for a government programme to improve and ensure the quality of teaching. So, while the specific metrics and measurements of the TEF continue to stoke controversy, the overall need for an assessment of “teaching excellence and student outcomes” seems to be backed by the student bodies of HEIs in the UK.

Regarding the fact that graduate employment was placed as the least important of the seven factors in the survey, we would like to make a remark. Though we agree that future employment is not necessarily an adequate measure of the quality of teaching, it is nonetheless an important part of education as a whole: equipping students with the appropriate skills for future workplaces is definitely part of the responsibility of HEIs. When the notion of future employment is proposed from this perspective, the student bodies of UK-based HEIs rate its relevance much higher, as shown in the 2017 report “Student Digital Experience Tracker” by JISC. 82% of the consulted learners at HEIs felt “digital skills will be important in the workplace”, but only half of them agreed that their course prepared them for a digital workplace. With this in mind, there is a strong incentive to exercise and enhance the digital skill set of students during their education.

The Call for a Digital Approach to Education

Already in 2015, the report “Make or Break: The UK’s Digital Future“ from the House of Lords’ select committee on digital skills spoke openly on the need to adapt to digital technologies as part of the human condition: “Digital skills — the skills needed to interact with digital technologies — are life skills, necessary for most aspects of life”.

And for many of us, these skills are applied in almost every corner of our daily life. When we wake, it is by the alarm set on our iPhone, not a mechanical alarm clock. While drinking our morning coffee (which was brewed on a coffee maker connected to the internet, so we get notifications on when it needs to be cleaned), we read the news on a tablet or computer. If our car is broken, we use our phone to check the nearest means of public transportation. While sitting in the bus or in the metro, we check our email or our Facebook account.

These are just a fraction of the ways our lives are dependent on digital skills. Digital devices are used everywhere: in health care, in engineering, in media, in design, in manufacturing – it is hard to imagine an industry where digital devices are not used and digitisation is not relevant. And these are the industries the majority of future students of HEIs are headed towards.

According to the JISC report “Student Digital Experience Tracker”, around 90% of all new jobs require good digital skills. 

Providing Quality in Teaching and Assessment 

Expanding the digital capabilities of HEIs is not just a measure to increase employment outcomes for future students. Digital tools are bound to play a larger and larger role in driving the academic imperative. In fact, for most of the processes in education, digitalisation is already present. Teaching situations are very different from just a few years ago.

The materials used, such as encyclopaedias, articles and cases, are increasingly made available online instead of printed out. The formats are also diversified, making audio, video and mobile applications plausible for teaching and learning purposes. The teaching itself is usually conducted with the help of digital aids, such as smartboards or PowerPoint presentations. Even the structure surrounding teaching and learning has been digitised with the use of VLEs (Virtual Learning Environments).

The students’ methods have also changed. Where they used to take notes on paper, they now increasingly use computers or tablets. For many students, the computer is also their primary tool for reading. Because of this, the competencies have changed and moved in a digital direction, meaning students have less eligible handwriting and have become used to writing and spelling aids on their computers. They are also increasingly exposed to and use other aids on the internet, such as subject matter discussion fora, study help, user-driven encyclopaedias, test and practice fora, “How to”-YouTube channels and much more.

Research-based Education Presupposes Digitisation

But by and far, one of the most digitised areas within education is research. For example, social networking and online communities have enabled researchers to overcome the challenge of collaborating over great distances, making multi-institutional research networks which can share knowledge and resources, to sharpen and improve their focus on applying large-scale critical mass research to challenges in the real world.

Documenting research results digitally and hosting research data online has greatly increased the transparency in research by making research findings and the accompanying full datasets more available to both researchers and students.

Technological advances have likewise affected the way research can be conducted, for example by making in-silico research more plausible and accessible to research departments. They can now use computer simulation to assess experiments with enormous quantities of data much faster and cheaper than by physical experimentation – and risk-free. At Columbia University, this has – among other things! – led to the creation of Illustris, a realistic computer simulation of the evolution of the universe.

Examples are abundant. The main point is that research is one of the most digitised areas within education – and with good reason. By aligning the research environment with digital trends and applying digital solutions to real-world difficulties, the research departments of HEIs have greatly increased both their output, their efficiency and their ability to collaborate.

Aligning Teaching and Exams

Providing the best education, both backed by and based on solid research, is of crucial importance to just about any educational provider. Especially within HEIs, where education is most often based on research-based teaching. But when we think about it, something is not altogether agreeable in that sentiment.

While the teaching and learning processes within HEIs are fast becoming digital, the exam and assessment process is often left thoroughly behind. This poses a serious problem, because ‘teaching and learning’ and ‘exams and assessments’ cannot be seen as separate tasks. They are a part of a single process, continually affecting each other.

While educators might wish that their teaching and the learning outcomes set will determine the students´ approach to learn, it is widely acknowledged that when presented with a curriculum, students attune their approach to learn according to what will be assessed. This is not innately harmful; in fact, it is a fully understandable reaction when met with a challenging circumstance, which is what an assessment essentially is. The reaction is simply part of being an efficient student, focused on solving the tasks put forth.

Concerning approaches to learn, we differentiate between a deep approach to learn and surface approach to learn[1]. The difference between them is measured qualitatively in regard to understanding and can be seen as positioned differently in the SOLO taxonomy. Taking a surface approach means aiming to reproduce certain information to meet a set of external demands – e.g. an exam. When taking a surface approach, the students are positioning their approach to learn in the early levels of the taxonomy – what Biggs and Collis call unistructural and multistructural understanding -, and focus primarily on identifying and remembering facts and basic concepts. Taking a deep approach to learn means thinking critically about the recently learned material, understanding the context of the information and creating new meaning from it. In this case, the students approach to learn might begin within the early levels of the taxonomy but will elevate into the higher levels – relational and extended abstract understanding by Biggs’ and Collis’ terms -, focusing on drawing connections, analysing relations and ultimately formulating new ideas.

To support and encourage students in choosing the intended approach to learn, the set learning outcomes must be aligned with the learning practices and processes that foster this specific approach, whether deep or surface. And since an exam and the ensuing assessment is the practical measure of the success of the learning outcomes, these must be aligned to the methods used and practices applied in the learning process. In its essence, securing the alignment of methods and practices in teaching and the ensuing assessment means making sure the students experience comparable circumstances, both when the knowledge is initially comprehended and when it is tested. By not doing so, HEIs are setting unfavourable conditions for the success of both its students and its learning outcomes.

An example:

An educator has planned a history course on the Cold War with the following learning outcome: “Students will obtain knowledge and conceptual tools to account for and analyse the history and key events of the Cold War from its conception in the early 1940s to the collapse of the USSR in 1991. Students will be able to discuss issues and historiographical debates within the subject and relate the nature of the Cold War to recent global political developments and reflect on differences and similarities.”

The educator uses a mix of media, such as historical images and video and audio-recordings from some historical events. The written learning material is both physical books, online encyclopaedias and PDFs of relevant research articles, made available through the institution’s VLE.

The exam? A multiple-choice from a Scantron.

How do you think the students will approach this course if they know this is the exam format? Hardly the deep approach, even though it seems very much intended for the set learning outcome.

According to the 2017 report “Student Digital Experience Tracker” by JISC, 88% of HE learners use personal laptops to access learning.

While both fictitious and hyperbolic, the example nonetheless exemplifies the problems with the lack of alignment in modern education. Because how do we effectively align learning outcomes based on digitally influenced methods and learning strategies with analogue methods in exams? It seems to be an increasingly invalid approach to quality education. When applying specific methods in the teaching of students, the proficiencies the students acquire are related to these methods. So, to test their proficiencies adequately, we should likewise use related methods to assess them.

According to the 2017 report “Student Digital Experience Tracker” by JISC, 70% of HE learners agree that “when digital technology is used on their course they are more independent in their learning and can fit learning into their lives more easily”.

Going back to the example, a digital exam format would enable the educator to create an exam that not only varied the formats to include both essay and multiple choice, but also include the media applied in the teaching situations in the exam. It becomes possible to align the methods the students have used in their initial comprehension of the subject; how they processed the relevant data; the specific way they analysed movies and sound bits; the comparison of photographic evidence to historical accounts; the reflection they performed using different sources and different types of media.

By drawing clearer lines between the methods applied in teaching the material, and the way the students’ comprehension of the material is tested, not only are you able to secure more successful learning outcomes, you are also able to ensure a better transparency in your exams.

Digitisation Is a Necessity in HEIs

There might be many ways to improve the quality of teaching or other means that might vastly improve the traditional assessment model, but the need for digital change and adaption within HEIs is undeniable.

In the Trendence report, 86% of the students indicated that IT should be a deciding factor in the measure of the quality of teaching.

Creating a digital infrastructure around your educational institution’s exams and assessments will most likely improve the assessment conditions and learning outcomes for the students. By making digital exam instruments available to support the methods applied in modern education and tools like Rubrics to structure and formalise the assessment’s alignment with the set learning outcomes, a digital exam and assessment platform can help (re-)build the bridge between learning and exams.

With a digital frame surrounding the testing environment of HEIs, it becomes easier to test the academic proficiencies of students within comparable circumstances and with the same methods with which the proficiencies were initially acquired. With this configuration, exams get to be part of the learning experience for students.

A Quick Summary

To end, here is a short recap of the points we make:

• To ensure higher employment probabilities, HEIs need to equip students with digital skills, and exercise and enhance this digital skillset.

• While most HEIs are well on their way to create a fully digital frame around their institution and educational process, the exam and assessment part of this digitisation process is severely undervalued.

• Ensuring that exams and assessments are also digital will create a better foundation for aligning teaching with learning outcomes and the final exam, ultimately resulting in a higher quality of teaching and learning.

[1]The Role of Metalearning in Study Processes, British Journal of Educational Psychology, Biggs, John B., 55, 3 pp. 185-212, November 1985