The integration questions universities should ask before they shortlist an exam vendor

Integration has moved to the front of exam platform evaluations. The questions we all bring to it were written for a later moment in the process — and what institutions talk to us about once an integration is actually running is not quite what the standards conversation prepares you for.

Integration used to be the last thing before signature

It used to arrive last. A platform evaluation would open with pedagogy and exam formats, move through security and pricing, and reach integration somewhere near the end — the loose ends to tie off once the shortlist was down to two and the contract was already being drafted. That ordering made a kind of sense when an exam platform sat next to two or three other systems.

It makes much less sense now. The number of systems an exam platform has to work side by side with has multiplied: a national or institutional student records system, a learning management system, an identity federation, document storage, a plagiarism or originality service, a proctoring component, and increasingly an in-house student lifecycle tool that nobody outside the institution has ever heard of. None of them contains the others. They form an ecosystem, and an exam platform is one part of it rather than a layer underneath or on top. Add the fact that more institutions are genuinely re-evaluating incumbent platforms rather than renewing on autopilot, and the evaluation question changes shape. It stops being “does this do exams well” and becomes “does this fit with the twelve systems we already run”.

So integration has moved to the front. We think that is the right instinct. The questions themselves have mostly stayed as they were, though — any of us would reach for them, and they were written for the pre-signature conversation, where the job was to confirm a decision that had effectively been taken. Asked at the start instead, they can carry a good deal more weight. And there are a few worth adding to them.

A standard is a floor, not a guarantee

The obvious upgrade to an integration checklist is to ask about standards, and it is a real upgrade. LTI 1.3 is the current version of 1EdTech’s Learning Tools Interoperability specification, built on OAuth2 and JSON Web Tokens, and LTI Advantage layers three services on top of it: Assignment and Grade Services, which creates gradebook columns and posts grades back; Names and Role Provisioning Services, which supplies users and roles; and Deep Linking, which lets someone pick specific content from the tool and link straight to it. QTI 3.0, finalised in May 2022, does the equivalent job for assessment content, enabling what 1EdTech describes as “the exchange of item and test content and results data” between the systems that author, store, deliver and score assessments.

There is even a standard built for the exchange this article keeps circling: OneRoster, which moves rosters, enrolments and grades between a student records system and the systems that teach and assess. It rarely comes up, reasonably enough — its centre of gravity has been the school sector, and the records systems European universities run are largely national infrastructure that predates these specifications and exposes its own interfaces instead. A yes on OneRoster can be perfectly honest and still not help you, because the system at the other end does not speak it.

Asking for those by name is better than asking whether a platform “integrates”. But it is worth being precise about what a standard actually buys you, because the standards body itself is careful here. 1EdTech describes LTI as “a technical standard (not a product)”, and on QTI it encourages procuring organisations to require conformance certification — which is a polite way of saying that the specification on its own guarantees nothing. Conformance is tested and certified separately, and products appear in a public directory only once they have passed.

There is a second, less discussed caveat. A standard is not uniformly finished. Parts of QTI 3.0 are still maturing: certification for computer adaptive testing is listed as coming, and work on the results reporting and usage data best practices continues. So “we support QTI” and “your item bank will move into our platform and back out again with its scoring intact” are different claims, and only one of them is verifiable today. The checklist question is therefore not “do you support LTI and QTI” but “which version, certified against which suite, and which parts of our content and results survive the round trip”.

Once you are live, the conversation stops being about standards

This is the part that only becomes visible on the other side of signature, which is why it is worth passing on.

We run integrations with institutions across Europe — with learning management systems such as Canvas, Moodle, Blackboard and Brightspace, with national and institutional student records systems, with identity federations, and with a long tail of in-house tools, among many others. During an evaluation, those conversations are about whether integration is possible at all. Once a platform is live, they are never about that again. They become far more specific, and they are almost never about a standard.

They are about whether a candidate number arrived from the records system this morning. Whether a grade and an appeal status can be written back to it. Whether the integration can work with the institution’s own identifiers rather than looking them up before every write. Whether anyone in the exam office can see that a transfer completed.

Institutions that have been through an implementation tend to compress the first of those into a single demand, and it is a good one to put to any vendor:

Show us how candidate numbers reach the platform from our records system — and what happens to grade return and to appeals on the day they do not arrive.

That is an entirely solvable piece of work. It is, in fact, most of what a good implementation consists of, and we have built that link with institutions against a wide range of records systems. The point is that it costs very little to specify before signature and a great deal to discover during a results week.

None of that is a standards problem. It is a this-system-must-work problem, and it is the one an institution lives with for the next five years. Standards still matter, for a reason one step removed from daily operations: they are how you avoid commissioning a custom build, and how you keep the option of leaving. They are the floor. What sits on top of the floor is what you will actually experience — and the questions that get at it are hard to ask in a demo and easy to ask of a reference customer.

“Do you have an API” is not the most useful question

Every vendor answers yes, ourselves included, and the answer is perfectly true. It just tells you less than it appears to, because the variable that matters is not whether an API exists but how much of the platform it reaches.

This is, by some distance, the subject that dominates integration conversations once an institution is up and running. Not whether there is an API — that was settled before signature — but how far into the product it goes. Can credit weighting, exam form, assessment periods, cover sheets, individual timings and extensions, room bookings and administrator assignments all be set programmatically? Or does somebody click through the interface for each exam in a sitting of several thousand, which is the moment an institution discovers that “we have an API” and “we can automate our exam set-up” were different promises?

The second thing these conversations surface is what the API costs to use at scale: bulk endpoints, pagination limits, whether writes accept the institution’s own identifiers, and whether a response returns enough to avoid a second call for every record. The sharpest version of that question we have heard an integration team ask is worth borrowing:

Can your API accept our identifiers on writes? If it cannot, we resolve ours to yours before every write, and double our request volume to do it.

None of this shows up in a feature comparison, and it is not a question any vendor can fail outright — we know of no platform in this market that exposes every operation through its API, and the boundary moves with every release, ours included. That is precisely why the question is worth asking proportionally rather than as a yes or no. What share of what an exam officer can do in the interface can be done through the API? Which operations are read-only? What are the rate limits at enrolment peak, and are there bulk endpoints for the things you will run ten thousand times in a week? And is there a test environment you can build against before go-live, rather than discovering the answers in production?

The failure checklists tend to miss is the silent one

If we had to name the failure mode that causes institutions the most trouble in live operation, it would not be the one checklists are built around. Checklists worry about upgrades — a platform release or an LMS update breaking a connection. In our experience that happens rarely, and when it does it announces itself loudly enough that somebody fixes it the same day.

The one that actually hurts is the synchronisation that quietly does not happen. A status that sits at “sync requested” with no failure state behind it. Nothing that tells the exam office the transfer did not complete. Candidate numbers that never arrived, discovered at the point grades are due rather than at the point the sync ran. The requirement an experienced exam office writes down is therefore about visibility rather than reliability:

When a synchronisation fails, something has to tell the exam office — clearly, and at the moment it fails, not when grades are due.

An integration that works is not the same as an integration you can see working. The first is a go-live condition; the second is what determines whether a problem surfaces in week one of an exam period or on results day. We would put it high on the checklist precisely because it so rarely comes up in a demo: when a sync fails, who finds out, how, and how quickly? And a related one, since it has caught institutions out during appeals: how long are integration and system logs retained, and does that window outlast your appeals deadline?

Three questions that look technical and are not

Three more belong at the front. The first is identity. “We support SSO” is the generic answer, and the generic answer usually conceals a custom integration project. Ask a vendor to name the protocol or the federation — OIDC, SAML, a specific national academic identity federation — and to say whether supporting yours is a configuration or a piece of work. The difference between those two answers is measured in months.

The second is where the data goes. Every integration is a data flow, and in European higher education each one eventually has to appear in a record of processing, a data protection impact assessment or a sub-processor list. Ask which sub-processors an integration introduces, where the data is held, and whether the vendor can put that in writing before the tender closes. Assembling it afterwards is slow, and it is among the likeliest things to stall a procurement that was otherwise ready to sign.

The third is ownership after go-live. An integration that functions on day one but needs a named person inside your IT team to keep functioning after every platform patch and identity provider change is not really integrated; it is a second system you now maintain. When the LMS or the identity provider pushes an update, whose ticket is it? This is the same argument we have made about implementation generally — that onboarding is a change process, not an installation — and it applies with particular force to the seams between systems, because those are the places where nobody’s job description quite reaches.

Consistency is not the same as buying everything from one supplier

It would be convenient for us to conclude that the answer to all of this is a single platform. We have argued something close to that before, and we still think one well-governed exam platform beats a patchwork of tools for most institutions, most of the time. But that is a judgement about the common case, not a rule. An institution that deliberately runs best-of-breed components can be entirely coherent, provided the seams are instrumented and the record can be reassembled — which is the argument we made when we said that interoperability is a governance question, not an IT one. Several of the institutions we work with run their LMS as the front door and attach assessment to it, which is exactly why we built Marking Studio as an LTI tool inside the LMS rather than insisting on one door in.

What is not defensible is arriving at either architecture by accident, because the integration questions were left until the paperwork stage, by which point the answers had stopped changing anything.

A simple takeaway

None of this replaces the standards questions. Those are worth asking, most evaluation templates already carry them, and they establish that a platform can connect at all. The seven below take it a step deeper: they are the ones that tell you what the connection will be like to live with once it is running, and they tend to settle far more than they cost to ask. Put them before any feature demo, and write them down before the tender rather than during it.

• Which standards, at which version, certified against which conformance suite — and which parts of our content and results survive a round trip out and back?
• What share of what an exam officer can do in the interface can be done through the API, which operations are read-only, and is there a test environment to build against?
• What are the rate limits and bulk endpoints at enrolment peak, and will the integration need a lookup call before every write?
• Can the platform read and write to our student records system without manual exports, and has the vendor done it for an institution on the same system — not merely a similar one?
• When a synchronisation fails, who finds out, how, and how quickly? And how long are the logs kept relative to our appeals deadline?
• Which sub-processors does each integration introduce, where is the data held, and can we have that in writing before the tender closes?
• Which identity protocol or federation, by name — and once it is live, whose ticket is it when the LMS or identity provider pushes an update?

An institution that can get all seven answered in one meeting has learned more about a platform than a feature matrix will ever tell it. An institution that cannot is not necessarily looking at the wrong vendor — but it has found the conversation worth having before signature rather than after.