What are exams for? On measuring ability and disability

Students sitting an exam. But what are they really being tested on?

Equal rights 

I’m going to go out on a limb here and assert that equality of opportunity is a good thing. There, I’ve said it. Gone are the bad old days when jobs and privileges were determined at birth. No longer do you have to be an aristocrat or wealthy land-owner to study science; Michael Faraday broke that mould. Neither is being born with a Y-chromosome still a prerequisite for academic success. While that playing field may not be as level as it should be, at least officially-sanctioned sexism has been abolished since Rosalind Franklin’s day. Encouragingly, I currently teach a cohort of undergraduate mathematicians, at Leeds University, with a near-equal female:male gender ratio of 52:48.

Belatedly, we have seen improvements in equality of opportunity for people with disabilities. An inspiring leap forward was made by the London 2012 Paralympics in dispelling some of our social prejudices. Meanwhile, with the introduction of the Equality Act 2010, educational establishments have set up new procedures to ensure that disability does not result in inequality.

For a simple and obvious example, consider a physics undergraduate student who uses a wheelchair. Their inability to walk has no bearing on their potential quality as a physicist. So their university has a responsibility to make sure that they are not disadvantaged during their learning and assessment. It would be unfair to arrange their exams to take place at the top of a steep flight of stairs. Their institution needs to be aware of their condition and make sure that they can access the exam.

Similarly, universities must make exams accessible to blind or partially-sighted students by printing their exam papers in Braille or large print. It is obvious that poor eyesight should not prevent a person being a good physicist. So we lecturers and examiners must make sure that our formal assessments of a physicist’s abilities reflect only those abilities relevant to being a physicist, while taking appropriate account of a candidate’s medical conditions.

A student with a disability can visit a university’s Equality and Diversity Unit to have their needs assessed by a qualified professional, who will write a formal Assessment of Needs: a document that is circulated to their teachers, explaining what special provisions are required to prevent the student being disadvantaged by their condition. So a student with hearing difficulty might have an Assessment of Needs containing a statement such as, “Lectures should be arranged in a room with a hearing loop.” It makes sense, and can be very helpful.

Learning equality 

Things become a lot more complicated where a specific learning difficulty (SpLD) is involved because, whereas hearing or walking are not crucial abilities for STEM subjects (Science, Technology, Engineering, Mathematics), learning is a university’s core business. The Equality Service at the University of Leeds has useful information about SpLDs. It says,

“Each SpLD is characterised by an unusual skills profile. This often leads to difficulties with academic tasks, despite having average or above average intelligence or general ability.” This makes a thought-provoking distinction between ability with academic tasks and intelligence. It presupposes particular definitions of “intelligence” and “academic”. I don’t know how to define either of those things, but it seems safe to say that the particular type of intelligence that is relevant to university work could be called “academic intelligence”.

When learning is itself the subject of an Assessment of Needs, as it is for people with Asperger syndrome or dyscalculia, for two examples, then the assessor’s own academic background becomes relevant. Assessors and staff of Equality and Diversity Units often have medical or humanities training. (I confess this is an anecdotal observation, not based on good data.) So their views of STEM-subject exams are not based on experience. Yet they and other medical professionals are required to write Assessments of Needs that carry the weight of law, and dictate some parameters of the teaching and assessment delivered by the subject-specialists.

For instance, while good writing style is deemed relevant to an English degree, physics examiners are often instructed not to mark a particular student’s work on the basis of their grammar. The assumption is that ability to write good English is not part of the discipline of physics, and can be separated from it as easily as the ability to walk or to hear. The instruction assumes that the exam should only test the student’s ability to calculate or recall facts, rather than a holistic ability to understand an English description, translate it into a calculation, solve the problem, interpret the solution and communicate it well. Of course, no examiner would mark a physicist’s work exclusively on their writing style, so we are only talking about a handful of marks at stake.

Of necessity, when the 2010 Equality Act became law, new systems were hastily put in place, without much time for consultation. As a consequence, examiners were never asked, for instance, whether we should expect a physicist to demonstrate good communication ability. As we iron out the system’s early teething troubles, we need to address these kinds of question. What exactly is an exam supposed to test? To what extent can we separate our assessment of a physics student’s linguistic ability from their other skills? This is not a rhetorical question. I don’t know the answer, but I do know that it is complicated and not obvious, and should be debated before the rules are set in stone.

Here is a cartoon that brings the issue into sharp focus.

“If you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid.” Source: Quickmeme

It all hinges on what the selection is for. If this is the exam for a swimming qualification, it is entirely unsuitable. If it is a job interview for a steeplejack, then it’s a fair test that discriminates appropriately between the best and the worst. It is easy to define the appropriate skills for a steeplejack. How should we define a scientist?

A matter of time 

To avoid any misunderstanding, I want to make a clear distinction between teaching and examination. Any good teacher needs to have empathy for their students, and pitch their teaching at a suitable level and tempo for each individual. Being armed with the maximum possible information about the student’s particular needs and abilities is always helpful, and the good teacher will make appropriate provisions whenever possible. The extent to which special provisions should be made during exams is an entirely separate question.

The most common provision in an Assessment of Needs is the stipulation that a particular student should be given extra time (typically 25% extra) to complete their exam. This raises a fundamental question. If a person can solve a particular puzzle more quickly that another person, although both might get there in the end, should their university award them a higher grade? A person’s intellectual ability cannot be quantified on a single, one-dimensional scale. It is many-faceted, and speed of problem-solving is one aspect of it. One might suggest that exams do not exist to test a person’s intellectual ability, but only to test how much they have learnt during a particular course of study. That sounds like a reasonable idea but, in fact, we do not measure a student’s scientific ability at the beginning and end of a science degree course, and award the qualification for self-improvement, irrespective of whether they are any good at the subject. On the contrary, the letters “BSc” are purported to be a standardized benchmark, indicating a particular absolute level of ability. That ability might have been innate when the student arrived at university, or might be the result of more-than-average hard work. The most common method for determining ability in any academic subject is by timed exams. An exam tests what a student can achieve within a finite time interval. At the end of that interval, anyone who has not finished misses out on the marks that they were too slow to accrue. An exception is made if a medical professional has predicted, rightly or wrongly, that the student would need extra time for their exams, and has written their prediction in an Assessment of Needs. This inconsistency presents a problem. A more accurate and individually-tailored assessment of each student’s needs could be made in the exam hall. We could unambiguously identify the students who need extra time, as a result of their own unique abilities and disabilities. They are the ones who run out of time!

So there would be advantages (as well as massive logistical difficulties) in having un-timed exams. They would allow each student to demonstrate their abilities, whilst removing the element of speed from the assessment of their expertise. With the best intentions, we have stumbled into the new age of equality with a flawed mixture of two systems. Candidates, whose needs have not been assessed, have their abilities measured by a fixed-duration exam, while others have the duration of their exam determined by their abilities.

The question that must urgently be addressed is this. Do we want exams to test what a candidate is able to achieve within a fixed time, or do we only want to know what they can achieve when given as much time as they require? Creating fair and meaningful methods of assessment requires an open debate on what we want from an exam, what we want from a degree classification, and what we want from a physicist.

Mike Evans

Mike Evans

Dr Mike Evans is a lecturer in Applied Mathematics at Leeds University. His research is on non-equilibrium statistical mechanics, soft matter and complex systems
Mike Evans

Latest posts by Mike Evans (see all)


Comment via Facebook

Comment via Disqus


Comment via Google+