'Judges are being asked to do the impossible’: research into bias in dressage

Dressage judges are being asked to do the “impossible” – as research on dressage judging bias found they are predisposed to give higher marks to those who share their nationality and have previously performed well.

The study by Inga Wolframm, professor of sustainable equestrianism at Van Hall Larenstein University of Applied Sciences, the Netherlands, considered whether the volume of complex information a judge has to evaluate could exceed what can reasonably be expected, and so if the current system of judging predisposes them to using biases, or cognitive shortcuts, that inadvertently favour certain combinations.

Dr Wolframm analysed more than 500 scores from seven CDI5*s between May 2022 and April 2023. Her results show judges are predisposed to give higher scores to riders who share their nationality, and judges gave higher scores to combinations who had previously performed well, as indicated by FEI ranking and starting order.

“Taken together, these biases form what may be referred to as a bias cascade, which is likely to lead to inadvertent, yet nonetheless unfair, advantages for certain riders,” Dr Wolframm wrote.

She told H&H she is not blaming individuals; judges are not at fault “unless you fault them for being human”.

“The reason these biases exist is because our brains cannot compute all the information they are being confronted with,” she said.

“Based on all the requirements, and criteria, in the FEI judging handbook, judges are, in theory, required to consider up to seven pages of descriptors per movement. But it’s humanly impossible to assess all of those in seconds, especially as judges cannot press the pause button.”

“In practice, this means judges must be able to weigh up the relative importance of each of the different criteria prior to deciding on the score.

“Does the criterion of impulsion, for example, weigh more or less heavily than the criterion of contact? Does this – or should this – change per movement and perhaps even from one horse to the next?”

Dr Wolframm said this means that when assessing horses, we must be clear on what we want to see.

“I think technology can definitely help, but to skip where we decide on what we find important and jump straight into artificial intelligence (AI) would be premature,” she said.

“AI is only as good as the information you put in, and we’re not unequivocal on what we consider a good performance from an evidence-based point of view, so how can we possibly teach AI to assess effectively?”

One technological solution Dr Wolframm considers important is eye-tracking, recording the movements of a person’s eyes to understand where they focus their attention, as this shows what judges are looking at.

“We’ve found that when you’re looking at a horse when it’s moving, you can’t possibly take it all in, so instead you focus on different aspects of the horse,” she said.

“What becomes apparent is judges have their own visual search patterns, which appear to be very specific to the particular movement. This gives us useful insight into what areas are delivering judges the most important information, and what value labels they are attaching to that.

“By doing this we not only create more awareness of what our most experienced judges consider important, we also create guidelines for future judges.

Dressage judging bias: “It would be sad if the computers took over”

“It can sound more tedious than saying ‘Let’s let AI fix it’, but this is how evidence-based research works. We need to get insight into what is actually happening and visualise what has until now been subconscious.

“Information like that will help us come together as a sector to develop more transparent and objective judging systems to safeguard principles of fair and equitable sporting standards as well as the welfare of the horse.”

FEI five-star judge Isobel Wessels, who will be on the judging panel for the 2024 Olympics, told H&H it takes about 15 years to become an international dressage judge.

“We’re constantly training to improve ourselves and develop our skills, it’s very much a marathon, not a sprint,” she said.

“There’s always the human element, you can sneeze or blink and miss something, but at the big championships, we have the benefit of the judges’ supervisory panel who can adjust marks where something like that has happened.

“Of course, I’m not saying it’s easy, but I try to assess every single thing as a clean slate to the best of my ability, not favouring the famous or being too strict on those I don’t know.

“I agree that alternatives like AI will not be doing judges or riders any good. Any changes have to come from what we’ve learned and developed over the years – from within us.

“You have this passion, riders are passionate, breeders are passionate, the horse brings this out in us, and it would be sad if computers take over.

“The only thing we can do is take on board advice from people like Inga and further research what can be done to make things fairer and easier but always with the beating heart of the judge, horse and rider in mind.”

You might also be interested in:

Horse & Hound magazine, out every Thursday, is packed with all the latest news and reports, as well as interviews, specials, nostalgia, vet and training advice. Find how you can enjoy the magazine delivered to your door every week, plus options to upgrade your subscription to access our online service that brings you breaking news and reports as well as other benefits.