Human-in-the-loop Is Not Enough

What Sainsbury’s facial-recognition failure tells us about designing technology around people

An innocent customer recently walked into Sainsbury’s to buy supplies for an event. He scanned his shopping and Nectar card when two managers approached him and told him he could not be served because of an earlier incident. He protested, but was asked to leave. As he left, he saw his face on an overhead CCTV monitor, circled in red. He had been wrongly identified.

Sainsbury’s later apologised and paused its Facewatch facial-recognition system at the East Dulwich store while colleagues received further training. Sainsbury’s and Facewatch both said the technology itself had not made the mistake. People in the store mishandled the alert. That distinction matters technically, although I am not sure it matters much to the customer.

He did not experience an AI error followed by a colleague’s error. He experienced Sainsbury’s getting it wrong, and that, for me, is the real story.

Human-in-the-loop sounds more reassuring than it is

Retail theft and aggression towards colleagues are serious problems. Sainsbury’s has a legitimate reason to explore technology that helps identify repeat offenders. Therefore, the easy conclusion that facial recognition is simply bad technology is not particularly useful.

The better question is what the colleague is expected to do when the technology raises an alert. Do they act on it, challenge it, or do something else? What evidence can they see? What behaviour should prompt them to pause? What discretion do they have?

“Human-in-the-loop” sounds reassuring, but a human without a clearly defined role is not much of a safeguard. The human may be in the loop, but human judgement may not be.

I have seen this before with self-checkout

A few years ago, I worked with a DIY retailer introducing self-checkout. The business case looked compelling. Customers would scan their own shopping, and the retailer would reduce colleague hours at the checkout.

But the original model underestimated the exception rate. The problem was particularly evident in garden centres. Self-checkout systems use weight tolerances to help verify that the item scanned matches the item placed in the bagging area. That works reasonably well when product weights are predictable, but potted plants are not. A plant watered in the morning can weigh far more than the same plant later in the day. The system did not understand horticulture; it understood tolerances.

The business case assumed that roughly one in six items would require colleague intervention. In parts of the garden centre, the rate was closer to one in three. The technology had not failed. The assumption about it had. And once the exception rate doubled, the labour saving looked very different.

A technology business case is only as good as its understanding of the exceptions.

If every exception requires human intervention, the human role is part of the technology design.

The bank that created “talking application forms”

I encountered a different version of the same problem while working with an Irish bank. The bank had invested heavily in tablets for customer consultations. The adviser and customer could sit side by side on a sofa rather than face each other across a desk with a computer screen between them. The intention was to foster a more natural, collaborative conversation.

A great deal of work had gone into security and application design. Far less attention had gone into how the colleague should use it. When we observed the conversations, advisers simply read each question from the tablet, recorded the answer, then moved on to the next. We started calling them “talking application forms”.

The customer might as well have completed the form themselves. The technology had been carefully designed, but the interaction had not. We helped colleagues turn the process back into a conversation, capturing the required information naturally rather than letting the tablet dictate the exchange. The technology returned to its proper role, supporting the colleague rather than controlling the conversation.

Technology should enhance human judgement, not reduce humans to an interface.

Where do the saved hours go?

Organisations should ask another question when technology is justified through productivity savings. Where does the saved time go? There is nothing wrong with using technology to remove routine work. But if every saved hour disappears from the labour budget, the promise that technology will “free colleagues to serve customers” soon becomes hollow. If self-checkout reduces cashier hours, are enough colleagues still available when customers need help? If AI handles routine enquiries, are the remaining advisers better equipped to handle more complex cases? If facial recognition identifies a potential offender, does the manager have sufficient time, information and training to make a considered judgement?

The leadership question should not simply be: How many hours will this save? It should be:

What better outcome for customers or colleagues will those saved hours enable us to create?

Three things need to be designed together

The more I see technology introduced into customer journeys, the clearer this becomes. Three things need to be designed as a whole.

Technology design

What can the system do reliably? Where will it fail? What decisions should it never make on its own?

Human operating design

What is the colleague there to do? What context should they bring? What discretion and confidence do they need?

Recovery design

What happens when either the technology or the colleague gets it wrong? Who can override the decision? How quickly can the problem be corrected? How will trust be restored?

Recovery is often the missing element. Organisations usually test whether the system works and may train people to operate it. But far fewer appear to design for what happens when the combined human-and-technology system fails. Yet no technology is perfect, and the customer experience depends heavily on what happens next.

Customers experience one organisation

This is where internal distinctions become dangerous. The technology team may say the system worked correctly, while Operations may say a colleague misinterpreted the alert. The supplier may say the issue was outside its control, and Training may say the process was not followed. All of those things may be true, but none of them changes the customer experience.

Customers do not separate the algorithm from the adviser, the self-checkout from the colleague who fails to arrive, or the tablet from the conversation it generates. They experience the organisation. The technology may issue the alert, and the colleague may make the decision, but the organisation owns the outcome.

My final thought

The lesson from Sainsbury’s is not that retailers should stop using facial recognition, nor is it that humans should always overrule technology. It is that the intersection of the two needs far more deliberate design. For too long, organisations have invested most of their resources in making technology work. They now need to put equal effort into defining how people work with it.

What should the colleague notice? What should they question? What can they override? And what happens when the system gets it wrong?

Human-in-the-loop is not enough.

The human needs a clearly defined role in the loop, and the organisation needs a clearly defined recovery plan when the combined system fails. Customers will not remember whether the mistake came from the algorithm, the colleague or the process. They will simply remember that the organisation got it wrong.

RetailCX customer journey design

Technology design + Human operating design + Recovery design = the complete customer journey

This is part of the RetailCX Blueprint – a growing collection of practical ideas, frameworks and real-world examples to help organisations create better customer and colleague experiences. 

At RetailCX, we specialise in helping organisations harness the power of leadership and employee engagement to enhance customer experiences. Contact us to learn how we can support your journey toward a more innovative and customer-centric future.