Bias in AI at work

Before any of the theory, let's watch it happen live in a tool you probably already use at work.

Pick one of the prompts below and paste it into any commercial image AI (ChatGPT with image mode, Microsoft Copilot Designer, Google Gemini, Adobe Firefly). Three images per prompt is plenty for the pattern to jump out, and it keeps free-tier quotas sensible.

Look at the results. Who's actually in the pictures, and who isn't?

In 2018, Amazon quietly scrapped an internal AI it had spent roughly four years building to score CVs. The tool had taught itself, from a decade of Amazon's own hiring data, to penalise CVs that contained the word 'women's' (as in 'women's chess club captain') and to downgrade graduates of two all-women's colleges. Nobody had told it to be sexist. It had learned the pattern from the humans who came before it, and it was faithfully playing that pattern back.

Amazon caught it and killed the project. Plenty of organisations haven't, and the receipts are showing up in tribunal cases, class-action lawsuits, Information Commissioner's Office guidance and the EU AI Act. Bias in AI stopped being an ethics-panel talking point a while ago. It's a compliance risk, a brand risk, and (this is the one that keeps operations awake at night) an efficiency risk.

Over the next hour we'll take the mechanism apart, look at where the skew hides, work out what it actually costs, and finish with the practical habits any team can start using tomorrow morning.

Every AI system, from a CV screener to an image generator to a meeting-notes summariser, learns from a huge pile of examples that its developers fed it during training. It doesn't understand any of that pile the way you or I do. What it actually does is spot statistical patterns in the pile and play those patterns back whenever we ask it for something new.

So if the training pile happens to be dominated by white male CEOs (because the last thirty years of business news photography largely was), the AI ends up quietly assuming that a CEO looks like a white man. It isn't taking sides, and it isn't making a moral judgement about who ought to be in the boardroom. It's being loyal to the ratio it was shown.

That gives us the first rule of AI bias: skewed data in, skewed answers out. The unfairness isn't a bug or a glitch. It's the AI acting as a very honest mirror of whatever a data team chose to feed it.

Bias doesn't only sneak in through the training data. It hides in three different places, and it helps enormously to know which one you're actually dealing with, because the fix is different in each case.

Before you tap the cards below, have a think: where in the whole pipeline of building and using an AI could unfairness creep in?

The first two hide inside the AI. The third hides in the humans around it. Real workplace cases usually involve at least two of these at once, but naming the dominant flavour is the first step to knowing which team owns the fix.

A little bit of skew in an AI doesn't sound frightening in isolation. Five percent worse for one accent, one shortlisted man too many out of ten. It sounds like a rounding error.

The reason it matters is scale. AI systems get used thousands or millions of times a day. A small skew in one decision becomes a big pattern across a year of decisions. It stops being a rounding error and starts being who gets a job offer, who gets a mortgage, whose insurance premium goes up, whose face unlocks a workplace door.

That's before we get to the bill. Amazon spent roughly four years building the recruiting tool they eventually scrapped. Facial-recognition contracts have been cancelled by councils and police forces after independent accuracy audits. Under the EU AI Act, deploying a high-risk AI system without an adequate risk assessment can attract fines up to 3% of global annual turnover. Under the UK Equality Act 2010, disparate impact is disparate impact whether a person or an algorithm produced it. Regulators, tribunals and journalists have started asking to see the audits.

You really don't need a data science team to catch AI bias in a tool you're using. Four small habits, done properly, will catch most of what you meet in day-to-day work.

Before you tap the cards below, have a think for yourself: if you were teaching a colleague to spot AI bias, what would you tell them to watch out for?

These aren't a checklist you tick once and forget. They're a way of reading anything an AI hands you before you hand the result on to a customer, a candidate or a regulator.

The four habits catch bias after it appears. There is also something you can do BEFORE you hit return: write a better prompt.

AI systems fill in the blanks. If you leave the blanks wide open, they fill them with whatever the training data was loudest about. If you close the blanks, the AI has to try harder, and the results usually get more useful (and less legally worrying).

One rule: name what you want to see, and name what you don't want it to leave out. The two prompts below ask for the same thing. Only one of them steers the AI away from the obvious skew, and only one of them is safe to paste into an operational workflow.

Time to put everything together. You're using an AI shortlisting tool on a batch of 120 applications for a mid-level analyst role. The tool returns its top ten candidates. All ten are men. Nine of the ten graduated from the same three universities. Not a single non-graduate, career-changer or candidate over forty.

You know the applicant pool wasn't like that. Roughly 40% of the CVs came from women, and applicants came in via multiple routes. You've got the four habits AND the good-prompt rule. HR is asking for the shortlist by end of day.

You need a plan, and 'trust the tool' is not it.