AI, gender and the problem with judging candidates on polish

A polished application can feel reassuring when you’ve got a long list to review. Someone has explained their experience confidently and struck a professional tone. It’s understandable that the writing shapes your impression.

But research reported by HR Dive raises a question worth sitting with: how much of that impression might come from the way an AI tool responds to the candidate’s prompting style? The answer matters if we let the finish on an application stand in for evidence of ability.

What changed when the prompt changed

Johns Hopkins University researchers tested GPT-4, Llama, Gemma and Mistral on workplace correspondence, including job applications. They added language associated with women, such as hedging with “maybe” and “I think”, collective phrasing such as “we”, and expressive adjectives such as “lovely”.

In these tests, those language features led the tools to produce less complex, less formal correspondence than prompts using language associated with men. HR Dive also describes the responses as less sharp and clear. The researchers reported that the pattern persisted even after accounting for changes in the writer’s tone.

These are associations, not rules about how women and men speak. For hiring teams, the risk is that women and others using those features could be marked down if reviewers treat more elaborate, formal writing as evidence of greater capability.

Less complex doesn’t mean less capable

There is an important distinction here. Complexity and formality describe writing; they don’t, on their own, tell us whether it does its job. A concise, straightforward explanation may be exactly what a colleague or customer needs. A longer, more formal one may simply take longer to read.

That doesn’t make presentation irrelevant. An application that’s confusing or inappropriate for its audience can raise a legitimate question about communication. But the question should be tied to the role, then explored with comparable evidence. The reported findings are a reason to review how we score writing, not proof that any individual candidate has been disadvantaged.

Give writing a job-relevant test

Before reviewing applications, ask: what does this piece of writing need to demonstrate for this role? If you’re using it to find relevant experience, keep your notes anchored to the examples, decisions and results described. A fluent paragraph still needs substance; an understated one deserves the same search for evidence.

Where writing is central to the job, don’t abandon assessment. Make it more direct. For example, a customer-support candidate could respond to a fictional complaint about a delayed order, using supplied delivery information and refund rules. You could assess whether they explain the situation accurately, acknowledge the customer’s concern and offer an available next step.

On that basis, a brief, warm reply could outperform an elaborate one that never explains what happens next. Equally, friendliness wouldn’t excuse promising a refund the supplied rules don’t allow. That’s a much more useful distinction than which response sounds more sophisticated.

Give candidates a common brief and clear rules on assistance, chosen to reflect how they would work in the role. If you allow AI, provide comparable tool access; if you’re assessing unaided writing, make that explicit. Consider replacing an existing writing stage rather than simply adding another task.

Before scoring, have reviewers discuss sample responses against those criteria. When someone says a response “sounds more professional”, ask what specifically makes it more effective for this audience. Sometimes there will be a good answer. Sometimes you’ve found a preference that shouldn’t decide who progresses.

Don’t turn a model problem into a candidate flaw

Coaching applicants to sound more direct when using AI might seem like a quick fix. But the researchers describe these language patterns as largely unconscious and difficult to change. That makes prompt coaching an inadequate response on its own.

As lead author Katherine Van Koevering put it: “The companies need to fix the models, rather than putting all of the burden on the user.”

For hiring teams, that leaves a decision we do control: how much weight the wording carries. This isn’t a basis for guessing a candidate’s gender or AI use from their prose. It’s a reason to revisit your next shortlist and ask whether each writing-related judgement reflects something the job requires. If you can’t make that connection, the judgement needs more work before it affects someone’s opportunity.

Original reporting: HR Dive.