AI-written applications: what are we actually scoring?

A clear application is a welcome thing. You can find the relevant experience, follow the examples and understand why someone wants the job, without having to piece the story together yourself. When you’re deciding who to interview, that helps.

But clear and impressive aren’t quite the same. If a beautifully turned paragraph earns a candidate extra credit, what exactly have they earned it for?

New Johns Hopkins research, reported by HR Dive, adds a wrinkle. The researchers found that language associated with women led chatbots to produce less complex, less formal drafts. When a candidate uses one of these tools, some of the polish on the page may reflect how they asked for help.

The prompt has a say in the finished draft

The language involved was ordinary enough: softening a request with “maybe” or “I think”, referring to “we”, or describing something as “lovely”. The team added features like these to prompts for workplace correspondence, including emails, job applications and resignation letters, and tested them across GPT-4, Llama, Gemma and Mistral.

Prompts with woman-associated language produced less complex, less formal writing, while man-associated language produced longer, more complex and more formal responses. The researchers reported that the difference persisted after accounting for the writer’s tone.

One published example makes the change easy to picture. A direct request for a reply to a thank-you email produced a formal acknowledgement. A warmer, more tentative request produced an enthusiastic response about praise that had “warmed our hearts”.

There’s quite a difference between those voices. But would the more formal reply always be the better choice? That depends on the message it needs to convey and the person receiving it. Complexity alone won’t answer that question.

What does “professional” mean on your scorecard?

A criterion such as “professional communication” sounds reasonable. Of course you want someone who can communicate well. The difficulty is agreeing what that means when an application lands in front of a reviewer.

One reviewer might look for a clear explanation of a difficult decision. Another might favour the elaborate language they associate with seniority. Both could tick the same box while rewarding different things. Add an AI draft, and there’s another possibility to consider: some of the style attracting those marks came from the model’s response to the prompt.

The coverage reports differences in generated text, rather than hiring outcomes, and candidates can edit what a chatbot gives them. The screening concern arises where reviewers reward the formality that survives into the application. A difference introduced by the tool could then influence the score.

So “sounds professional” needs a little more work before it belongs on a scorecard. What, specifically, should the candidate’s writing help someone understand or do?

Start with what the reader needs

Think about someone replying to a customer complaint. You want them to explain the decision accurately, recognise the frustration and make the next step clear. A policy briefing asks something different of the writer: make the meaning precise and help a busy colleague find what applies to them.

Both involve judgement. Neither is well assessed by simply counting impressive words or favouring the most formal response.

For the customer role, a criterion such as “explains the decision accurately and makes the next step clear” gives reviewers something to look for. They can point to the explanation and the proposed action, then discuss whether those meet the brief. “Polished” leaves much more to individual taste.

Writing can matter even when it isn’t the main job. Keep criteria that reflect the work, and remove preferences you can’t connect to it. When reviewing experience, ask what the candidate did, what happened and how they know. If they describe a team achievement, follow up on their contribution. Using “we” should neither cost them marks nor settle the question of what they personally delivered.

Could candidates learn to prompt their way around the problem? The researchers describe these language patterns as difficult to notice and change, and lead author Katherine Van Koevering puts responsibility on model developers. Hiring teams have a more immediate choice: make the basis for awarding marks clearer.

Give writing skills a proper test

Where writing matters, give candidates a task that reflects it. A customer support applicant could answer a refund request; a policy candidate could prepare a short briefing. Use the same brief, agree the scoring criteria and explain whether AI assistance is allowed.

If it is, deciding what to keep, change or cut can be part of the skill. Removing three impressive paragraphs because the reader needs one clear answer may be very good judgement. Ask candidates to explain their edits so you can explore the decisions behind the finished text.

Then have reviewers score some practice responses and compare their reasoning. If someone says an answer “sounds more professional”, ask what makes it more useful. Is the decision easier to understand? Can the reader tell what to do next? Those questions give the team something concrete to discuss.

Before the next hiring round, take one vague writing criterion and work through it together. Keep going until reviewers can explain what earned the mark, and why it matters for the job.

Original reporting: HR Dive. Additional source: Johns Hopkins University.