Detecting Age Discrimination in Job Advertisements using GPT and BERT

About Textmetrics
Textmetrics is a company that specializes in text optimization and content creation tools. They provide software that helps businesses improve the quality and effectiveness of their written content, for example job advertisements, by analyzing and offering suggestions for enhancements. Their platform is often used to ensure that content is clear, inclusive, and aligned with brand and SEO strategies.

Task
One of the components of an inclusive job advertisement is avoiding age discrimination. Textmetrics currently detects this with a set of regular expressions. However, these expressions do not match the full variety of sentences, so recall is low. They also struggle with sentences containing a justification. For example, “You need to be at least 18” is discriminating, but it can be justified in a sentence such as “because you need to work late” or “in order to serve liquor”. Our goal was therefore to find a method that significantly improves recall while preserving good precision.

Method
We first explored whether GPT could detect age discrimination with the help of an extensive prompt. In a set of 1,000 job advertisements (22,366 sentences), it found 599 potentially discriminative sentences compared to 71 found by the original regular expressions. After manually removing false positives, 418 discriminative sentences remained, forming a fairly reliable labeled dataset.

On the left, the age discrimination found by the regular expressions. On the right, the age discrimination found by GPT.

We then used this dataset to fine-tune GPT in order to create a much larger labeled dataset. This resulted in 100,000 sentences, of which 2,504 received a positive label. At this point, we divided discrimination into four categories:

• Age discrimination (2,100), consisting of direct phrases like “you are at least ... old”, but mostly indirect phrases such as “we are looking for a young person”.
• Health discrimination (122), consisting of sentences like “you need to be fit” without specifying why.
• Tone of voice (221), where slang is used to attract younger applicants and discourage older ones.
• Justified discrimination (60), sentences that ask the applicant to be of a certain age or be physically fit due to the nature of the job.

After creating this dataset, many iterations followed in which we manually improved the data. We mainly focused on the positive labels, because the number of non-discriminative sentences was too large to review manually. Textmetrics did not have the resources for a large annotation project with many annotators, so using GPT to create a labeled dataset was far more feasible. We compared its performance with the original regular expressions to validate the results. A disadvantage of this technique is that false negatives remain largely unknown, and not all false positives are corrected. In the next step, we aimed to reduce those false positives further.

Results
We trained five BERT models using cross-validation on a representative set of 5,000 non-discriminative sentences and 1,945 discriminative sentences. By examining the wrong predictions from all models, we were able to refine the labeled dataset even more. Below is a confusion matrix of the five combined models:

In the end, we decided to split the workflow into two BERT models: first, a model detecting age discrimination, health discrimination, and tone of voice; second, a model deciding whether the detected discrimination is justified or not. Note that this method works at sentence level, so it does not take surrounding sentences into account. A justification therefore needs to appear in the same sentence. Below are the final confusion matrices:

Discussion
This project shows that fine-tuning a BERT model achieves adequate results for a sentence-level prediction task such as detecting age discrimination. It also means you are not tied to a paid LLM in production. We did, however, use GPT strategically to create a labeled dataset once, instead of investing the time and resources required for a large manual annotation project.