Counterfactual Fairness Testing as an Auditing Mechanism for Algorithmic Hiring Systems

Abstract

Algorithmic hiring systems increasingly shape recruitment decisions through automated screening, assessment filtering, interview ranking, and final shortlisting. Although these systems are often adopted to improve efficiency and decision consistency, they can reproduce discriminatory patterns through historical labels, threshold effects, and proxy-sensitive variables. This study proposes counterfactual fairness testing as an auditing mechanism for evaluating whether hiring outcomes remain stable when protected attributes are altered while qualification-related attributes remain constant. The audit was designed across five methodological stages: dataset validation, counterfactual profile construction, stage-level instability measurement, group-level disparity analysis, and governance risk classification. From 5,200 initial applicant records, 4,180 records were retained after preprocessing, representing an 80.38% audit-ready retention rate. Results showed that counterfactual instability increased across the hiring pipeline, from 7.8% in resume screening to 12.4% in assessment filtering, 16.9% in interview ranking, and 21.6% in final shortlisting. Mean score deviation followed the same pattern, rising from 2.4 to 7.1 points. Group-level testing revealed the highest instability for disability status at 19.8%, followed by ethnicity at 17.4%, age at 14.9%, and gender at 11.6%. Proxy sensitivity analysis identified institution type as the strongest indirect risk factor with a peak score of 0.23, followed by residential location at 0.19 and career gap at 0.16. The final shortlisting module produced the highest composite governance risk score at 0.257, indicating the need for deployment restriction and model revision. These findings demonstrate that counterfactual fairness testing can convert technical fairness diagnosis into actionable AI governance evidence for algorithmic hiring systems.