HIRING BY MACHINE

This was the case for the small, enthusiastic group of Army veterans who co-founded the non-profit company, Strategeion, after having been honorably discharged during the 2008 recession. Building on their previous experiences supporting various military operations with IT solutions, this group of programmers set out to create jobs for themselves and improve the lives of others by producing an online platform that would enable veterans to stay in touch with their cohorts and share experiences dealing with civilian life. The co-founders did not stop there, however. Having been instilled with a strong sense of civic virtue, and having witnessed first-hand the problems of poverty, joblessness, and homelessness that many American communities were facing during the economic downturn, the developers knew they wanted to use their programming skills to effect broad social change. The group was always looking for interesting new technical problems to address, and vowed to develop services, platforms, and technical solutions for the benefit of all. As the company matured, the platform expanded to include a range of servicesfrom social networking to personal blogging and even a location-based search app that helped individuals moving to new communities discover local points of interestwhich were popular across many demographics. Strategeions unofficial motto became Leave no one behind.

In recognition of its high employee satisfaction and retention rates, Wealth magazine listed Strategeion among its 100 Best Companies to Work For 2013. This resulted in a surge in job inquiries. At one point, Strategeions human resources (HR) team became so overwhelmed with the number of resumes it received that they had to cease hiring to deal with their backlog. HR representatives complained on Strategeions internal message board that they now expended so much energy on the first-round selection process that they no longer had enough time to execute other essential aspects of the job, such as performing background checks and processing new hires. A group of Strategeions developers interpreted the messages from HR as a call for help. In keeping with the companys tradition of developing in-house solutions for internal problems, they offered to create a bespoke resume vetting system to help HR deal with the influx of resumes. Diagnosing the problem as a simple issue of information overload, this group of developers expected it could be easily solved by implementing some clever technical tricks to automatically pre-sort resumes according to a candidates desirability, optimizing especially for projected fit within the company.

After having weighed several options, the team decided to implement a system that utilized natural language processing (NLP) and machine learning (ML) to look for markers in resumes that distinguished the best candidates. They dubbed the system PARiS, in tribute to the Trojan hero who was tasked with judging a contest to determine the most beautiful goddess among the deities of Mount Olympus. PARiS rollout was met with a collective sigh of relief from HR. Because poor matches could be discarded automatically, HR no longer had to devote the overwhelming number of hours required for humans to read each resume the company received. And while some members of the team were initially hesitant about delegating first-stage application sorting to an algorithm, skepticism about PARiS quickly abated as the system revealed its impressive capacity to learn. After only a few weeks in operation, the lists of candidates PARiS suggested consistently reflected those that would have been assembled by human HR agents, instilling confidence that the system had absorbed Strategeions values. But PARiS was so much faster and more efficient than humans! Over time, growing trust in the system meant that the HR representatives felt less and less need to double-check PARiS work, and they began shifting their energies elsewhere.

Hara, a promising and hard-working computer science student from Athens, GA, received an automated rejection email from Strategeion within hours of applying for a job through its website. She was surprised at having been tossed aside so quickly, as she had been convinced she was an ideal candidate for the company. She had strong academic qualifications and she had carefully crafted her resume to reflect her civic commitments and experience working with non-profit organizations that advocated for wheelchair users such as herself. Her ambitions to develop transparent, responsible tech solutions to improve the lives of those with disabilities seemed a perfect match for Strategeions mission to leave no one behind. Disappointed at her rejection, Hara wrote to the company asking for feedback on her application. She also published a blog post about the experience, promising to share any future response from Strategeion. Haras request made its way to the HR department, and the representative who received it was also puzzled by her rejection. After having thoroughly reviewed Haras application, he judged her to be on par with Strategeions very best employees in terms of both interests and credentials. Indeed, based on her resume alone, he expected she would make an excellent addition to the company, and he couldnt see a reason for her application being automatically discarded. He decided to flag Haras case for internal review. At that point, his supervisor decided to use some of the extra time the team had on their hands since the introduction of PARiS to convene a meeting with the systems engineers in order to figure out why the system had rejected Haras application. One potential concern going into the meeting was that PARiS may have used Haras disability status as a reason to deny her application. However, the systems engineers reassured HR that they had explicitly designed the algorithm so that it would not discriminate against protected categories. Furthermore, Strategeions policy of hiring ex-military personnelmany of whom were wheelchair usersmeant that the systems training data was not biased against those with physical disabilities. But if it wasnt her disability, then what was it that PARiS had found in Haras resume that had caused it to categorize her as a bad fit? What was it that the humans couldnt see? After much digging, PARiS engineers found the unlikely answer: sports. It turned out that there was a strong positive correlation between participation in athletics and military service. Given the overrepresentation of veterans among Strategeions employees and their propensity to excel at the company, PARiS had learned to connect a history of playing sports with good fit. And while it was true that many of Strategions ex-military employees no longer participated in sports, their resumes typically reflected a history of having done so. Hara, on the other hand, had never been interested in sports. And, having used a wheelchair her entire life, she also had no history of athletic activities.

In the interest of openness and honesty, the HR representative in charge of Haras case reached out to her with the teams findings. He explained that the company had recently incorporated an AI system into its hiring processes. And while PARiS was generally a success, he admitted that there were still some bugs that would need to be worked out. In Haras case, PARiS had considered her resumes lack of references to physically demanding activities to indicate a weak cultural fit for Strategeion. The HR agent apologized on behalf of the company, invited Hara for an interview, and promised that the company was already searching for solutions to PARiS shortcomings. Hara was dismayed to learn that Strategeion had delegated decision-making in hiringan area that could have a profound impact on her life prospectsto an AI system. Even worse, that system had then wrongly discriminated against her! Frustrated and angry, she published the companys response on her website, where her readers joined in discussing several ethical concerns surrounding PARiS.

Hara ultimately decided to reject Strategeions offer of an interview and, instead, she filed an official complaint with the company, incorporating many of these arguments. Upon receipt of Haras complaint, Strategeions Board of Directors launched an investigation to determine the merits of her claims. Strategeion first needed to address the legal allegations in Haras complaint. The Board handed the accusations of inappropriate data use and discrimination to their in-house counsel, who could ascertain: 1) whether Strategeion had committed a legal wrong by using their employees resumes to train PARiS without their knowledge or explicit consent; and 2) whether PARiS had fallen afoul of US anti-discrimination law. Both accusations were serious, but the lawyers were especially concerned about the latter, given Strategeions conception of itself as a company that provides fair, transparent, honest tech solutions in service of the public good. US anti-discrimination law has evolved significantly over the last half-century. The US Constitution has been interpreted to prohibit discrimination against protected categoriesincluding persons with disabilitiesby federal and state governments against public employees. Private corporations are also subject to a growing body of anti-discrimination law. The Rehabilitation Act of 1973 and the Americans with Disabilities Act (ADA) of 1990 require private employers to treat all prospective job applicants equally. More recently the ADA Amendments Act of 2008 defined equal treatment clearly within the framework of the equal opportunity principle, meaning that persons with disabilities cannot be at a placed at a disadvantage in hiring by virtue of their disabilities. If PARiS had been directed to discriminate against applicants on the basis of their disability status, Strategeion would clearly have violated US law. But that was not the case. PARiS was not intentionally discriminating against resumes based on protected attributes; rather, redundant encodings in Strategeions data had allowed the system to infer such attributes from other, seemingly innocuous data. Thus, Strategeions lawyers believed they could prove the company was legally in the clear.

But even if the lawyers were able to show that Strategeion had not acted illegally, it was not clear that the company had behaved in accordance with its own ethical principles. Over the course of the investigation, it became clear that Strategeion would need to take a long, hard look at itself. Throughout its history, Strategeion had consistently promoted a robust notion of fairness through its positive efforts to recruit employees from a group they believed to be in dire need of help (i.e., veterans). Indeed, many of those individuals had injuries and ailments that would qualify as disabilities. Yet despite the comoanys best efforts to promote fairness in hiring, the Board had no choice but to acknowledge that, in deploying PARiS, Strategeion had failed to live up to its ambitions. Something would need to be done to ensure that all strong applications were given a fair shot. The question was: what? A complete overhaul to the companys hiring policies would be difficult. Strategeion wished to be a positive force in the world, but it also wanted to hire individuals who would be in it for the long haul. Thus, projected cultural fit was an extremely important part of their hiring criteria. In order for PARiS to make a determination on fitness, the systems engineers had decided to train it on samples from past and present Strategeion employees. But while this approach meant that PARiS was adept at picking out resumes of people who most resembled successful Strategeion employees, because of the companys historical hiring practices that favored of military types, it also meant that people who did not fit that mold would be discriminated against. In other words, given the system design, Strategeions biased data would produce biased results and promote biased outcomes. One option to address PARiS bias problem was to implore the systems engineers to infuse more diversity into their models. A second option called for rethinking the value of a homogenous workforce. Recent reports in management studies have shown that more diverse project teams are able to evaluate products and services from a wider range of perspectives, typically resulting in all-around better outputs, as well as more productive workplaces. Upon reading some of this literature, even Strategeions co-founders tentatively agreed that it might be worth considering a change in hiring priorities

Assume that you are an HR representative for Strategeion. After the experience with Hara, you are concerned about PARiS and its continued use in hiring. Come up with a plan to take action and raise your concerns to management. Consider the following questions

• How can you get it done effectively and efficiently?
• What is at stake for the key parties?
• What are the main arguments you are trying to counter? That is, what are the reasons and rationalizations you need to address?
• What levers can you use to influence those who disagree with you?
• What is your most powerful and persuasive response to the reasons and rationalizations you need to address? To whom should the argument be made? When and in what context?