The Machine Made Me Do It!

In an earlier piece, I used Detective Danny Reagan’s line from Boston Blue as a practical frame for AI accountability: “The tech is just a tool.” That sentence matters because it cuts through the noise. Tools do not bear responsibility. People do. Organizations do. Professionals do. Institutions do.

This article looks at what happens when that principle is forgotten.

A chatbot gives the wrong answer. A lawyer files citations that do not exist. A facial recognition system flags the wrong customer. A hiring system screens out qualified applicants. A trading system fires millions of orders into the market. A government debt system treats vulnerable people as numbers before it treats them as people. A financial system inside a public institution says workers stole money, even when the workers insist the system is wrong.

The first instinct is familiar.

The machine made the mistake.

That sentence feels modern, but the pattern is old. Human organizations build systems, deploy them, rely on them, defend them, and then point to them once harm appears. The software becomes the actor. The algorithm becomes the explanation. The chatbot becomes the speaker. The automated decision becomes the authority. The human decision environment disappears behind the screen.

The earlier article asks the governance question: who answers for the decision? This article follows the evidence trail. In each case, the machine may have produced the visible error, but the human system around the machine decided whether that error would reach a customer, a court, a worker, a shopper, a market, or a citizen.

That is where accountability begins.

The clearest version begins with Air Canada.

Jake Moffatt used Air Canada’s website after the death of his grandmother. He asked about bereavement fares. The chatbot told him he could book travel and later apply for a reduced bereavement rate within 90 days. He relied on that answer, bought the tickets, then learned from Air Canada that its actual policy did not allow retroactive bereavement fare applications. Air Canada admitted in email that the chatbot had used misleading words, but when the dispute reached the British Columbia Civil Resolution Tribunal, the company argued it could not be held liable for the chatbot’s information. The tribunal described the practical effect of Air Canada’s position: the company was suggesting the chatbot was a separate legal entity responsible for its own actions. The tribunal called that submission remarkable, then stated what should have been obvious. The chatbot was part of Air Canada’s website, and the company was responsible for the information on that website whether it came from a static page or an interactive tool. Air Canada was ordered to pay $812.02.

That case matters because the dollar amount is small, but the storyline is enormous. The customer does not experience the chatbot as a philosophical experiment in machine agency. The customer experiences it as Air Canada. The company places the interface on its site, lets it answer policy questions, then wants the customer to treat the answer as unreliable only after the customer relies on it. The tribunal rejects that split. The machine is not outside the business. The machine is part of the business.

The Air Canada case is the cleanest example of the Danny Reagan principle in business form. The chatbot was the tool. The company was the actor. The tribunal did not treat the chatbot as a decision maker floating outside the airline’s responsibility. It treated the chatbot as part of Air Canada’s own website and held the company responsible for what that tool told a customer.

That is the difference between tool use and responsibility avoidance. A tool can generate the wrong output. A human organization decides whether that output reaches someone, whether it is checked, whether it is corrected, and whether the person harmed by it is told to blame the machine.

The same storyline appears in a different setting when lawyers rely on generative AI.

In Mata v. Avianca, a federal court in New York dealt with legal filings that cited cases that did not exist. The filings contained fabricated judicial decisions that looked enough like real law to pass a casual glance but failed when the court searched for them. One lawyer testified that it never crossed his mind that the cases were bogus. The court still imposed sanctions, required notice to the affected client and judges, and ordered a $5,000 penalty. The court also made the governing point with precision: there is nothing inherently improper about using reliable artificial intelligence for assistance, but existing rules impose a gatekeeping role on attorneys to ensure filing accuracy.

The story is not simply that ChatGPT invented case law. The story is that trained professionals moved the machine’s output into a human institution that depends on verification. The court did not need to pretend the tool had intent. It looked at the human chain: who submitted the filing, who signed the affidavit, who failed to verify the citations, and who continued to represent invented material as law. The AI may have produced the false text, but the lawyers carried it into court.

By 2025, the same pattern had become more severe. In Johnson v. Dunn, a sanctions order from the Northern District of Alabama describes citations that attorneys admitted were hallucinated by ChatGPT. One attorney stated that he used ChatGPT to identify supporting case law, failed to verify the citations through independent review in Westlaw or PACER, and later acknowledged that the citations were inaccurate or did not exist.

That is the professional version of Air Canada. The tool sits inside a workflow, but the human still owns the point of submission. Once the output leaves the screen and enters a court, a client file, a public statement, a hiring process, a financial decision, or a customer interaction, the machine is no longer a private experiment. It becomes part of an institutional act.

Rite Aid shows the same problem in the physical world.

The Federal Trade Commission said Rite Aid used AI based facial recognition in hundreds of stores to identify customers who might be connected to shoplifting or other behavior. The FTC said the system falsely tagged consumers, particularly women and people of color, as shoplifters. The agency said employees acting on false positive alerts followed consumers, searched them, ordered them to leave, called the police, or publicly accused them, sometimes in front of friends or family. Rite Aid was prohibited from using facial recognition technology for surveillance purposes for five years and required to implement safeguards for automated biometric systems.

This example is important because the machine’s output does not stay inside a dashboard. It changes how employees treat people. A false match becomes suspicion. Suspicion becomes confrontation. Confrontation becomes humiliation, removal, or police involvement. The company might want to describe the harm as the result of inaccurate technology, but the human storyline is larger. Someone chose to deploy the system. Someone chose the data. Someone trained or failed to train employees. Someone created the procedures that told workers how to act when the system raised a flag.

The machine made the match. The organization made the match matter.

The hiring context gives the pattern another form. The Equal Employment Opportunity Commission announced in 2023 that iTutorGroup would pay $365,000 and provide other relief to settle a discrimination suit. According to the EEOC, the company programmed application software to automatically reject female applicants age 55 or older and male applicants age 60 or older. More than 200 qualified U.S. based applicants were rejected because of age, according to the agency.

That case reads differently because no chatbot speaks to a customer and no lawyer files invented citations. The machine simply sorts people. Yet the human issue remains the same. The company uses software to make a decision at scale, then the software becomes the first line of exclusion. The applicant may never know a human did not meaningfully review the application. The organization may describe the process as automated screening, but the decision still expresses a human design choice: who counts, who gets filtered, and who never reaches the next step.

Automation makes the mistake look clean because it removes the face of the decision maker. That is precisely why the pattern is dangerous.

Knight Capital shows that this is not only a generative AI issue.

In 2013, the Securities and Exchange Commission charged Knight Capital Americas after an automated trading incident disrupted markets. The SEC said Knight agreed to pay $12 million to settle charges tied to an August 1, 2012 trading incident. The agency said Knight did not have adequate safeguards to limit market access risks, failed to prevent millions of erroneous orders, and failed to conduct adequate reviews of control effectiveness.

The SEC’s account is useful because it strips away the novelty of the current AI conversation. Long before public generative AI tools became common, organizations were already learning that “software glitch” is not the end of the story. Knight’s problem was not merely that code behaved badly. The issue was safeguards, testing, controls, responses to automated messages, written procedures, and supervisory review.

This is where the article’s surface story begins to show its deeper structure. Machine blame usually tries to isolate the visible error. The chatbot gave the wrong fare rule. The AI made up the cases. The facial recognition system created the false match. The hiring system rejected the applicant. The trading router fired the orders.

But the real question is broader. What process allowed the output to act on people before it was checked?

Robodebt extends the same pattern into government.

The Royal Commission into the Robodebt Scheme was presented and tabled on July 7, 2023. Its recommendations focus heavily on the human consequences of automated compliance processes, including the need to design policies and processes around the people they serve, avoid conduct that reinforces stigma, and act with sensitivity to financial and other forms of stress. The report also recommends clear documentation of exclusion criteria, including technical specifications for how people are kept out of a compliance activity when they should be excluded.

Robodebt belongs in this story because it shows what happens when a system becomes a bureaucracy’s shield. The machine does not need to look like a chatbot or speak like a person to become an authority. It can sit inside a compliance process and turn assumptions into demands. Once that happens, people are forced to argue against a system that the institution has already decided to trust.

The machine becomes hard to challenge because the institution has made it official.

The Post Office Horizon scandal is the starkest historical warning. UK government materials state that between 1999 and 2015, hundreds of subpostmasters and subpostmistresses were wrongly convicted after money shortfalls appeared in their branches because of faults with Horizon software. The government described legislation to overturn those wrongful convictions and routes to financial redress.

Horizon shows the most destructive form of machine trust. The institution does not merely blame the system after the fact. It believes the system before it believes people. Workers say the numbers are wrong. The system says money is missing. The institution sides with the system. That is the dangerous inversion. The machine becomes the witness, and the human becomes the suspect.

Taken together, these cases show a common storyline.

First, an organization places a machine between itself and a human being. Second, the machine produces an output that causes harm. Third, the organization points back to the machine as though the machine sits outside the organization’s responsibility. Fourth, the court, regulator, commission, or public body pulls the issue back to the human system that allowed the machine to act.

The pattern is not anti technology. It is anti abdication.

AI and automated systems can help organizations move faster, process more information, and scale services. But speed changes the size of failure. A single wrong policy answer becomes customer harm. A single hallucinated citation becomes a court filing. A false biometric match becomes a public accusation. An application filter becomes age discrimination. A code deployment issue becomes millions of market orders. A public benefits process becomes a national scandal. A faulty accounting system becomes wrongful conviction.

The lesson is not that the machine is never wrong. The lesson is that “the machine was wrong” is only the beginning of the inquiry.

Who put it there?

Who tested it?

Who monitored it?

Who acted on it?

Who ignored the human who said the system was wrong?

That is where responsibility lives.

This is why the Danny Reagan quote carries beyond policing. “The tech is just a tool” is not a slogan against AI. It is a discipline for using AI without surrendering accountability. The problem is not that machines make mistakes. The problem is that organizations sometimes let the machine become the place where responsibility disappears.

Air Canada shows the customer service version. Mata v. Avianca and Johnson v. Dunn show the professional version. Rite Aid shows the retail surveillance version. iTutorGroup shows the hiring version. Knight Capital shows the automated market version. Robodebt and Horizon show the public institution version. The pattern changes setting, but the accountability question stays the same: who allowed the tool’s output to become action?

The machine made me do it may sound like the defense of the future, but the record points in the opposite direction. The machine can produce the output. The human system decides whether that output becomes policy, filing, accusation, rejection, trade, debt, prosecution, or harm. Danny Reagan’s line holds because it restores the proper order: the tech is the tool, not the accountable actor. Someone still has to wield it. Someone still has to check it. Someone still has to answer when it goes wrong.

Bottom Line

The machine can make the mistake, but the machine does not answer for the decision. That remains human work. The organizations that understand this will build review, escalation, documentation, and ownership into every consequential AI or automated system. The organizations that do not will keep learning the same lesson in public, one chatbot, filing, false match, rejected applicant, trading failure, debt notice, or wrongful accusation at a time.

References

British Columbia Civil Resolution Tribunal. (2024, February 14). Moffatt v. Air Canada, 2024 BCCRT 149.
https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html

Federal Trade Commission. (2023, December 19). Rite Aid banned from using AI facial recognition after FTC says retailer deployed technology without reasonable safeguards.
https://www.ftc.gov/news-events/news/press-releases/2023/12/rite-aid-banned-using-ai-facial-recognition-after-ftc-says-retailer-deployed-technology-without

Puglisi, B. C. (2025, October 18). The real AI threat is not the algorithm. It’s that no one answers for the decision. @BasilPuglisi.
https://basilpuglisi.com/the-real-ai-threat-is-not-the-algorithm-its-that-no-one-answers-for-the-decision/

Royal Commission into the Robodebt Scheme. (2023, July 7). Report of the Royal Commission into the Robodebt Scheme.
https://robodebt.royalcommission.gov.au/publications/report

Securities and Exchange Commission. (2013, October 16). SEC charges Knight Capital with violations of market access rule.
https://www.sec.gov/newsroom/press-releases/2013-222

U.S. District Court, Northern District of Alabama. (2025, July 23). Johnson v. Dunn et al., sanctions order.
https://docs.justia.com/cases/federal/district-courts/alabama/alndce/2%3A2021cv01701/179677/204

U.S. District Court, Southern District of New York. (2023, June 22). Mata v. Avianca, Inc., opinion and order on sanctions.
https://law.justia.com/cases/federal/district-courts/new-york/nysdce/1%3A2022cv01461/575368/54/

U.S. Equal Employment Opportunity Commission. (2023, September 11). iTutorGroup to pay $365,000 to settle EEOC discriminatory hiring suit.
https://www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-discriminatory-hiring-suit

UK Government. (2024, May 8). Financial redress factsheet: Post Office Horizon System Offences Bill.
https://www.gov.uk/government/publications/post-office-horizon-system-offences-bill-supporting-documents/financial-redress-factsheet-post-office-horizon-system-offences-bill