The next generation of enterprise AI contracts must answer a more demanding question than who pays when something goes wrong, posits Rohan Sharma, AI strategist and senior executive. They’ll have to answer this: Who can prove what happened?
When a job applicant challenges an AI-influenced rejection, an employer may discover that its contract provides audit rights, security commitments and indemnification but not the evidence needed to explain what happened.
The continuing litigation against Workday, the HR, finance and IT platform, illustrates this problem. The plaintiffs in Mobley v. Workday say that Workday’s algorithmic applicant-screening tools discriminated based on protected characteristics. Workday disputes the allegations, and the court has not found that discrimination occurred.
In June, a judge allowed several California Fair Employment and Housing Act and Americans with Disabilities Act claims to proceed while dismissing other allegations. The ruling addressed the sufficiency of the pleadings, not the ultimate merits of the case.
For compliance leaders, the most important development might be a May discovery order that revealed how evidence can be fragmented across an AI vendor, its customers and their lawyers. The court held that certain Workday bias-testing data was protected by attorney-client privilege because Workday’s lawyers curated the underlying data and used the results to provide legal advice. It also declined to compel Workday to produce customer applicant data because the plaintiffs had not established that Workday legally controlled that data under federal discovery rules.
At the same time, the court ordered Workday to produce certain EEO-1 and Office of Federal Contract Compliance Programs documents because they were relevant to Workday’s potential knowledge of demographic disparities.
The order does not establish that Workday failed to maintain legally required records. It does expose a broader enterprise risk: The records needed to evaluate an AI-influenced decision may be distributed among parties that do not share the same retention obligations, access rights or litigation strategy.
A bias audit cannot reconstruct a decision
Organizations frequently use “auditability” as if it describes a single control. It does not. A bias audit generally evaluates outcomes across a population. It may compare selection rates among demographic groups, examine error rates or test whether a system generates statistically significant disparities. That evidence can help identify systemic risk, but it may not explain what happened to one applicant.
Decision reconstruction asks a different set of questions:
- Which model or rules engine was deployed?
- Which customer configuration and screening thresholds applied?
- What applicant data entered the system?
- What features, criteria or derived variables influenced the result?
- What output, score, ranking or recommendation did the system generate?
- Was the system changed between validation and deployment?
- Did a person review the result, and what authority did that person have?
- What final action occurred and when?
A vendor could pass a population-level assessment yet remain unable to recreate a particular decision. Conversely, an employer might retain the application and final disposition but lack access to the vendor’s model version, configuration history or validation evidence.
A New York City law illustrates the difference. It requires employers using covered automated employment decision tools to obtain an independent bias audit, publish a summary and notify affected candidates. It also requires disclosure of information about the data collected and the employer’s data-retention policy. Those controls improve transparency, but they do not automatically allocate every technical record needed for individual decision reconstruction.
California’s rules go further on retention. The state’s automated-decision regulations require employers and other covered entities to retain employment records, including automated-decision-system data, for at least four years. Providers of automated-decision systems must also retain relevant records for at least four years after the system was last used by the employer or covered entity.
Retention, however, is not the same as production. A record can exist without the employer having a contractual right to obtain it promptly, interpret it or provide it during an investigation.
10 Questions Every Organization Should Ask a Potential AI Vendor
Adopting AI without understanding how it was built and how it handles data can expose an organization to risks that surface only once something goes wrong
Read moreDetailsAdd an AI evidence schedule to the contract
An enterprise buying an AI hiring product should attach an evidence schedule to the vendor agreement. The schedule should identify which party creates, controls, retains and produces each category of record.
At minimum, the schedule should address seven areas.
- System identity and versioning. The vendor should identify the model, rules engine or system version deployed for the customer. The agreement should establish how updates, retraining, configuration changes and material feature changes are recorded and communicated.
- Decision-event records. For each consequential screening event, the parties should determine whether they can connect the applicant, job requisition, system version, customer configuration, relevant input, output, timestamp and final employment action. This does not necessarily require exposing proprietary source code. It requires enough evidence to establish which system operated, under which conditions and with what result.
- Validation and impact-assessment evidence. The contract should specify which validation materials the employer receives, which testing the employer may conduct and what happens when subgroup data are insufficient. It should distinguish routine compliance testing from analyses conducted for legal advice because those categories may receive different treatment in litigation.
- Data control and legal holds. The agreement should not merely state whether the employer or vendor “owns” applicant data. It should address possession, legal control, access, exportability, subpoena response, preservation notices and the effect of customer objections. The May discovery order in Mobley v. Workday demonstrates why this matters. A vendor may technically host data without having an unconditional legal right to produce it.
- Human review. A checkbox stating that a human approved the result is weak evidence. The record should identify the reviewer, information presented, action taken, authority to override the system and, where appropriate, the reason for accepting or rejecting its recommendation.
- Retention and termination. Contracts should define a retention period that accounts for applicable employment, civil-rights, privacy and litigation-preservation obligations. The employer should receive usable exports before termination, migration or deletion.
- Audit and remediation rights. Audit rights should cover more than policy documents. They should permit testing of relevant configurations and workflows, access to appropriate validation evidence, investigation of material disparities and corrective action when controls fail.
The agreement should also establish escalation deadlines and circumstances under which the employer may suspend automated screening without breaching minimum-volume or exclusivity commitments.
Standards provide a framework, not a litigation safe harbor
International and federal standards can help organizations structure these controls, but they should not be portrayed as substitutes for law.
ISO/IEC 42001 establishes requirements for an organizational AI management system. It addresses governance, risk management, transparency, traceability, performance evaluation and continual improvement. It does not certify that a particular hiring decision was lawful. ISO/IEC 42005 provides a complementary framework for documenting AI system impact assessments. It can help organizations evaluate intended and unintended effects throughout an AI system’s lifecycle.
NIST’s AI risk management framework similarly recommends documenting intended uses, testing assumptions, evaluation techniques, data practices and system performance. NIST is revising the framework, so companies should monitor that work rather than freeze contracts around a single framework version.
The compliance objective is not to insert a list of framework names into a contract. It is to translate legal requirements and standards into enforceable evidence obligations. Indemnification still has value, but it is a financial remedy, not an accountability architecture. An indemnity clause can pay a legal bill. It cannot recreate an AI-influenced hiring decision when the employer and vendor failed to preserve the model, configuration, output and human review.
Rohan Sharma is a US delegate to ISO/IEC JTC 1/SC 42 on AI and an expert in the OECD.AI expert group on AI risk and accountability. He is a Fulbright Specialist with the US Department of State, an Aspen Institute civic AI leader and a member of the US ACM Technology Policy Committee’s law subcommittee. Sharma is the author of “AI and the Boardroom,” published by Springer Nature, and a World Economic Forum Agenda contributor. 
