AI Collaboration Literacy: Validity Evidence Across Eval, Spec, and Handoff

AI Collaboration Literacy (ACL) is the umbrella construct covering the skills a person needs to do useful work alongside a language model — not the narrow skill of writing clever prompts, but the broader literacy that includes designing evaluations, translating ambiguous goals into testable specifications, and handing off intermediate work between human and model. Where AI Output Evaluation focuses on the judgment side of the loop, ACL covers the upstream and downstream skills: framing the task well enough that the model has a chance, then integrating model output back into the broader work product. The construct has emerged as one of the most diagnostic predictors of on-job performance in AI-paired roles, but its validity evidence is still maturing.

This article walks through the construct decomposition, the emerging validity evidence base, AIEH’s ACL prompt-to-spec sample and how it integrates into the Skills Passport composite, the recurring measurement pitfalls, and the takeaway for teams designing AI-aware selection.

Data Notice: Validity evidence for ACL as a selection construct is still accumulating. Specific weights AIEH applies to ACL evidence in the AI fluency pillar are documented in the scoring methodology and may evolve as calibration data accrues.

What ACL measures

The ACL construct decomposes into three primary layers:

• Eval design. Given a task, can the candidate define a small evaluation set — inputs paired with expected outputs or scoring criteria — that lets them measure whether a given prompt or model is actually working? The skill is closer to test-driven development than to prompt engineering.
• Prompt-to-spec. Given an ambiguous goal, can the candidate translate it into a specification precise enough that a language model can attempt it? The skill captures the disambiguation work that has always existed in software engineering but is now surfaced more frequently when humans pair with models.
• Handoff. Given a partially-completed task — some done by the model, some done by the human — can the candidate manage the seams? Knowing when to ask the model to continue, when to take over, when to restart with a different prompt, when to abandon the AI route entirely.

The construct is distinct from AOE (which focuses on judging finished outputs) and from prompting skill (a narrower construct than ACL). A candidate strong in ACL may use a model to scaffold a piece of work, write their own evaluation harness, run the model output through that harness, iterate on the prompt until the output passes, then integrate the result. A candidate weak in ACL may write a prompt, accept whatever output comes back, and move on without evaluation.

The validity evidence base

The validity evidence for ACL as a selection construct is younger than the work-sample literature, but the inheritance argument is sound. An ACL task is a work-sample test of AI-paired work, and Schmidt and Hunter’s 1998 meta-analytic synthesis established that work-sample tests deliver corrected operational validity around ~0.54 against supervisor performance ratings. ACL work samples inherit that ceiling when they preserve point-to-point correspondence with the criterion.

Sackett and Lievens (2008) frame the predictor-criterion fidelity question as the core driver of validity. An ACL assessment that asks the candidate to define an eval set, write a prompt against an ambiguous goal, and manage handoff with the model in real time inherits the validity ceiling; an assessment that probes trivia about prompting techniques sits well below.

Liang et al’s 2022 HELM benchmark established the methodology for evaluating language models across multiple dimensions, and the same dimensional thinking translates to the human side: a candidate who designs evaluations spanning factuality, calibration, robustness, and bias is doing higher-quality work than one who designs single-dimension evaluations. Bender et al (2021)‘s “Stochastic Parrots” critique frames the underlying motivation: language models can produce fluent but wrong text, and the human in the loop has to build evaluation infrastructure that catches the failures. ACL is the operational name for the skill of building that infrastructure.

Kojima et al (2022)‘s zero-shot prompting work established that small variations in prompt design substantially shift model performance — implying that prompt-to-spec skill (a sub-component of ACL) has measurable construct variance worth selecting on.

AIEH ACL test integration

AIEH treats ACL evidence as a primary input to the AI fluency pillar of the Skills Passport composite. The ACL prompt-to-spec sample presents candidates with an ambiguous goal, asks them to produce a specification precise enough for a model to attempt, then asks them to design a small evaluation set against which the model output can be scored. The rubric covers specificity of the spec, coverage of the eval set, and the candidate’s ability to articulate what they would do with model output that fails the eval.

The composite weights ACL evidence at substantially higher relevance for roles where AI fluency is diagnostic. For an AI-collaborative engineering or research role, ACL may carry weight comparable to a domain skill test; for a role with low AI exposure, the weight is smaller. The scoring methodology documents default weights and role-bundle modifiers.

For hiring teams, the hire workspace surfaces ACL evidence within the AI fluency pillar, with provenance visible alongside the calibrated number. The ai fluency in hiring article covers the broader frame for why AI fluency has emerged as a stable selection construct distinct from general cognitive ability.

Construct stability and rapid model evolution

A specific concern for ACL assessment is the speed of model evolution. Capabilities that required substantial prompt engineering eighteen months ago — chain-of-thought reasoning, structured output generation, tool use — have become easier or zero-shot in newer model generations. A candidate’s ACL skill measured against a prior-generation model may not directly transfer to the current generation, where the bottleneck has shifted from “getting the model to attempt the task” to “evaluating whether its attempt is good enough.”

The implication is that the ACL construct itself is moving — the eval-design layer is becoming more diagnostic relative to the prompt-to-spec layer as prompt-to-spec gets easier. AIEH’s ACL prompt-to-spec sample is calibrated to current-generation model capabilities and will be updated as the capability frontier shifts. The scoring methodology documents the recalibration cadence.

The Kojima et al (2022) finding that small prompt variations substantially shift model performance remains relevant — it establishes that ACL skill has real construct variance — but the magnitude of the shifts produced by prompt variation is decreasing as models become more robust to instruction-design choices. The diagnostic emphasis correspondingly shifts toward eval design and handoff judgment. The ai fluency in hiring article tracks this construct evolution.

Pitfalls that collapse ACL test validity

The recurring failure modes:

• Reducing ACL to prompting. A test that scores only the prompt the candidate writes — not the evaluation harness or the handoff judgment — measures a narrower construct than ACL. The full construct requires a multi-step task.
• Static rubrics for dynamic work. ACL is partly about responding to what the model does; a rubric scored only on the candidate’s first prompt ignores the iteration loop. Rubrics should score the candidate’s response to model output, not just their initial spec.
• Mono-domain content. A test built only around software-engineering tasks predicts a narrower criterion than one covering writing, analysis, and customer-communication tasks. For general AI-fluency selection, the assessment should span domains.
• Ignoring eval design. Many AI assessments score prompting skill but not evaluation-design skill. A candidate who writes elegant prompts but cannot evaluate the output is shipping defects; the eval layer is the higher-value diagnostic.
• Time pressure dominating. A multi-step ACL task scoped for ninety minutes and curtailed to twenty measures speed under pressure, not collaboration literacy. Calibrate the window to the median completion time of strong candidates in pilot testing.

For practical guidance on integrating ACL evidence with broader interview signal, see the structured interview design and interview question design articles.

Takeaway

AI Collaboration Literacy is a measurable skill with a work-sample-grade validity ceiling when the assessment preserves point-to-point correspondence with the criterion of doing AI-paired work in realistic contexts. The construct decomposes into eval design, prompt-to-spec, and handoff, and a defensible assessment exercises all three layers. AIEH’s ACL prompt-to-spec sample feeds into the AI fluency pillar of the Skills Passport composite at high relevance for AI-collaborative roles.

For hiring teams, the practical implication is that ACL evidence becomes increasingly diagnostic as roles shift toward AI-paired work. A candidate who treats the language model as a magic box and ships its output unevaluated is producing the kind of defects Bender et al (2021) warned about; a candidate who builds an evaluation harness and iterates is producing defensible output. The ai fluency in hiring article covers the broader case for treating AI fluency as a distinct pillar, and the skills-based hiring evidence article covers the portability frame.

Sources

• Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274.
• Sackett, P. R., & Lievens, F. (2008). Personnel selection. Annual Review of Psychology, 59, 419-450.
• Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of FAccT 2021, 610-623.
• Liang, P., Bommasani, R., Lee, T., et al. (2022). Holistic evaluation of language models (HELM). arXiv preprint arXiv:2211.09110.
• Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35, 22199-22213.