Announcing NIST's Artificial Intelligence Technology Evaluation (AITE)

The Technology Test and Evaluation Division at NIST is launching a new program to provide researchers with a sequestered testbed environment for the evaluation of AI model performance in a variety of meaningful tasks across diverse datasets, modalities, and domains.

The Artificial Intelligence Technology Evaluation (AITE) provides volunteer testing of AI models on blind data and its sequestered environment mitigates the risk of train/test data contamination to ensure rigorous, objective assessment. The three initial tasks focus on image analysis using large vision language models (VLMs) in the context of (1) quantum science, (2) genomics, (3) public safety. Additional tasks will be added over time. The infrastructure provided by NIST will provide common data, metrics and scoring to help developers understand the performance of their models.

AITE will rely on engagement from participants in two different tracks, each offering distinct advantages:

Participation is open to all who wish to engage in one of the ways described above, and who can abide by the AITE Participation Agreement and rules. To request participation or ask questions, contact us at aite-poc [at] list.nist.gov (aite-poc[at]list[dot]nist[dot]gov).

To learn more, we invite you to explore the AITE evaluation overview and specifications for the initial three tasks. These resources can be found by clicking the button below.Learn More