This standard specifies test methods for evaluating the robustness of a Natural Language Processing (NLP) service that uses machine learning. Models of NLP generally feature an input space being discrete and an output space being almost infinite in some tasks. The robustness of the NLP service is affected by various perturbations including adversarial attacks. A methodology to categorize the perturbations, and test cases for evaluating the robustness of an NLP service against different perturbation categories is specified. Metrics for robustness evaluation of an NLP service are defined. NLP use cases and corresponding applicable test methods are also described.
- Sponsor Committee
- C/AISC - Artificial Intelligence Standards Committee
- Status
- Active PAR
- PAR Approval
- 2022-05-13
Working Group Details
- Society
- IEEE Computer Society
Learn More About IEEE Computer Society - Sponsor Committee
- C/AISC - Artificial Intelligence Standards Committee
- Working Group
-
RAIBS - Robustness of Artificial Intelligence Based Service
Learn More About RAIBS - Robustness of Artificial Intelligence Based Service - IEEE Program Manager
- Christy Bahn
Contact Christy Bahn - Working Group Chair
- Qing An
Other Activities From This Working Group
Current projects that have been authorized by the IEEE SA Standards Board to develop a standard.
No Active Projects
Standards approved by the IEEE SA Standards Board that are within the 10-year lifecycle.
3129-2023
IEEE Standard for Robustness Testing and Evaluation of Artificial Intelligence (AI)-based Image Recognition Service
Test specifications with a set of indicators for common corruption and adversarial attacks, which can be used to evaluate the robustness of artificial intelligence-based image recognition services are provided in this standard. Robustness attack threats and establishes an assessment framework to evaluate the robustness of artificial intelligence-based image recognition service under various settings are also specified in this standard.
These standards have been replaced with a revised version of the standard, or by a compilation of the original active standard and all its existing amendments, corrigenda, and errata.
No Superseded Standards
These standards have been removed from active status through a ballot where the standard is made inactive as a consensus decision of a balloting group.
No Inactive-Withdrawn Standards
These standards are removed from active status through an administrative process for standards that have not undergone a revision process within 10 years.
No Inactive-Reserved Standards