Active Standard

IEEE 3168-2024

IEEE Standard for Robustness Evaluation Test Methods for a Natural Language Processing Service That Uses Machine Learning

The natural language processing (NLP) services using machine learning have rich applications in solving various tasks and have been widely deployed and used, usually accessible by application programming interface (API) calls. The robustness of the NLP services is challenged by various well-known general corruptions and adversarial attacks. Inadvertent or random deletion, addition, or repetition of characters or words are examples of general corruptions. Adversarial characters, words, or sentence samples are generated by adversarial attacks, causing the models underpinning the NLP services to produce incorrect results. A method for quantitatively evaluating the robustness the NLP services is proposed by this standard. Under the method, different cases the evaluation needs to perform against are specified. Robustness metrics and their calculation are defined. With the standard, understanding of the robustness of the services can be developed by the service stakeholders including the service developer, service providers, and service users. The evaluation can be performed during various phases in the life cycle of the NLP services, the testing phase, in the validation phase, after deployment, and so forth.

Sponsor Committee
C/AISC - Artificial Intelligence Standards Committee
Status
Active Standard
PAR Approval
2022-05-13
Board Approval
2024-03-21
History
Published:
2024-08-09

Additional Resources

Erratas
3168-2024_errata.pdf

Working Group Details

Society
IEEE Computer Society
Learn More About IEEE Computer Society
Sponsor Committee
C/AISC - Artificial Intelligence Standards Committee
Working Group
RAIBS - Robustness of Artificial Intelligence Based Service
IEEE Program Manager
Christy Bahn
Contact Christy Bahn
Working Group Chair
Qing An

Other Activities From This Working Group

Current projects that have been authorized by the IEEE SA Standards Board to develop a standard.


No Active Projects

Standards approved by the IEEE SA Standards Board that are within the 10-year lifecycle.


3129-2023
IEEE Standard for Robustness Testing and Evaluation of Artificial Intelligence (AI)-based Image Recognition Service

Test specifications with a set of indicators for common corruption and adversarial attacks, which can be used to evaluate the robustness of artificial intelligence-based image recognition services are provided in this standard. Robustness attack threats and establishes an assessment framework to evaluate the robustness of artificial intelligence-based image recognition service under various settings are also specified in this standard.

Learn More About 3129-2023

These standards have been replaced with a revised version of the standard, or by a compilation of the original active standard and all its existing amendments, corrigenda, and errata.


No Superseded Standards

These standards have been removed from active status through a ballot where the standard is made inactive as a consensus decision of a balloting group.


No Inactive-Withdrawn Standards

These standards are removed from active status through an administrative process for standards that have not undergone a revision process within 10 years.


No Inactive-Reserved Standards
Subscribe to our Newsletter

Sign up for our monthly newsletter to learn about new developments, including resources, insights and more.