This standard provides a comprehensive set of metrics and methodologies for evaluating the performance of Vision Language Models (VLMs) across various basic tasks and application tasks in the field of computer vision. VLMs combine visual and textual data to perform tasks such as visual question answering, multi-modal generation, and image-captioning. This standard establishes a two-tier evaluation system consisting of basic tasks (covering scenes such as animals, architecture, and landscapes) and application tasks (covering scenes such as short video creation and advertising marketing), aiming to provide developers, users, and enterprises with a unified evaluation framework across multiple dimensions including functionality, accuracy, safety, and reliability. Guided by the three core principles of "objectivity and comprehensiveness, fairness and impartiality, and user perspective," the standard thoroughly analyzes the performance and potential of VLMs in all aspects, helping ensure a systematic and comprehensive understanding of their multimodal processing capabilities, and generalization capabilities.
- Standard Committee
- CIS/SC - Standards Committee
- Status
- Active PAR
- PAR Approval
- 2026-06-04
Working Group Details
- Society
- IEEE Computational Intelligence Society
- Standard Committee
- CIS/SC - Standards Committee
- Working Group
-
VLMs-Evaluation - Vision Language Models Evaluation
- IEEE Program Manager
- Sandra Maru
Contact Sandra Maru - Working Group Chair
- Weidong Liu
Other Activities From This Working Group
Current projects that have been authorized by the IEEE SA Standards Board to develop a standard.
No Active Projects
Standards approved by the IEEE SA Standards Board that are within the 10-year lifecycle.
No Active Standards
These standards have been replaced with a revised version of the standard, or by a compilation of the original active standard and all its existing amendments, corrigenda, and errata.
No Superseded Standards
These standards have been removed from active status through a ballot where the standard is made inactive as a consensus decision of a balloting group.
No Inactive-Withdrawn Standards
These standards are removed from active status through an administrative process for standards that have not undergone a revision process within 10 years.
No Inactive-Reserved Standards