VMLU.ai: Advancing Vietnamese Language Model Understanding

VMLU.ai is a comprehensive Vietnamese Multitask Language Understanding Benchmark Suite designed to evaluate and improve large language models across a wide range of academic disciplines. Built by ZaloAI and Jaist, it features 10,880 multiple-choice questions spanning 58 subjects in four key domains—STEM, Humanities, Social Sciences, and Others—covering difficulty levels from elementary to professional standards. The dataset draws from authentic educational sources, including national exams and university assessments, ensuring high relevance and rigor. Available in multiple formats (Vi-MQA, Vi-Dialog, Vi-Drop, Vi-SQuAD), it supports research, development, and benchmarking with full transparency via GitHub, including documentation, sample code, and performance results. Ideal for AI researchers and developers focused on Vietnamese language capabilities, VMLU.ai drives progress in multilingual AI comprehension.

Key Features:

• 10,880 multiple-choice questions across 58 subjects
• Four domain categories: STEM, Humanities, Social Sciences, and Others
• Multi-level difficulty: elementary to professional (undergraduate and graduate exam standards)
• Data sourced from real Vietnamese educational institutions and national exams
• Support for multiple data formats: Vi-MQA, Vi-Dialog, Vi-Drop, Vi-SQuAD
• Open access via GitHub with full documentation, sample code, and benchmarking results
• Designed for evaluating and improving large language models in Vietnamese

Pricing: VMLU.ai is offered as a free resource for research and development, with no cost to access or use the benchmark suite.

Conclusion: VMLU.ai is a powerful, open-access benchmark that sets a new standard for evaluating Vietnamese language understanding in AI models, empowering researchers and developers to advance multilingual AI with high-quality, real-world data.