OpenAI has introduced IndQA, a new benchmark evaluation designed to assess the linguistic abilities and grasp of Indian cultural context within Indic large language models (LLMs). This benchmark comprises 2,278 questions across 12 languages and 10 cultural domains, developed through a partnership with 261 experts from across India. Notably, IndQA includes Hinglish to account for the prevalence of code-switching in everyday conversations.
OpenAI stated that its own models might face a disadvantage during this evaluation, as the test questions were specifically chosen based on areas where their models previously encountered difficulties. The introduction of IndQA aims to address key challenges in developing Indic LLMs, which include a scarcity of high-quality datasets and a lack of localized benchmarks suitable for evaluating these models.
Currently, most existing language benchmarks are predominantly focused on English and European languages, often limiting evaluations to tasks like translation or multiple-choice questions. This historical focus potentially hinders AI adoption in India, where AI-powered speech recognition requires the processing of diverse accents and the common practice of mixing English with local languages.
Source: https://indianexpress.com/article/technology/artificial-intelligence/can-openai-new-indqa-benchmark-help-indic-llms-close-gap-10348131/