IMDA launches experimental AI platform to reveal cultural gaps in current landscape

share on

twitter /
facebook /
linkedin /
- email
- telegram
- whatsapp
- wechat
- pinterest
- line
- snapchat
- reddit

The Infocomm Media Development Authority (IMDA) and the AI Verify Foundation have created a first of its kind Generative AI (Gen AI) Evaluation Sandbox. The Sandbox aims to reveal gaps in the current landscape of Gen AI evaluations, particularly in domain-specific and cultural-specific areas which are currently under-developed, it said in a statement.

The Sandbox will bring global ecosystem players together through concrete use cases, to enable the evaluation of trusted AI products.

Some of these players include key model developers such as Google, Microsoft, Anthropic, IBM, NVIDIA, Stability.AI and Amazon Web Servies (AWS) as well as app developers with concrete use cases such as DataRobot, OCBC, Global Regulation Inc, Singtel and X0PA.AI.

It will make use of a new evaluation catalogue as a shared resource that sets out common baseline methods and recommendations for Large Language Models (LLM).

Don't miss: Ant Group sets up in SG, partners IMDA to push sustainable digital consumption

This is part of efforts to have a common and standard approach when it comes to assessing Gen AI and to support a safer and more trustworthy adoption of Gen AI, it said.

IMDA added that beyond just a starting baseline of evaluation tests, how AI is implemented across the ecosystem of those who build AI also needs to be developed.

The Sandbox will then help to build evaluation capabilities beyond what currently resides with model developers and will include the application developers who build on top of the models.

It will also involve players in the third-party testing ecosystem, to help model developers understand what external testers would look for in responsible AI models. These include companies such as Resaro.AI, Deloitte, EY and TÜV SÜD.

Each Sandbox case will ideally involve an upstream Gen AI model developer, a downstream application deployer and a third-party tester to demonstrate how the different players in the ecosystem can work together.

"By involving regulators such as the Singapore Personal Data Protection Commission (PDPC), the Sandbox will provide a space for experimentation and development and allow all parties along the supply chain to be transparent about their needs," it said.

These use cases are expected to reveal gaps in the current landscape of Gen AI evaluations and will likely reveal domain-specific and cultural-specific areas which are currently under-developed.

The Sandbox will then develop benchmarks for evaluating model performance in specific areas that are important for use cases, and for countries such as Singapore because of cultural and language specificities.

IMDA cited an example of a Sandbox project that used the catalogue by referencing its collaboration with Gen Ai developer Anthropic.

"IMDA will leverage Anthropic's models and research tooling platform to develop and tailor red-teaming methodologies for Singapore’s diverse linguistic and cultural landscape, for example, testing AI models for their abilities to perform well for Singapore’s multi-lingual context," it said. Red-teaming is a process of rigorous testing to find gaps in safety.

“Promoting an ecosystem of independent, open source, and third-party model evaluations is critical to building safe and trustworthy AI. The AI Verify Foundation and Generative AI Sandbox is an important step in that direction. We appreciate our strong cooperation with IMDA and look forward to deepening our partnership.” said Dario Amodei, co-founder and CEO of Anthropic.

This comes as Singapore makes new strides in building responsible AI as the world grows increasingly more dependent on it.

It introduced the AI Verify Foundation this year and the launch of the Sandbox marks the advancement of AI Verify into Generative AI by tapping on the collective power and contributions of the global open-source community.

share on