Evaluating AI for Social Impact in Africa and Asia

Charting a Course for Ethical AI Deployment in Africa and Asia

Born out of the AI for Development Funders’ Collaborative, AEASI is a cross-donor effort and the first phase of a broader collaboration focused on advancing AI evaluation. The initiative brings together key global partners: the Abdul Latif Jameel Poverty Action Lab (J-PAL), the UK’s Foreign, Commonwealth & Development Office (FCDO), the International Development Research Centre (IDRC), IDinsight, Community Jameel, and Google.org.

The Artificial Intelligence for Development (AI4D) program has long fostered ethical, locally-driven innovation, recognizing the immense capacity of local actors to drive solutions according to their own contexts. As an early supporter of Africa’s AI ecosystem, the AI4D program has continually driven the ethical use of technology, influencing policy, strengthening AI capacity, and promoting Southern perspectives in global research discourse.This foundational work, which localises decision-making and supports the decentralisation of knowledge, set the stage for a critical new phase of collaboration.

Marking a significant step toward evidence-based policymaking, the formal launch of the AI Evidence Alliance for Social Impact (AEASI) took place at the AI for Africa Conference in Cape Town, hosted by G20 South Africa, the African Union, and UNESCO.

Defining the Evidence Agenda

The overarching goal of the alliance is to build a robust evidence base to shape future AI investments and governance, particularly across lower- and middle-income countries (LMICs) in Africa and Asia. The projects will generate and share actionable research on how AI affects development outcomes, focusing on understanding which AI tools work, which do not, and for whom they work.

George Richards, Director at Community Jameel, underscored the necessity of this evidence-first approach, “AI has the potential to tackle some of the world’s most pressing challenges, but we need evidence of which AI solutions work effectively, safely and fairly in order to maximise its impacts. We're excited to be launching this new alliance to help generate rigorous evidence that we need and scale effective AI solutions to benefit communities around the world.”

Advancing Responsible AI and Coordination

One of the alliance’s core sub-projects, "Advancing Responsible AI: Evaluating AI4D Investments for Impact and Equity," aims to fill a critical gap by embedding evaluation directly into AI4D’s design and implementation. This helps identify “good bets” in AI for development and ensures that technologies benefit marginalized populations.

This effort consists of three core pillars:

• Co-developing a strategic learning agenda aligned with program goals.
• Conducting targeted evaluations of AI4D-funded projects to assess social impact, effectiveness, and scalability.
• Facilitating knowledge sharing across the AI4D Funders Collaborative and the broader ecosystem.

Furthermore, a learning agenda will be developed to map the AI for development landscape, enabling funders and implementers to coordinate efforts and avoid duplication.

The Four Levels of Evaluation

This AI evaluation framework by The Agency Fund, the Center for Global Devleopment and the Abdul Latif Jameel Poverty Action Lab (J-PAL) provides a scope that both funders and organisations can use to assess AI’s true impact.

1. Level 1: Model Evaluation answering the question, Does the AI behave the way you want it to?
2. Level 2: Product Evaluation answering the question, Is the product being used as intended
3. Level 3: User Evaluation answering the question, Does using the product positively influence users’ thoughts, feelings, and actions
4. Level 4: Independent impact evaluation, answering the question, Is the AI product a cost-effective way to improve development outcomes?

The second sub-project, Evaluating AI for Development Use Cases, directly supports this framework by funding and supporting a portfolio of experimental evaluations. This initiative will also strengthen local African and Asian research leadership and capacity, ensuring findings are translated into clear guidance for policymakers and practitioners.

Looking Ahead

The AEASI represents a collective commitment to responsible scaling and inclusivity. Maggie Gorman Velez, Vice president, Strategies, Regions and Policies at IDRC, confirmed this, sharing that “IDRC is proud to support this new evaluation work as part of our ongoing commitment to the responsible scaling of proven, safe, inclusive and locally relevant AI innovations.”

Iqbal Dhaliwal, Global Executive Director of J-PAL and co-leader of AEASI, emphasized the Alliance’s potential to maximize social utility: “J-PAL has a long history of evaluating innovative technology and its ability to improve people’s lives. While AI has incredible potential, we need to maximise its benefits and minimise possible harms. We’re grateful to Community Jameel, FCDO and IDRC for their catalytic investment to launch AEASI which will help us expand evidence on the impacts of AI.”