If a funder read our monitoring report and knew that AI helped write it, would they trust it
less?
It’s a question we’ve been thinking about at the Trevor Noah Foundation, not because we
have a clean answer, but because we believe asking the question is part of what responsible
MERL practice looks like right now. This year’s Glocal Evaluation Week theme: Evaluation,
Evidence and Trust in the Age of AI, landed close to home. Rather than hosting an event, we
are offering something more honest: a reflection from inside the work.
The unglamorous reality
We are a small team at TNF. Our work spans across three flagship programmes, the Khulani
Schools Programme (KSP), the Education Change Makers Programme (ECM), the Khulani
Nathi innovation Fund (KNIF) and a wide set of ecosystem engagements with partners,
grantees and stakeholders across the education space. Within KSP specifically, we work in
three school communities, namely: Braamfischerville, Ivory Park and Eldorado Park. We
coordinate programmes, track outcomes, engage stakeholders, produce reports and
communicate findings, often simultaneously and almost always with limited capacity.
AI has quietly become part of how we manage that. We use it to draft data collection tools,
we use it to synthesise qualitative inputs from school reports, and it is able to pull themes
from feedback, something that would otherwise take days to process.
None of this is dramatic, it looks less like a technological transformation – at least for now –
and more like having a very fast, very available colleague who is good at first drafts. The
value is real; a small team producing documentation quality that genuinely punches above
its weight without sacrificing precision.
This is, in fact, how our Managing Director, Shalane framed it from the start. The
encouragement to use AI within our team has never just been about doing more with less. It
has been about recovering time, so that we can spend it where it cannot be replicated, on
the ground, in schools, building relationships and trust that makes our programmes work.
The logic is simple.
What does time buy?
Imagine two versions of the same Tuesday.
Version one. A report for the AI Teacher Training programme is due on Friday. You scroll
through the facilitator WhatsApp group chat and monthly reports, drop recent updates into
an AI prompt, and get back a clean summary: “increased confidence among participants “,”
improved attendance patterns”. It’s accurate. It’s enough. You file it and move on to the next
task, without going to the school that week. Do that often enough, and you could write a
competent report about a school or community you would barely recognise if you walked into
it.
Version two. The draft report takes you twenty minutes instead of two hours and after looking
at your draft, you decide to go to the school to attend the training. You sit in on the session.
You notice a teacher has gone quiet these past few days, something no AI model would
have flagged. Over tea afterward, you find out why.
Because that “colleague” has never worked in Ivory Park, Braamfischerville or Eldorado
Park. They have never set foot there or seen the change physically. That human interaction
(HI) changes how the programme facilitators show up for that teacher next week, and it’s the
kind of change only human interaction (HI) could have surfaced.
The difference between the two Tuesdays is not the AI. It’s what you do with the time it gives
back.
A monitoring report drafted faster is only valuable if the time saved goes back into knowing
the school and community well enough to write something that is true about them. The
closeness to our school communities is not incidental to our M&E practices, it is what makes
it credible in the first place. Lean too hard on AI, and you wouldn’t notice at first. The
relationships would simply get quieter, and only later would that silence show up in how
honestly we are able to report.
The voice problem
AI produces competent texts. It’s neutral, professional and well structured. It is the kind of
language that reads as ‘credible’ because it sounds like no one in particular.
That is a problem when your evidence is supposed to carry the texture of real places and
real people. When a teacher in Ivory Park describes the changes in her school and school
community, that story carries a specific weight. It reflects a relationship built over months, a
context that takes time to understand, and a particular way of speaking specific to that place.
When AI synthesizes across 10 stories and produces a clean thematic summary, something
gets smoothed out. The synthesis may be accurate, but whose voice is in it?
If the output of our MERL work is evidence that is credible to funders and meaningful to
communities, then the process of producing that evidence has to be held to account. AI does
help us work faster, but it cannot tell us whose story we are relaying.
So should funders trust it less?
Not necessarily, but trust has to be earned differently now. The monitoring report that AI
helped draft is not less credible because AI was involved. It is less credible if no one
interrogated the synthesis, if the voice of the community was edited out in an AI prompt
wanting a ‘professional tone’ and if the data that went in was thin and the confident language
that came out disguised that.
What we have learned is that the human judgment required in MERL has not decreased with
AI. It has changed. The analytical work now happens much earlier. We need to decide and
know much earlier on what type of data matters, what questions to ask and what a
community’s account should be allowed to mean and in the final review. In the final review,
we need to make sure that someone who actually knows the context reads the output and
ask whether it is true, not just whether it sounds right.
This requires investment in people, in relationships with communities and in an
organisational culture that sees MERL as a practice of integrity.
Where we still need to be more intentional
Honestly, we do need a formal in depth document that guides us on how to use AI in our
organisation. We are always thinking about what information, if any, is appropriate to include
in AI prompts, how to ensure AI outputs are interrogated and how to ensure that we still
build analytical capacity and not outsource the thinking that build professional judgement.
Why this matters
Rigorous evaluation is more important now, not less. Organisations and institutions need to
take care and be careful not to generate data at a large scale without it being true.
Organisations that are careful and accountable to the communities they serve are providing
something that is genuinely becoming scarce.
We do not think AI is a threat to true, accountable MERL outputs, unless it is unreflective.
– Mcebisi Kunene, Special Projects & MERL Coordinator
