Who Defines Harm?

What a Housing Story Reveals About AI Governance.

Image: Canva Stock

Maria was an immigrant, so she had moved into his house. Eleven years, a marriage and a child later, everything had changed. The relationship had turned sour.

On paper, she was still married. The house was in his name. He had a good income. As far as the database was concerned, she belonged to a household that owned its home and was not in urgent need.

In reality, he had thrown her out of that house and moved his new partner in. She was raising their nine-year-old alone and trying to find somewhere safe and stable to live. There was no legal divorce yet, no formal separation order - just a very real gap between the life the system saw and the life she was actually living. It was messy.

Maria was a New Zealand citizen now, so she applied for public housing support. She assumed the system would see what any human could see: a mother and child with nowhere secure to go. She did everything the system asked of her. Instead, she received a letter saying she did not qualify. No priority on the waitlist. No explanation of the reasoning. Just “no”.

In this version of the story, she knew enough to ask for the reasons. She knew enough about the process, enough to challenge the system, to write back and say: “Tell me why.” Many people would not. Many people never do.

So let me tell the story as it usually goes.

Freda is in exactly the same situation as Maria. She receives the same refusal letter, and although she had lived in New Zealand for thirteen years, she still didn’t know how to navigate the welfare system. She does not know that she can appeal. She does not know what data the system is using or how to challenge the assumptions behind it. She is already exhausted from the separation, the humble bank account, the hunt for cheap accommodation, the exhaustion she has to hide from her son, and the shame of asking for help. The letter lands like a full stop.

So, she sends her ten-year-old to school and reports to work – all the while living out of her car. Every day she tries to hide the stress and turmoil in her life. Desperate to keep up appearances. But she knows it is not sustainable. The winter cold bites different in a car.

Freda spends her lunch breaks on the phone, on hold, trying to get through to someone who will help. Eventually, in desperation, she walks into an organisation she once donated to, never thinking she would one day be on the receiving end - the Salvation Army - and there she is connected to a local advocate. Only then does she learn that there is an appeals process, that the decision can be challenged, that the “no” on the letter doesn’t have to be the end of the story.

By that point, the harm is already done. Her son’s asthma is acting up – his health compromised by living in the car. Freda no longer trusts the system she once believed in. The system that did not see her in her hour of need now asks her to step back into it – but does she have the stamina to appeal?

The question to ask the system in this scenario is: Who defines harm?

The forms Maria and Freda filled out asked the wrong questions. They did not capture their whole stories. And there was no human on the other end to capture the nuances. So, the answer to their request for public housing was ‘no’, and no red flag was raised.

That is what it looks like when protection only activates at the point of trouble with the design, not at the point of design itself. The system is reactive not remedial, so harm is inevitable.

Thankfully, the Algorithm Charter 2020 does not only shape how agencies think about Māori partnership under its Treaty commitment; it also sets out the only broad cross-government commitment that covers everyone else affected by algorithmic systems.

However, let us examine this promise. The Charter promises that agencies will retain human oversight and provide avenues for challenging decisions influenced by algorithms. On paper, that sounds reassuring. In practice, it raises harder questions.

· What happens when a decision is challenged?

· How is it reviewed, and by whom?

· How is the decision overturned?

· On what grounds can it be overturned?

· How is bias identified and corrected, not only in a single case but in the system itself?

· How are tikanga, cultural competence, and conscience built into the oversight process in ways that communities can trust?

· Are there guardrails?

These questions are the specification the Charter should contain and doesn’t. The truth is this - You cannot retain what was never held. The Charter promises to keep humans in control of a process they were never in control of. The algorithm was always doing the heavy lifting.

If the system is built on the wrong data points at the start, then “human oversight” becomes reactive rather than protective. It activates after the harm, not before it. It assumes that the person harmed understands the process, trusts the system enough to challenge it, and has the time, energy, literacy, and support to act. Those are not small assumptions. For many vulnerable people, they are enormous ones.

Freda eventually does get emergency housing - but only after push back, after she has mentally stilled herself for a fight. After her trust in the system is broken and her child is sick. This is not a small cost, especially because she is just one of many.

A protection that only works if the harmed person knows the rules, trusts the referee, and can afford to keep playing is not much of a protection at all. This protection lets down the people it is meant to protect.

The Charter lets us down on the people commitment too.

Commitment 3: “People”, says agencies should focus on people by “identifying and actively engaging with people, communities and groups who have an interest in algorithms, and consulting with those impacted by their use.”

At first glance, that sounds reasonable. On closer reading, it is thinner than it looks.

This is the only place in the Charter where Pasifika communities, CALD communities, disabled people, children, refugees, and other groups affected by algorithmic decisions really appear as a collective concern.

This commitment has three clear weaknesses:

The first issue is the ceiling of the engagement. The Charter sets consultation as the highest guaranteed relationship with impacted people. Sherry Arnstein’s classic ‘Ladder of Citizen Participation’ places consultation in the middle band of tokenism, while partnership sits higher up in the band of citizen power. Consultation can be a legitimate step, but if it is not linked to shared decision-making power, it remains what Arnstein called a “window-dressing ritual”. A rubber stamp that only ticks the box. So, when the Charter promises consultation to those most affected, it is not promising co-design. It is not promising shared authority. It is promising a tokenism-rung relationship to the people it most needs to protect.

The second one sits in the verbs- consultation vs engagement. Agencies are asked to ‘actively engage’ with those who “have an interest” in algorithms, and to “consult” those who are “impacted” by their use. The people the system is actually meant to support - the ones who bear the harm - get the weaker verb, those with an “interest” - the experts, researchers, industry voices, observers looking in from the outside - are invited into the room to ‘actively engage’. The hierarchy is upside down. Those who live with the consequences are given consultation - the window-dressing token. Those who study, manage, or debate the issue are given active engagement.

The third one is timing. Commitment 3 is silent on when engagement and consultation happen. Consult after the blueprint is drawn, and the agency has still technically met the wording. Consult once the architecture is fixed, and the obligation is still, on paper, fulfilled. The Charter does not require a clean slate. It does not require communities to be there when the harm is first defined or when the system logic is first set.

And again, underneath these three weaknesses sits a more fundamental question: who decides who counts as “impacted” in the first place?

If a system claims to focus on people by consulting those who are impacted, then someone has to define what the harm is, where it sits, and who carries it. But if the people who live with that harm are not in the room when those definitions are formed, then the harm is still being described from the outside in.

It is like designing a house you have never lived in. You can sketch the floor plan. You can choose where the walls go. You can decide how many bedrooms there should be. But if you have never had to move through that house as a single mother with a child, or as a disabled tenant, or as a refugee family navigating systems in a second language, then you do not know what rooms are actually needed.

That is the weakness in Commitment 3. It has the feel of a problem being studied and documented, not solved. Those “with an interest” are actively engaged. Those who live the consequences are consulted.

This is why I keep returning to the difference between open loop and closed loop design.

An open loop observes the problem from the outside. It studies the issue, measures the outputs, waits for complaints, and responds when something goes visibly wrong. It treats people as data points and cases to be managed.

A closed loop does something different. It keeps going back to the people who live the consequences and asks: what did we miss? What has changed? What are we not seeing from here? It understands that people, situations, and harms are dynamic. They evolve. Any system built around human lives has to be able to evolve with them. The people ARE the data. Without them, you don’t understand the picture.

That is one of the deepest weaknesses in the charter. It does not require continuous revision with the people affected. It does not require agencies to return to communities once the tool is deployed and ask whether the assumptions still hold. It does not require a loop to be closed.

There is also a quiet demotion risk within the Algorithm Charter 2020. Commitment 2, “Partnership”, is Treaty-specific and sits in a different category from Commitment 3, “People”. That distinction is key. If agencies start treating Māori as simply another community to consult under Commitment 3, rather than as Treaty partners under Commitment 2, then Māori standing is quietly demoted from partnership to stakeholder status. The Treaty relationship begins to dissolve into the general people bucket. What should be a distinct commitment to shared authority becomes ordinary consultation.

This is not a minor wording issue. It changes the power structure of the whole document.

Freda sleeping in her car should not have to become an expert in appeals before the system sees her. Her child should not have to experience the harm before human oversight wakes up. And the people most affected by algorithmic decisions should not be consulted as an afterthought while outsiders are actively engaged as the main conversation partners.

This is why my Ubuntu-informed AI governance lens keeps returning to the same principle: Communities are not the subject of the system. They are the source of it.

^{Denise Barlow-Byarugaba is a Human Systems SME and the founder of The Room, where she develops Ubuntu-informed AI governance — a framework for designing automated systems with the communities that carry the cost of their failures. She brings 20+ years working alongside Māori, Pasifika, and CALD communities across Aotearoa and East Africa, and works on both sides of the glass: the frontline, where she has seen what these systems do to people, and inside the AI labs, where she has seen how they are built. She partners with NGOs, iwi, Pasifika and CALD organisations, and government agencies on how emerging technologies land in real lives.}