The New Zealand Department of Internal Affairs (DIA), commissioned an independent Facial Recognition Trial of their Identity Check system using DIA passport face images to test against. Buried in the report was high probability of bias against Māori. DIA now say they will further test after asked by media in relation to this article.
The independent test was conducted by BixeLab, one of only a handful of certified auditors in the world. Their report (version 3) sates it was only tested on 60% of the recommended number of people to get a 95% accurate rate, and only on internal DIA staff.
The report also ignored Māori with facial moko, and shows a high probability of bias to Māori with facial moko, consistent with Māori who have moko not being identified with their personal devices such as iPhone ID and Samsung Face Recognition. But the test hints that Māori with facial moko will likely be more likely victims of Face Recognition fraud.
Furthermore, the Facial Recognition test is not scientific, discriminates against Māori and Māori with moko, and the report obfuscates the data to hide these facts with several media reporting that the Facial Recognition does not discriminate against Māori, echoing false statements in the report. Despite the report claiming otherwise. RNZ did a follow up to their original story, where the DIA stated they “are committed to engaging and collaborating further with Māori and the wider community and intends to conduct further studies“.
Stats
The recommended number of peoples to gain a 95% confidence level in the system is 245 people. DIA only used 148 of their own staff members in the trial. This is not representative of the community who will be impacted by the usage of Facial Recognition, but a representative group of well-educated government burecarts of Māori and Pacific ethnicity who live in the main centres of New Zealand where there is a DIA office.
Demographically, DIA sated that Māori and Pacifica represent 30-50%. The report included 39% ratio of Māori/Pacifica staff members as volunteers in the trial but failed to differentiate between Māori and the multiple ethnicities in the Pacific region.
Moko
No Māori with a facial moko were tested, despite being acknowledged in the report as a 10-30% of the demographic. This hides a bias of the system with any Māori with a facial moko because match score distributions for impostor comparisons (the system’s ability to distinguish between genuine and impostor attempts) show the Pasifika and Māori groups clearly have a higher proportion. A manual check by staff of this inconsistency found that “shadows, and reflections on a face” is the reason. This shows that Māori with a facial moko will likely be more impacted and at a higher risk of fraud and discrimination with FRT technologies.
What do Māori actually look like?
The DIA staff who volunteered for the trial only had to self-identify their ethnicity. For Māori, we have to consider that the Māori population has been heavily colonised resulting in mixed marriages and children since the 17th century. So much so, that there are no longer any 100% blood quantum Māori individuals. Historically, the further south and port townships were far more colonised than remote areas such as the far north and the Urewera’s.
This results in the fact that there is no one distinctive Māori look, also noting here that Māori migrated here from multiple islands with each tribe often having unique appearances relative to their tribe. Māori physical features range from identifiable Polynesian appearances with various shades of brown skin/eyes and black hair to Caucasian looking Māori.
In New Zealand legislation there is no Blood Quantum rules to identify as Māori and the legislation simply states that anyone with a Māori ancestor can claim to be Māori. The same is true with Māori cultural customs where any genealogical connection to a Māori ancestor is all that is required. The trial relied on the integrity of staff, while it is common for people to identify as Māori when they are not.
There is no differentiation of the appearance of the Māori who were tested – were they Polynesian looking, Caucasian or in between? If the test had all Māori who physical appearance was a Caucasian, this would discriminate against Māori of Polynesian appearance.
Conclusion
The report is difficult to consume and identify the key issues highlighted in this article that Māori are discriminated against with the DIA Facial Recognition Technology.
To seek an accurate and statistically probable result the testing should be applied to the recommended minimum amount of people and include Māori, count how many and ensure they are from diverse physical appearances and with facial moko to check for bias.
The approved auditor should also have a Māori culture expert to peer review how the test is being conducted to ensure that it is representative and does not create further bias against Māori data and misleading statements about the Facial Recognition System.
Leave a Reply