The State of Facial Recognition Software in 2022

No one could have predicted how much the facial recognition market would change in 2020. 

Even at the start of the year, the sector was facing a growing tide of popular dissent¹ and calls from around the world for increased legislation, oversight and accountability². Then, almost immediately, the world was wearing a mask. Police officers who once would caution and fine citizens for covering their faces³ were now arresting them for not covering them⁴.

At the same time, the repurposing or new deployments of facial recognition systems with a COVID-19 agenda re-fermented discussion, controversies, and talk of legislative action⁵. In response, many of the biggest facial recognition providers on the planet called a moratorium on selling these services to law enforcement.

It's a year like no other, with nearly all the long-term market analysis predictions wrong-footed by the pandemic. Therefore in this article, we'll take a broader look at the current capabilities of computer vision software development for facial recognition purposes, including some of the technical, social and increasing regulatory challenges that the sector will need to overcome to regain its former momentum in a post-COVID-19 market.

According to a recent NIST report¹³, the leading facial recognition algorithm in 2020 has an error rate of just 0.08% in comparison to the 4.1% error rate for the market leader in 2014, with over 30 other algorithms also exceeding the best score from 2014.

However, several factors stand as an obstacle to the wide applicability of any single facial recognition accuracy ranking system.

NIST observes that the inconsistent quality obtained from a wide diversity of capture inputs presents a problem in terms of both achieving and monitoring accuracy across the available range of facial recognition systems.

If that data is too inconsistent, the machine learning models powering a modern facial recognition system tend to converge at a mean quality that may not accurately represent any of the input data¹⁴.

When the image data varies wildly in quality, the model must either generalize or else reject data below a certain quality threshold, depending on how it is weighted. Either way, some potential accuracy is sacrificed in the process.

For law enforcement networks, available inputs will vary from relatively crisp mugshot databases to images from blurred or indistinct video feeds, such as from CCTV systems that were outfitted decades ago and intended for manual monitoring rather than as a source of primary evidence.

In 2019, a study out of the University of Notre Dame¹⁵ examined the problem of varying facial capture quality and concluded that today there is no clear theoretical guidance for designing new architectures to specifically fit this task.

However, it notes that there is ongoing research into multi-dimensional models (see below) that are more capable of utilizing images of any quality without losing accuracy across the range of images¹⁶.

A second approach is to identify 'quality tiers' of data inside the neural network itself, and use this information to adapt the model's weights and to make it less reductive where data quality varies — the aforementioned 'multidimensional' method.

A research collaboration between Microsoft, Tencent, and various universities¹⁷ proposes a Feature Adaptation Network capable of distinguishing multiple scales of quality in data input via a Random Scale Augmentation (RSA).

The technique claims to achieve normalization while maintaining the feature applicability of each level of quality from diverse sources.

Some facial recognition techniques are beginning to employ Generative Adversarial Networks (GANs)¹⁸ and adapted encoder/decoder models to produce highly detailed faces from extremely low-quality images.

This 'super resolution' technique leverages the abstraction capabilities of a neural network¹⁹: irrespective of input image quality, a machine learning model will turn a bitmapped image input into a mathematical vector formulation at the encoder stage — a unique hash that represents the essential characteristics of the input face.

The model will later use a decoder to read and reproduce that same vector information at the requested scale.

The model decoder cannot reproduce the image with greater fidelity than the original, because the original facial topography data was too sparse, and because there is usually a hard limit on input resolution for the extracted faces.

However, a GAN-trained system, or an adapted encoder/decoder configuration, can examine and explore the characteristics of high-resolution output from other high-resolution images, and use the vector/bitmap relationships it observes as a guide to making a detailed face, even from a blurred or indistinct input image.

Though super-resolution modelling gives the impression of restoring detail based on similar paths from degraded to high-res faces, it uses the original 'degraded' material only as a broad template. The result is essentially a 'smart guess'.

For entertainment purposes, this method has a lot of potential. For law enforcement, however, such approaches are destined to be called into question irrespective of their success, since a miscalculation can create a convincing but non-existent person who may by chance resemble someone unrelated to the input image.

Bias takes a prominent place on the list of. Machine learning models are designed to extract averages and sub-sets out of massively diverse data inputs in order to identify trends, patterns and tendencies. But if a dataset contains a majority of data points with a high proportion of a single label or classification, such as ethnicity=white, it will begin to see that label as a 'baseline' and anything different to it as an 'outlier'.

Furthermore, it will be better at distinguishing between examples of the dominant classified type (where the similarities form a central clique and the differences within that clique have greater room to emerge), than between the outliers, who may show only radical differences among themselves — and who inevitably become defined by their relationship to the core group rather than their relationship to each other.

In our recent review of facial recognition software pros and cons, we described some of the major controversies of recent years regarding racial bias in machine learning, which include: 

• Amazon, IBM and Microsoft temporarily withdrawing the sale of their facial recognition technologies to law enforcement entities until new legislation defines clear parameters to prevent unfair bias in derived systems (and a recurrence of the public pressure which forced them to this point).
• A late 2019 study from NIST claiming that facial recognition systems deployed by US law enforcement departments have an ID error rate 10-100 times worse for African-American and Asian faces as opposed to Caucasian faces²⁰.
• Public outcry when it was revealed²¹ that the Federal Bureau of Investigation and Immigration and Customs Enforcement (ICE) transposed state driver’s license records into a facial recognition database without consent.

The popular solution for racial bias in facial recognition systems is to ensure datasets have a diverse ethnicity²². As scandals around bias have multiplied, a number of prominent companies have sought to redress this imbalance: IBM developed a more multi-ethnic million-face dataset in 2019²³, and later that year Google Contractors were even caught bribing students and homeless people of African-American origin to volunteer their faces for the Pixel 4 facial recognition model²⁴.

Well-resourced generic global facial recognition systems, such as Apple’s and Google's Face ID systems, can be well-served by such diversity; but localized applications (such as a face-based entry system to a Chinese research lab, or a security clearance database for a school in Zimbabwe) may need to devote limited model space to a narrower range of facial types in order to achieve precise IDs within the broad parameters of one or two racial face-types.

The limitations of model space are rarely addressed in the heated discussions around racial bias in face datasets, but they are a hard reality for low-latency AI-based ID systems that need to run locally rather than on the cloud; for facial recognition systems that operate in monocultures; and especially for models that must run on the limited resources of mobile-scale deep learning systems.

Part of the problem may lie with the common use of existing semi-trained models and unsuitable open-source face datasets such as CelebA, since these are frequently dominated by one racial type or another²⁵.

Unless the facial recognition development community reduces its dependence on semi-trained open-source templates, which often contain ineradicable low-level features, it may have no practical or safe way to take accountability for the software it produces.