Facebook’s facial recognition will one day find you, even while facing away

Facebook researchers have unveiled new research that allows for faces to be more easily recognized based on other contextual information, such as hair style, clothing, and body shape.

The research, which was published on arXiv.org in January 2015, was presented at a conference in Boston earlier this month and first reported by New Scientist on Monday.

According to the researchers, the system, dubbed Pose Invariant PErson Recognition (PIPER), is accurate 83 percent of the time—far higher than the current state-of-the-art, which primarily requires clear, full frontal photos to work well.

As the authors write:

While each poselet is not as powerful as a custom designed face recognizer, it leverages weak signals from specific pose pattern that is hard to capture otherwise. By combining their predictions we accumulate the subtle discriminative information coming from each part into a robust pose-independent person recognition to the state-of-art face recognition system by using deep convolutional nets. Most of the existing face recognition systems require constrained setting of frontal faces and explicit 3D face alignments or facial keypoint localizations. Some other works have addressed robust face recognition systems to deal with varying illumination, occlusion and disguise. Due to our unconstrained setting, most of conventional face recognition systems have limited performance on our dataset.

But the technology probably won't hit the social networking site in the near future.

"This is so far, experimental, long-term research," Ari Entin, a Facebook spokesman, told Ars. "I don't think this is something that we would see any time soon."

Ning Zhang, the paper’s lead author and a University of California, Berkeley doctoral student, did not immediately respond to Ars’ request for comment.

Facebook has long been at the forefront of such research, which has come with some controversies.

Notably, Facebook was sued in April 2015 by a Chicago man, alleging that he and others in Illinois had their rights violated by Facebook as they did not give their express permission for facial recognition. (Texas also has a similar law on the books.) In 2012, German authorities ordered the company to destroy its facial recognition database, finding that it was in violation of German law.

Pushing frontiers

Facial recognition experts note that this kind of research is very much at the forefront of the field.

"It is an excellent piece of research that is pushing the frontiers of face recognition in unconstrained settings," Anil Jain, a professor at Michigan State University, told Ars by e-mail. "The focus and challenge (both for the government and social media companies) now has shifted towards unconstrained face recognition in unconstrained settings and uncooperative subjects. By this I mean, faces captured from surveillance cameras or photos posted on the social media sites."

"To work in this space, we cannot rely on traditional face recognition algorithms which relied on ‘face alignment’ based on locations of the two eyes," he continued. "In unconstrained imagery, eyes may not be visible, e.g., person wearing sunglasses or face is occluded. So, recognition by parts instead of recognition by ‘holistic’ method is needed. This requires that we train a face recognition system with a very large number of images per subject. That explains the approach by Facebook."

Facial recognition has been rapidly expanding in recent years both in commercial products, as well as by law enforcement.

Earlier this month, after more than a year of discussions, all nine privacy advocates have stormed out of a government-organized "multi-stakeholder process" to sort out details around the best practices for facial recognition technology. The sticking point was that corporations apparently refused to concede that there was any scenario during which a person’s consent to scan their face was needed.

In January 2015, Robert Morgester, the senior assistant attorney general and the head of the state’s eCrime Unit, told a group in San Francisco that the "possibilities are limitless" when it comes to using facial recognition to solve crimes.

In September 2014, the FBI announced that its facial recognition project that stores millions of mug shots and other photos is out of the pilot stage and is at "full operational capability."

"Researchers have been working on improving face recognition algorithms," Jennifer Lynch, an attorney at the Electronic Frontier Foundation, told Ars.

"This seems pretty advanced," said Lynch. "It's interesting that they're designing their algorithm to include cues that humans use to identify people: how you stand, how you style your hair. It's not surprising that this is where facial recognition is going. I’m sure it’s not just Facebook that wants to solve this facial recognition problem, but law enforcement would love to be able to identify people from the side or from above. Facebook has the money to throw at this and so do big government contractors."