How to Find False Negatives in Facial Recognition with Neo4j

Sefik Serengil

Current cutting-edge facial recognition models offer human-level accuracy. Still, we can improve facial recognition model accuracies if we represent classifications in a graph. In this post, we are going to find false negative classifications of facial recognition models with Neo4j graph database.

A false negative classification

Vlog

You can either watch the following video or continue to follow this tutorial. They both cover finding false negative classifications in facial recognition with neo4j graph database.

🙋‍♂️ You may consider to enroll my top-rated machine learning course on Udemy

Decision Trees for Machine Learning

False positive vs false negative

We have just focused on detecting false positives in facial recognition with Neo4j. False positives are mis-classifications verifying different persons as same person. On the other hand, false negatives classifies a same person pair as different persons. Finding false positive classifications is harder because it requires some observations and calculations. If two strongly connected clusters are connected to each other weakly, then this is a pattern for false positive. We calculated the betweenness centrality scores to find weak connection between strong cluster. However, finding false negatives is much easier. We are able to find false negatives with simple cypher queries in neo4j.

False negative vs false positive

Storing facial images as nodes

We will run same procedures in our previous post to store facial database in the graph. Basically, the following code block will handle moving facial images to neo4j with image name and embedding.

import os
from tqdm import tqdm
from deepface import DeepFace
from neo4j import GraphDatabase, basic_auth

#initialize neo4j connection
driver = GraphDatabase.driver("bolt://localhost:7687", auth=basic_auth("neo4j", "*****"))
session = driver.session()

# store facial image database in neo4j
with session.begin_transaction() as trx:
    for root, dirs, files in os.walk("../deepface/tests/dataset/"):
        for i, file in tqdm(enumerate(files)):
            if '.jpg' in file:
                img_path = root+file
                img_name = file[:-4]
                embedding = DeepFace.represent(img_path=img_path, 
                               model_name="Facenet", 
                               detector_backend="mtcnn")[0]["embedding"]
                statement = f"""
                    MERGE (f{i}:Face {{name: '{img_name}'}})
                    SET f{i}.embedding = {embedding}
                """
                trx.run(statement)
    trx.commit()

Creating edges

Thereafter, individual nodes will be created in neo4j without relationships. To create edges, we will run the following cypher statement in the neo4j side. This will create edges between nodes if the euclidean distance value is less than the threshold value of 10. This threshold value is pre-tuned value in deepface for Facenet model and euclidean distance pair.

 MATCH (p1:Face)
 MATCH (p2:Face)
 WHERE p1.name <> p2.name
 WITH p1, p2, gds.similarity.euclideanDistance(p1.embedding, p2.embedding) as distance
 WHERE distance < 10
 MERGE (p1)-[e:distance]-(p2)
 SET e.distance=distance

Then, we will have nodes and edges in the graph side.

Current Graph

A false negative case

In this graph, you can see an interesting pattern on the left-down side. When I visualize the images for this cluster, that would be more interesting. All images are belonging to Jack Dorsey. That is expected. When you ask img59 – img62 pair to deepface, it verifies this pair as same person. That is correct. On the other hand, if you test the img59-img16, img59-img61 and img59-img17 pairs, then deepface will fail. It says these are pairs of different persons but actually they are not!

A false negative case

SQL

If we store these relationships in a relational database, SQL-like queries require to write nested loops to find in-direct relationships. For example, we can find 2-depth relationships with the following SQL. If you want to find 3-depth relationships, then the query will become much complexer.

SELECT b.p2
(
    SELECT p2
    FROM Face
    WHERE p1 = "img59"
) a
left join Face b 
on a.p2 = b.p1
WHERE b.p2 != "img59"

Cypher

Luckily, we have Cypher queries and Neo4j! The following query will return the nodes having relation to img59 from 1 to 2. If you want to increase the depth-size, you just need to modify *1..2 to *1..3. So, it will not increase the complexity of the query.

MATCH p=(n:Face)-[r*1..2]-(m:Face)
WHERE n.name = "img59"
RETURN n.name, m.name
Variable length relationships

So, even though img59 is not connected to any of img16, img17, img61, we can retrieve its relationship to those nodes via img62. Besides, we can get it with a simple cypher query in Neo4j.



Conclusion

So, we have mentioned how to find false negative classifications in facial recognition experiments with Neo4j graph database. Even if relationship are stored in a relational database and we can retrieve in-direct relationship with SQL queries, it becomes more and more complex when the depth-size increased. On the other hand, we can do this task very easily in Neo4j with same cypher queries.

Like this blog? Support me on Patreon

Buy me a coffee