Whether it’s for facial recognition, content-based image retrieval, or recommendation systems, the ability to search for similar vectors quickly and accurately is crucial. The use of vector models has become ubiquitous, extending beyond images. For example, even in language processing, words and sentences are now commonly represented as vectors, powering sophisticated natural language models like ChatGPT. In this blog post, we delve into the fascinating realm of vector similarity search with the help of PostgreSQL, a powerful and versatile relational database system. Leveraging the capabilities of the DeepFace library for Python, we explore how to convert facial images into compact numerical embeddings, which can be efficiently stored and indexed in Postgres, ultimately allowing us to perform fast and accurate similarity searches using SQL queries.
Vlog
The Importance of Postgres
No sql technologies comes with array data type and embeddings are actually 1-dimensional arrays from the perspective of programming. However, relational databases do not have an array data type. We need to store each dimension value of an embedding in a column or row. You can read this blog post to understand how painful it is: Deep Face Recognition with Sqlite.
🙋♂️ You may consider to enroll my top-rated machine learning course on Udemy
On the other hand, Postgres is the only exception supporting array data type among Oracle, DB2, MySQL, MS SQL and SQLite. So, we can do what we can do in no sql technologies as is in Postgres, too.
Configuring Postgres
I installed postgres server in my mac via homebrew. You can skip this if you already have a postgres.
$ brew install postgresql $ brew services start postgresql $ psql postgres
Then, we will create our database, and authorize our user for this database.
$ psql postgres create database deepface; create user <USER> with encrypted password '<PASSWORD>'; grant all privileges on database deepface to <USER>;
Python Client
Once database is created, then we will use its python client to run data definition language (e.g. create) and data manipulation language (e.g. insert) statements. To install postgres client, we will install psycopg package.
pip install psycopg2
Initialize Postgres in Python
We will then import psycopg package, connect our local postgres server and create a cursor. We are going to use cursor’s execute command to run DDL and DML statements.
import psycopg2
conn = psycopg2.connect(
host="localhost",
port="5432",
database="deepface",
user=user,
password=password
)
cursor = conn.cursor()
Initialize Database Schema
We already created deepface database via comand prompt. We are going to store everything in embeddings table. Let’s drop this table if it is already available, and create it from scratch with name and embedding columns. Notice that embedding column is an array.
cursor.execute("drop table if exists embeddings;")
cursor.execute("create table embeddings (name varchar, embedding decimal[]);")
Generating our vector database
We are going to use deepface library for python to represent facial images as vector embeddings. Besides, we will use its unit test items as our vector database. In this experiment, we will use Facenet face recognition model and mtcnn face detector as a configuration.
import os
from deepface import DeepFace
representations = []
for dirpath, dirnames, filenames in os.walk("deepface/tests/dataset/"):
for filename in filenames:
img_path = f"{dirpath}{filename}"
if ".jpg" in img_path:
obj = DeepFace.represent(
img_path=img_path,
model_name="Facenet",
detector_backend="mtcnn"
)
embedding = obj[0]["embedding"]
representations.append((img_path, embedding))
Storing embeddings into database
We represented facial images as vectors. We also created the database table. We can store these embeddings into our database.
for img_path, embedding in representations:
statement = f"""
insert into
embeddings
(name, embedding)
values
('{img_path}', ARRAY{embedding});
"""
cursor.execute(statement)
# commit once
conn.commit()
In that way, we inserted 62 instances into our database.
Searching a target
Now, we are going to represent an image not available in our database as vector embeddings.
Target is going to be a 128-dimensional vector because FaceNet model generates 128D vectors.
import cv2
target_path = "target.jpg"
target_img = cv2.imread(target_path)
target = DeepFace.represent(
img_path=target_path,
model_name="Facenet",
detector_backend="mtcnn"
)[0]["embedding"]
Nearest Neighbour Search
Now, we will find the distance between each item in database with target. To find the Euclidean distance, we basically find the subtraction of each dimension first, find its squared values, find the sum for each dimension calculation and finally finding the squared root value. We need to use unnest function of postgres to expose each dimension, and then group by statement to find the sum of all dimension values.
According to the deepface, pre-tuned threshold of FaceNet model and Euclidean distance is 10. In other words, if the distance is greater than this value, then they are different persons.
So, we can use the following query to search nearest neigbours.
threshold = 10
query = f"""
select name, distance
from (
select name, sqrt(sum(distance)) as distance
from (
select name, pow(unnest(embedding) - unnest(ARRAY{target}), 2) as distance
from embeddings
) sq
group by name
) sq2
where distance < {threshold}
order by distance
"""
Then, we will pass this query to cursor to retrieve results.
cursor.execute(query)
rows = cursor.fetchall()
for img_path, distance in rows:
print(img_path, distance)
img = cv2.imread(img_path)
fig = plt.figure(figsize = (7, 7))
fig.add_subplot(1, 2, 1)
plt.imshow(target_img[:,:,::-1])
plt.axis("off")
fig.add_subplot(1, 2, 2)
plt.imshow(img[:,:,::-1])
plt.axis("off")
plt.show()
print("----------------")
Results
When we performed that query, postgres returns the following identities as a result. We can clearly say that our postgres implementation is working fine because it can find the images of Angelina Jolie.
Face verification
If your task requires to run face verification instead of face recognition, then just add a where clause into your query. If that query returns nothing, then it means face verification fails. On the other hand, if this query is returning a record, then it means face verification succeeded.
identity = "img1.jpg"
threshold = 10
query = f"""
select name, distance
from (
select name, sqrt(sum(distance)) as distance
from (
select name, pow(unnest(embedding) - unnest(ARRAY{target}), 2) as distance
from embeddings
where name = '{identity}'
) sq
group by name
) sq2
where distance < {threshold}
order by distance
"""
We directly accessed the identity with where clause. Notice that name should have an index and that query must be performed very fast. Then, find the distance for that identity is less than the pre-tuned threshold. This is going to return either one record or nothing.
Conclusion
In conclusion, the combination of k-nearest neighbour algorithm, the DeepFace library, and PostgreSQL presents a powerful framework for efficient and accurate vector similarity search. By representing facial images as numerical embeddings and utilizing SQL queries, we can perform fast and precise searches for nearest neighbors. This approach showcases the widespread adoption of vector models, not only in image analysis but also in various other domains such as natural language processing. The integration of deep learning techniques with relational databases like Postgres opens up new possibilities for scalable and effective image search systems. As the field continues to evolve, the utilization of vector models and their applications in image understanding and retrieval will undoubtedly play a significant role in advancing the capabilities of AI and data-driven technologies.
I pushed the source code of this study into GitHub. You can support this work if you star⭐ its repo.
Special thanks to Murat Yildirim to teach me array data type feature of Postgres.
Like this blog? Support me on Patreon
