In today’s data-driven world, the ability to efficiently search and analyze vast amounts of information is crucial for many applications. With the proliferation of vector data, such as images, audio, and text embeddings, the need for fast and accurate similarity search has become increasingly important. Redis, a popular in-memory data structure store, provides native support for storing vectors and performing k-nearest neighbor searches using Lua scripts. However, when dealing with large-scale databases containing billions of vectors, the performance of traditional Redis k-NN searches becomes a bottleneck. This is where the RediSearch module comes into play, transforming Redis into a powerful vector database capable of performing approximate nearest neighbor searches in milliseconds. In this blog post, we will explore how RediSearch leverages advanced algorithms to index vector embeddings and enable lightning-fast vector similarity searches, revolutionizing the way we handle and search vector data within Redis.
Vlog
RediSearch Module
I could not install RediSearch module in my MAC. It seems a little bit problematic. You may need to run it with docker if you insist to use it on-prem. Instead, I will use Redis Enterprise Cloud as an environment in this experiment.
🙋♂️ You may consider to enroll my top-rated machine learning course on Udemy
We firstly need to create a subscription here. Its fixed plan is free and it comes with 30MB standard database which is fine for experimental purposes. Thereafter, we will create a database with Redis Stack type.
Then, we should see the summary of database as shown below. RediSearch must be available in advanced options. Besides, we will use the host and port in public endpoint later.
Besides, password of the default user will be seen in this console, too.
Python Interface
Our communication with Redis will be exclusively through Python. This includes tasks such as inserting vector embeddings and conducting searches. To achieve this, we will utilize the official Python interface for Redis. If you have not installed the package yet, you can easily install it using pip. Once installed, we will import the package along with relevant modules, as demonstrated below.
# !pip install redis import redis from redis.commands.search.field import VectorField, TagField from redis.commands.search.query import Query
Initializing Redis Client
After importing the redis package, we proceed to initialize the redis client with the assigned host and port obtained from the redis cloud. Subsequently, you will have the capability to ping the client and verify its connection.
r = redis.Redis(
host = "redis-17992.c73.us-east-1-2.ec2.cloud.redislabs.com",
port = 17992,
password = "*****",
ssl = False,
)
r.ping()
Vector Embeddings
In this experiment, we will utilize the deepface library for Python to represent facial images as vector embeddings. Additionally, we will employ its unit test items as our database. Specifically, the FaceNet model will be utilized for facial recognition, while MtCnn will be employed for face detection, serving as our chosen configurations.
from deepface import DeepFace
embeddings = []
for dirpath, dirnames, filenames in os.walk("deepface/tests/dataset/"):
for filename in filenames:
if ".jpg" in filename:
img_path = f"{dirpath}{filename}"
embedding = DeepFace.represent(
img_path=img_path,
model_name="Facenet",
detector_backend="mtcnn",
)[0]["embedding"]
embeddings.append((img_path, embedding))
FaceNet produces 128-dimensional vectors. We will need the number of dimensions later.
Storing Embeddings In Redis
After obtaining vector embeddings, we will proceed to store them in Redis. Vector embeddings are represented as Python lists consisting of 128 items, which need to be converted into byte arrays. Furthermore, we will create a dictionary to map arguments, which will include the vector embeddings. While it is possible to store additional metadata information within this mapping, for the purposes of this experiment, I will solely store the embeddings.
pipeline = r.pipeline(transaction=False)
for img_path, embedding in tqdm(embeddings):
key = img_path.split("/")[-1]
value = np.array(embedding).astype(np.float32).tobytes()
pipeline.hset(key, mapping = {"embedding": value})
# -------------------------------------
pipeline_results = pipeline.execute()
Herein, the number of deepface unit test items is limited. We can create synthetic data to force redis and see how robust it is on a large scale database. I will create 1M more vectors with 128 dimensions and store them all in redis in the following block.
min_value = -10
max_value = 10
size = 1000000
dims = 128
synthetic_vectors = np.random.uniform(min_value, max_value, size=(size, dims))
pipeline = r.pipeline(transaction=False)
for idx, synthetic_vector in enumerate(synthetic_vectors):
key = f"synthetic_{idx}.jpg"
value = np.array(synthetic_vector).astype(np.float32).tobytes()
pipeline.hset(key, mapping = {"embedding": value})
# -------------------------------------
pipeline_results = pipeline.execute()
Creating Index in Redis
After storing the vector embeddings in Redis, our next step is to create an index. This index functions similarly to those found in relational databases, allowing for expedited searches of similar vectors. Redis will construct a hierarchical decision tree in the background to efficiently organize the vectors. This method is referred to as the approximate nearest neighbor technique, with Redis employing the Hierarchical Navigable Small World algorithm (HNSW) for this purpose.
r.ft().create_index(
[
VectorField(
"embedding",
"HNSW",
{
"TYPE": "FLOAT32",
"DIM": 128,
"DISTANCE_METRIC": "L2",
},
)
]
)
The first argument of the vector field refers to the key of the mapping object where we store vector embeddings. The second argument specifies the algorithm we are using, which is Hierarchical Navigable Small World (HNSW). The third argument indicates the number of dimensions in our vector embeddings and the metric we will use to calculate similarities.
Query Vector
Next, we will represent an image that is not available in our facial database as vectors and perform a search within our database using an approximate nearest neighbor algorithm. Similarly, we will convert the embedding into byte array.
target_embedding = DeepFace.represent(
img_path="target.jpg",
model_name="Facenet",
detector_backend="mtcnn"
)[0]["embedding"]
query_vector = np.array(target_embedding).astype(np.float32).tobytes()
Query
Lastly, we will search the query vector in Redis using the following query. Regardless of the number of vectors stored in our key-value store, this operation will always be executed within milliseconds.
k = 3
base_query = f"*=>[KNN {k} @embedding $query_vector AS distance]"
query = Query(base_query).return_fields("distance").sort_by("distance").dialect(2)
results = r.ft().search(query, query_params={"query_vector": query_vector})
for idx, result in enumerate(results.docs):
print(
f"{idx + 1}th nearest neighbor is {result.id} with distance {result.distance}"
)
Visualizing the results helps to demonstrate that this system is functioning properly.
Conclusion
In conclusion, the RediSearch module in Redis revolutionizes the way we handle vector data by transforming Redis into a scalable and efficient vector database. With the ability to index vector embeddings and perform approximate nearest neighbor searches in milliseconds, RediSearch solves the challenge of searching and finding similar vectors in large-scale databases. By leveraging Python for implementation, we can seamlessly integrate RediSearch into our applications, unlocking the immense potential of fast vector similarity search. Redis with RediSearch empowers developers and data scientists to tackle complex problems and build intelligent systems that require rapid and accurate similarity search for vector data.
I pushed the source code of this study to GitHub. You can support this work if you star⭐ its repo.
Like this blog? Support me on Patreon
