Encrypted Machine Learning: Naive Bayes on Encrypted Data

In the previous post, we ran logistic regression and linear SVM on encrypted data: the client encrypted its features, the server ran its private model on the ciphertexts, and the client learned only the predicted class. Those models want numeric features, like income or debt. A lot of real-world data isn’t numeric, though. Does the patient have a fever, yes or no? Is the cough dry or wet? Does the email come from a known contact? For categorical data like this, the classic choice is naive Bayes: simple, fast, surprisingly accurate, and still the textbook spam filter.

A flat lay of red and blue transparent dice on a white background, perfect for game night themes by pexels

In this post, we’ll run naive Bayes on encrypted data with LightPHE in Python, Go, Java and TypeScript. A patient will get a diagnosis without revealing their symptoms, and an email will be classified without the filter reading it. As with hyperplane classifiers, the protocol comes from Bost et al. in Machine Learning Classification over Encrypted Data (NDSS 2015). And as we’ll see, there is a neat trick that turns naive Bayes into the same kind of problem we already solved.


🙋‍♂️ You may consider to enroll my top-rated cryptography course on Udemy

Public Key Cryptography From Scratch

Naive Bayes in a Nutshell

Naive Bayes picks the class that is most probable given the features. The “naive” part is assuming that features are independent given the class, so the probability of a class is just a product:

P(class | x_1, ..., x_n)  is proportional to  P(class) * P(x_1 | class) * ... * P(x_n | class)

taking logarithms turns the product into a sum, with the same winner:

score(class) = log P(class) + log P(x_1 | class) + ... + log P(x_n | class)
prediction   = argmax over classes of score(class)

Training is just counting. P(class) is the share of training samples in that class, and P(fever = yes | flu) is the share of flu patients who had a fever. The model is a set of probability tables, and those tables are what the clinic or the filtering service wants to keep secret.

The Trick: Table Lookup as a Dot Product

To score a class, the server has to look up log P(cough = dry | cold), but the value of the cough is encrypted. A server can’t use an encrypted value as a table index. What it can do is a dot product. So the client encrypts each categorical feature as a one-hot vector, e.g. a dry cough among (none, dry, wet) becomes [E(0), E(1), E(0)], and the server multiplies it with its row of log probabilities:

cough is dry:                     [0, 1, 0]                                (encrypted by the client)
log P(cough | cold):              [log 0.12, log 0.25, log 0.63]           (plain, server's own)

dot product:                      0 * log 0.12 + 1 * log 0.25 + 0 * log 0.63 = log 0.25
under encryption:                 E(0)^(log 0.12) * E(1)^(log 0.25) * E(0)^(log 0.63) = E(log 0.25)

The lookup becomes a weighted sum of encrypted bits with the server’s plain numbers as weights. Do this for every feature, add the log prior, and you get the encrypted score of a class. Look closely and you’ll notice something: over one-hot features, naive Bayes is a linear model, whose weights happen to be log probabilities. So it needs exactly the same machinery as logistic regression and linear SVM.

Required Homomorphic Features

1. Additive homomorphism and scalar multiplication. The encrypted table lookups above are multiplications of encrypted bits with plain constants, and the score is their sum. The server computes all class scores alone, with the public key.

2. Quantized log probabilities. Log probabilities are negative floats, and homomorphic encryption wants non-negative integers. So LightPHE shifts the log probabilities of each feature by their minimum across all classes. That adds the same constant to every class’s score, so it doesn’t change the winner. Then it scales them by 10^precision, 3 decimal digits by default, and rounds them. In our clinic model, the cold’s table for cough, [log 0.12, log 0.25, log 0.63], becomes [0, 734, 1658], and LightPHE works out how many bits the scores need, 13 in this case. The cost is that 3 decimal digits cannot separate two classes whose scores are closer than about 0.01, so increase the precision if your classes are often neck and neck.

3. No zero probabilities. If a value never occurred with a class in the training data, its probability is 0, and log 0 is minus infinity, which no cryptosystem can encrypt. LightPHE refuses such a model with Probabilities must be in (0, 1], got 0.0. Smooth zero probabilities, e.g. with Laplace smoothing. Laplace smoothing adds one to every count before dividing, which is standard practice for naive Bayes anyway: a single unseen word shouldn’t veto a class.

4. Encrypted argmax. The server ends up with one encrypted score per class, and must not simply send them to the client: the scores would reveal the server’s probability tables, query by query. Instead, it finds the greatest score under encryption with the interactive argmax of Damgard-Geisler-Kroigaard, with a few masked round trips to the client—3 for each class after the first one. We covered how it works in detail in the post on encrypted logistic regression and linear SVM. The client learns only the index of the winning class.

5. A small plaintext space with a cheap zero test. The comparisons inside the argmax check many blinded values for zero, which is only cheap when the plaintext space is small. That’s why the whole pipeline runs on Damgard-Geisler-Kroigaard, whose plaintext space is about 32 bits.

Which Algorithms Can Be Used, and Why

The answer is the same as for hyperplane classifiers: only Damgard-Geisler-Kroigaard. If you try another cryptosystem, e.g. Paillier, LightPHE stops you with does not support encrypted classification.

AlgorithmsEncrypted scoresEncrypted argmaxWhy
Damgard-Geisler-Kroigaard✅✅Additive with scalar multiplication, plus a small plaintext space, cheap zero test and secure comparison
Paillier, Damgard-Jurik, Exponential ElGamal, EC ElGamal and other additive schemes✅❌They can compute the one-hot dot products, but the comparisons of the argmax rely on a small plaintext space
RSA, ElGamal, Goldwasser-Micali and others❌❌Not additively homomorphic, so they cannot even add up the log probabilities

Example 1: A Diagnosis Without Revealing Symptoms

An online clinic has a naive Bayes model trained on its patient records, which tells cold, flu and allergy apart from 3 symptoms: fever (no, yes), cough (none, dry, wet) and fatigue (no, yes). For instance, 4 of its 5 flu patients had a fever, so with Laplace smoothing, P(fever = yes | flu) = (4 + 1) / (5 + 2) = 0.71. A patient’s app encrypts the symptoms, the clinic finds the most likely diagnosis, and only the patient learns it. We run three patients with different symptoms, so we see all three diagnoses.

A patient with a fever, a dry cough and fatigue gets flu, one with only a wet cough gets cold, and one without any symptoms gets allergy. In plain floats, the first patient’s scores for cold, flu and allergy are -4.88, -2.17 and -5.39, but the patient never sees them, only the winner. The clinic never sees the symptoms, and doesn’t even learn the diagnosis it computed. With 3 classes, each diagnosis takes 6 round trips. Java snippets use assert for brevity, so run them with -ea to enable assertions.

Example 2: A Spam Filter That Never Reads Your Email

This time, let’s train the model ourselves. A filtering service has counted its 60 ham and 40 spam emails by four features: whether they contain “free”, whether they contain “winner”, how many links they have (none, 1-2, 3+), and whether the sender is in the user’s contacts. One count is a problem: none of the 60 ham emails ever contained “winner”. Without smoothing, P(winner = yes | ham) = 0, and any email with “winner” could never be ham, no matter what else it says. So the service applies Laplace smoothing while turning counts into probabilities. Then a user’s email client encrypts the features of two emails, and learns whether each one is spam.





The first email says “free” and “winner”, has 3 or more links, and comes from a stranger: spam, by a wide margin, with scores of -10.35 for ham and -3.22 for spam. The second email also says “free”, but it has no links and comes from a known contact, so it’s ham. That’s naive Bayes at its best: no single word decides alone, the evidence adds up. And thanks to smoothing, even an email with “winner” can still be ham: from a known contact, without links and without “free”, it scores -5.82 for ham and -7.78 for spam. If you skip the smoothing, LightPHE refuses the model with the zero-probability error we saw earlier, which is a much better failure than a silently broken filter.

More Scenarios

  • Symptom checkers: A patient checks symptoms with an online clinic, whose diagnosis model was trained on years of patient records, so the patient learns the most likely diagnosis without revealing the symptoms, and the clinic doesn’t give away its model.
  • Spam and phishing filtering: An email client asks a filtering service about a message, and the service classifies the encrypted features, so it never reads the email, and spammers can’t query its probability tables.
  • Support ticket routing: A company routes a customer’s message to billing, technical or sales based on which keywords it contains, using a vendor’s model, without the vendor reading customer messages.
  • Credit risk tiers: A lender classifies an applicant from categorical attributes, such as employment type, home ownership and loan purpose, without seeing them before the applicant decides to apply.
  • Fraud screening: A payment processor scores a merchant’s transactions by merchant category, country and time of day, with a model trained across many merchants, without receiving the transactions.

Caveats

  • Categorical features only: Continuous values, such as body temperature, must be binned into categories first, e.g. normal, mild fever and high fever.
  • Close calls: With 3 decimal digits, classes whose scores are closer than about 0.01 may be swapped. Raise the precision if that matters, as long as the scores fit into about 30 bits.
  • The shape of the model is public: Both sides need to agree on the features and their categories, and the number of classes. Only the probabilities stay secret.
  • Labels leak a little: The client learns the predicted class, and with many queries, an attacker can approximate the model from labels alone. Rate-limit queries, as for any prediction API. Both sides are also assumed to follow the protocol honestly.

Conclusion

Naive Bayes on encrypted data comes down to one trick: encrypt each categorical feature as a one-hot vector, and the server’s table lookups become dot products with its log probabilities. From then on, naive Bayes is a linear model, and the recipe of encrypted logistic regression and linear SVM applies as is: additive homomorphism for the scores, quantization for the floats, and an encrypted argmax on Damgard-Geisler-Kroigaard to pick the winner. The only new requirement is that no probability may be zero, which Laplace smoothing fixes.

The patient gets a diagnosis without describing their symptoms to anyone, and the email gets classified without anyone reading it—in a few lines of Python, Go, Java or TypeScript.


Support this blog financially if you do like!

Buy me a coffee      Buy me a coffee


Leave a Reply