Tech Notes

My notes on Statistics, Big Data, Cloud Computing, Cyber Security

Conditional Probability, Bayes Theorem, Naive Bayes Classifier

Both kNN and NaiveBayes are classification algorithms. Conceptually, kNN uses the idea of “nearness” to classify new entities. In kNN ‘nearness’ is modeled with ideas such as Euclidean Distance or Cosine Distance. By contrast, in NaiveBayes, the concept of ‘probability’ is used to classify new entities.

Before someone can understand and appreciate the nuances of Naive Bayes’, they need to know a couple of related concepts first, namely, the idea of Conditional Probability, and Bayes’ Rule. (If you are familiar with these concepts, skip to the section titled Getting to Naive Bayes’)

Conditional Probability in plain English: What is the probability that something will happen, given that something else has already happened.

Let’s say that there is some Outcome O. And some Evidence E. From the way these probabilities are defined: The Probability of having both the Outcome O and Evidence E is:

(Probabilty of O occuring) multiplied by the (Prob of E given that O happened)

One Example to understand Conditional Probability:

Let say we have a collection of US Senators. Senators could be Democrats or Republicans. They are also either male or female.

If we select one senator completely randomly, what is the probability that this person is a female Democrat? Conditional Probability can help us answer that.

Probability of (Democrat and Female Senator)= Prob(Senator is Democrat) multiplied by Conditional Probability of Being Female given that they are a Democrat.

P(Democrat & Female) = P(Democrat) x P(Female / Democrat)

We could compute the exact same thing, the reverse way:

P(Democrat & Female) = P(Female) x P(Democrat / Female)

Understanding Bayes Rule

Conceptually, this is a way to go from P(Evidence/ Known Outcome) to P(Outcome/Known Evidence). Often, we know how frequently some particular evidence is observed, given a known outcome. We have to use this known fact to compute the reverse, to compute the chance of that outcome happening, given the evidence.

P(Outcome given that we know some Evidence) = P(Evidence given that we know the Outcome) times Prob(Outcome), scaled by the P(Evidence)

The classic example to understand Bayes’ Rule:

Probability of Disease D given Test-positive =
Prob(Test is positive/Disease) *P(Disease)
 (scaled by) Prob(Testing Positive, with or without the disease)

Now, all this was just preamble, to get to Naive Bayes.

Getting to Naive Bayes

So far, we have talked only about one piece of evidence. In reality, we have to predict an outcome given multiple evidence. In that case, the math gets very complicated. To get around that complication, one approach is to ‘uncouple’ multiple pieces of evidence, and to treat each of piece of evidence as independent. This approach is why this is called naive Bayes.

P(Outcome/Multiple Evidence) =  P(Evidence1/Outcome) x P(Evidence2/outcome) x … x P(EvidenceN/outcome) x P(Outcome) scaled by P(Multiple Evidence) Many people choose to remember this as:

P(outcome/evidence) = P(Likelihood of Evidence) x Prior prob of outcome

Notice a few things about this equation:

If the Prob(evidence/outcome) is 1, then we are just multiplying by 1.
If the Prob(some particular evidence/outcome) is 0, then the whole prob. becomes 0. If you see contradicting evidence, we can rule out that outcome.
Since we divide everything by P(Evidence), we can even get away without calculating it.
The intuition behind multiplying by the prior is so that we give high probability to more common outcomes, and low probabilities to unlikely outcomes. These are also called base rates and they are a way to scale our predicted probabilities.
How to Apply Naive Bayes to Predict an Outcome?

Just run the formula above for each possible outcome. Since we are trying to classify, each outcome is called a class and it has a class label. Our job is to look at the evidence, to consider how likely it is to be this class or that class, and assign a label to each entity. Again, we take a very simple approach: The class that has the highest probability is declared the “winner” and that class label gets assigned to that combination of evidences.

Fruit Example

Let’s try it out on an example to increase our understanding: The OP asked for a ‘fruit’ identification example.

Let’s say that we have data on 1000 pieces of fruit. They happen to be Banana, Orange or some Other Fruit. We know 3 characteristics about each fruit:

Whether it is Long
Whether it is Sweet and
If its color is Yellow.
This is our ‘training set.’ We will use this to predict the type of any new fruit we encounter.

Type         Long | Not Long || Sweet | Not Sweet || Yellow |Not Yellow|Total
Banana      | 400 | 100      || 350   | 150       || 450    | 50       | 500
Orange      | 0   | 300      || 150   | 150       || 300    | 0        | 300
Other Fruit | 100 | 100      || 150   | 50        || 50     | 150      | 200
 Total      | 500 | 500      || 650   | 350       || 800    | 200      | 1000

We can pre-compute a lot of things about our fruit collection.
The so-called “Prior” probabilities. (If we didn’t know any of the fruit attributes, this would be our guess.) These are our base rates.

P(Banana) = 0.5 (500/1000)
 P(Orange) = 0.3
 P(Other Fruit) = 0.2

Probability of “Evidence”

p(Long) = 0.5
 P(Sweet) = 0.65
 P(Yellow) = 0.8

Probability of “Likelihood”

P(Long/Banana) = 0.8
 P(Long/Orange) = 0 [Oranges are never long in all the fruit we have seen.]
P(Yellow/Other Fruit) = 50/200 = 0.25
 P(Not Yellow/Other Fruit) = 0.75

Given a Fruit, how to classify it?

Let’s say that we are given the properties of an unknown fruit, and asked to classify it. We are told that the fruit is Long, Sweet and Yellow. Is it a Banana? Is it an Orange? Or Is it some Other Fruit?

We can simply run the numbers for each of the 3 outcomes, one by one. Then we choose the highest probability and ‘classify’ our unknown fruit as belonging to the class that had the highest probability based on our prior evidence (our 1000 fruit training set):

P(Banana/Long, Sweet and Yellow) = P(Long/Banana) p(Sweet/Banana).P(Yellow/Banana) x P(banana)
                                   P(Long). P(Sweet). P(Yellow)
                                   0.8 x 0.7 x 0.9 x 0.5
                                 = ______________________
                                 = 0.252/P(evidence)
P(Orange/Long, Sweet and Yellow) = 0
P(Other Fruit/Long, Sweet and Yellow) = P(Long/Other fruit) x P(Sweet/Other fruit) x P(Yellow/Other fruit) x P(Other Fruit)
 = (100/200 x 150/200 x 50/150 x 200/1000) / P(evidence)
 = 0.01875/P(evidence)

By an overwhelming margin (0.252 >> 0.01875), we classify this Sweet/Long/Yellow fruit as likely to be a Banana.

Why is Bayes Classifier so popular?

Look at what it eventually comes down to. Just some counting and multiplication. We can pre-compute all these terms, and so classifying becomes easy, quick and efficient.

Let P(evidence) = z. Now we quickly compute the following three quantities.

P(Banana/evidence) = 1/z * Prob(Banana) x Prob(E1/Banana).Prob(E2/Banana)…

P(Orange/evidence) = 1/z * Prob(Orange) x Prob(E1/Orange).Prob(E2/Orange)…

P(Other Fruit/evidence) = 1/z * Prob(Other) x Prob(E1/Other).Prob(E2/Other)…
Assign the class label of whichever is the highest number, and you are done.

Despite the name, NaiveBayes turns out to be excellent in certain applications. Text classification is one area where it really shines.

Disclaimer : These are my study notes – online – instead of on paper so that others can benefit. In the process I have used some pictures / content from other original authors. All sources / original content publishers are listed below and they deserve credit for their work. No copyright violation intended.

Referencesfor these notes :

A discussion thread in Stackoverflow

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: