Naive Bayes for Text Classification

By the end of this lesson, you will be able to...

• compare competing text classes by combining a prior with token evidence;

• use log scores to classify a short message without multiplying tiny probabilities;

• explain what conditional independence and additive smoothing buy—and what they leave out.

Idea in one sentence: Naive Bayes classifies a message by asking which class would make its observed tokens most plausible under a deliberately simplified independence assumption.

Core Insight

A support inbox receives short messages such as “charged twice, need refund” and “video will not load.” A decision tree can ask a few local questions. But text contains thousands of sparse possible clues, and no single word needs to decide the class.

The tempting rule is to attach one decisive word to each category: refund means billing; video means technical. That works until a billing message says “video refund” or a technical message contains a quoted invoice number.

Naive Bayes uses a stronger but still inspectable idea:

start with how common each class is
-> ask how likely each observed token is in that class
-> combine the evidence
-> choose the class with the larger score

It is “naive” because it treats tokens as conditionally independent once the class is known. Language is not actually independent. The assumption is useful because it turns an unmanageable joint-language model into counts we can estimate from training messages.

A Tiny Inbox

For a teaching model, choose between two classes: billing and technical. Suppose the training inbox has 60 billing messages and 40 technical messages:

P(billing) = 0.60
P(technical) = 0.40

Those are priors: the class frequency before reading a new message. Now count tokens in each class after tokenization. The following probabilities are synthetic and simplified.

| Token | P(token | billing) | P(token | technical) | | --- | ---: | ---: | | refund | 0.20 | 0.01 | | charged | 0.15 | 0.01 | | video | 0.01 | 0.18 |

For the message charged refund, compare the two explanations:

score(billing)   ∝ P(billing) × P(charged | billing) × P(refund | billing)
                 = 0.60 × 0.15 × 0.20 = 0.018

score(technical) ∝ P(technical) × P(charged | technical) × P(refund | technical)
                 = 0.40 × 0.01 × 0.01 = 0.00004

The billing explanation has the larger score, so the classifier chooses billing. The symbol ∝ means “proportional to”: for choosing the larger class score, the same normalization factor would apply to both classes and does not change the winner.

This is Bayes' theorem made operational. A class prior is revised by evidence from the observed message.

Why the Formula Is Naive—and Useful

In full generality, we would need the probability of the whole token combination:

P(charged, refund | billing)

That quantity needs examples of every relevant combination. With a large vocabulary, most combinations are rare or unseen.

Naive Bayes replaces it with:

P(charged, refund | billing)
≈ P(charged | billing) × P(refund | billing)

This is the conditional-independence assumption. It is false in ordinary language: charged and refund tend to occur together. But it gives a workable model from token counts. The model does not claim the words are truly unrelated; it chooses to ignore their dependency in exchange for tractability.

The assumption works best when many individually useful clues point in the same direction. It is weak for negation, word order, long-range context, sarcasm, and meaning that depends on combinations rather than token counts.

Log Scores Keep the Calculation Usable

Real messages can contain dozens of tokens. Multiplying many probabilities such as 0.002 × 0.0004 × ... eventually produces numbers too small for reliable floating-point calculation.

Because logarithms preserve order, use a log score instead:

log score(class)
= log prior(class)
+ Σ log P(token | class)

For the same message, approximate values are:

-4.017 is larger than -10.126, so billing still wins. Adding logs is easier and numerically safer than multiplying probabilities; the classification has not changed.

Smoothing Prevents One New Word from Erasing a Class

Suppose a new billing message says refund duplicate. The token duplicate never appeared in the small billing training sample. Without smoothing, its estimated likelihood is zero:

P(duplicate | billing) = 0
-> billing score = 0

One unfamiliar word would erase all other billing evidence. Additive smoothing fixes this by adding a small count α before converting counts to probabilities:

P(token | class) = (token_count + α) / (all_class_token_counts + α × vocabulary_size)

With α = 1 (Laplace smoothing), unseen tokens receive a small nonzero likelihood. This does not make duplicate evidence for billing; it stops absence from the small training vocabulary from becoming impossible evidence against it.

Trade-off: Smoothing makes sparse text classification robust to unseen tokens, but it also changes every estimated likelihood and can dilute strong rare-token evidence. The smoothing strength is a hyperparameter to evaluate, not a universal constant.

Representation Changes the Meaning of Evidence

Multinomial Naive Bayes uses counts: a word that appears twice can contribute twice. It is a natural fit for bag-of-words count vectors. Bernoulli Naive Bayes uses presence or absence: the word contributes once if it appears at all.

Neither representation understands the sentence. Both discard word order. Choose based on the message format and validate the choice. A short alert-like message may behave differently from a long support thread.

The prior also matters most when the token evidence is weak. If billing messages are ten times as common as technical messages, an ambiguous one-token message can reasonably begin closer to billing. That is not a preference for billing; it is a statement about the training label distribution. A stale or biased label distribution can therefore bend results before any token is read.

Consider a token whose likelihood is the same in both classes:

P(update | billing) = 0.05
P(update | technical) = 0.05

The token contributes no comparison advantage. The prior decides the tie. This is useful because a classifier should not pretend that neutral evidence is decisive. It is also a boundary: class frequency should be checked against the current inbox, especially if a new product issue changes the mix of support requests.

One more representation consequence is document length. In a count model, repeated tokens contribute repeatedly. A very long quoted email can therefore have a different score from a short direct request with the same vocabulary. Tokenization, stop-word policy, duplicate text, and whether counts should be normalized are part of the model's representation boundary, not cleanup after classification.

Before accepting a text classifier, inspect a few high-scoring tokens per class and a few confident errors. This is not a proof that the model is sound, but it can reveal accidental artifacts such as ticket IDs, template language, or future-status words that will not exist when a new message arrives.

Keep that inspection separate from final evaluation: discoveries should lead to a new experiment, not silent tuning on the held-out test set.

What the Model Does Not Know

Naive Bayes can efficiently rank simple class explanations. It does not know that “not charged” reverses the meaning of charged, that a quoted message differs from a request, or that a label policy was biased.

It also does not guarantee calibrated probabilities merely because it is probabilistic. Evaluate held-out messages, inspect confusion patterns, and keep human review for consequences that need contextual judgment.

Boundary: The independence assumption improves tractability, but it costs semantic structure. The signal to watch is systematic error on phrases, negation, new vocabulary, or minority classes—not merely average training accuracy.

Check Your Understanding

Check 1: Why can the classifier add log likelihoods instead of multiply likelihoods?

Answer: Logarithm is increasing, so the largest product has the largest sum of logs. Adding logs also avoids numerical underflow.

Check 2: Does P(refund | billing) = 0.20 prove that refunds cause billing messages?

Answer: No. It is a token frequency conditioned on an existing label, not causal evidence.

Check 3: What does smoothing change when a class has never seen a token?

Answer: It replaces a zero estimated likelihood with a small nonzero one, so the class remains comparable rather than being eliminated automatically.

Practice: Compare Two Explanations

Use priors P(billing)=0.6, P(technical)=0.4, and these likelihoods:

For the message video refund, calculate the unnormalized score for each class and choose a label. Then name one reason this result should be checked on held-out messages.

Self-check: Billing: 0.6 × 0.01 × 0.20 = 0.0012. Technical: 0.4 × 0.18 × 0.01 = 0.00072. Billing wins in this toy model, though the result is close and the independence assumption may mishandle the unusual token combination.

Connections and Next Step

Naive Bayes is a model-family contrast: instead of a global score or local tree path, it accumulates token evidence under a stated assumption. The next lesson asks how to judge any classifier once it produces labels: which mistakes, metric, and threshold fit the decision?

Resources

• [TUTORIAL] scikit-learn: Naive Bayes — Focus: Bayes rule, conditional independence, text variants, and smoothing.
• [REFERENCE] scikit-learn: MultinomialNB — Focus: word-count features and additive smoothing parameter alpha.
• [BOOK/TUTORIAL] Introduction to Information Retrieval: Naive Bayes text classification — Focus: classic derivation and token-count classification.

Key Takeaways

1. Naive Bayes compares which class best explains observed tokens using a prior and token likelihoods.
2. Conditional independence is a simplifying assumption that trades language realism for tractable count-based learning.
3. Log scores preserve the winning class while avoiding tiny probability products.
4. Additive smoothing prevents unseen words from collapsing a class score, but its strength needs validation.