Classification: Logistic Regression and Metrics That Do Not Lie

Classification predicts a category, but most classifiers first produce a number. Logistic regression, the standard baseline for yes-or-no problems, computes the same weighted sum as linear regression and then squashes it into the range 0 to 1 with the sigmoid function:

A score of $z = 0$ maps to 0.5, large positive scores approach 1, large negative ones approach 0. Each weight has a readable meaning: increasing feature $j$ by one unit multiplies the odds $p/(1-p)$ by $e^{w_j}$.

Key idea: The model's output is a probability-like score, not a decision. Turning 0.37 into "fraud" or "not fraud" is a separate step, and most evaluation mistakes come from treating the two as one.

Logistic regression is trained by minimising log loss, also called cross-entropy:

For each example only one term survives: $-\log p$ when the true label is 1, $-\log(1-p)$ when it is 0. The penalty grows without limit as a wrong prediction becomes more confident.

That shape is the point. Log loss rewards a model for being uncertain when it should be, which is why models trained on it, logistic regression included, tend to output probabilities you can use as probabilities. Accuracy cannot tell 0.51 from 0.99; log loss can.

The default rule, predict 1 when $p > 0.5$, silently assumes both mistakes cost the same. They rarely do. If the probabilities are trustworthy, the cheapest rule flags an example when the expected cost of ignoring it exceeds the expected cost of acting:

Suppose a missed fraud costs 500 and a manual review of a false alarm costs 5. The threshold is $5 / 505 \approx 0.01$: review anything with more than a 1% chance of fraud. For a cancer screen where a false alarm means a follow-up test and a miss can be fatal, the threshold goes lower still. For a spam filter, where losing a real email is the expensive error, it goes up.

In practice: Keep the model and the threshold separate in your code and your conversations. The data scientist owns the scores; the people who bear the costs of each mistake should own the cut-off.

Take 10,000 card transactions, of which 100 are fraudulent. A model at its chosen threshold flags 150 of them, and 80 of those are real fraud. Every prediction lands in one of four cells:

Every classification metric is some ratio of these four numbers. The trouble is that the 9,830 true negatives dwarf everything else, so any metric that counts them generously will look excellent almost regardless of how the model treats fraud.

Two ratios ignore the true negatives and so stay honest on rare events:

Precision answers "when the model raises an alarm, how often is it right?" Here, barely half the time, which is what the review team experiences. Recall (also called sensitivity, or true positive rate) answers "of the real cases, how many did we catch?"

The two trade against each other through the threshold. Lower it and you catch more fraud but raise more false alarms; raise it and alarms get more reliable while more fraud slips through. F1 summarises both as their harmonic mean:

Gotcha: F1 weights precision and recall equally, which is itself a cost assumption. When a miss costs a hundred times a false alarm, optimise the costed threshold, not F1.

Here are four common metrics computed on the same confusion matrix. They describe one model at one threshold.

Specificity, $TN / (TN + FP) = 9{,}830 / 9{,}900$, is the share of legitimate transactions correctly left alone. Like accuracy it is dominated by the huge negative class, so both sit above 99%. The metrics that concern the rare class, recall and precision, tell a less flattering story.

None of these numbers is wrong. The mistake is choosing the one that answers a question nobody asked. A fraud team's question is usually "of our alerts, how many are real, and how much fraud gets through", which is precision and recall.

Sweeping the threshold from high to low traces a curve, which summarises the model across every possible cut-off.

Our fraud model sits at a false positive rate of $70 / 9{,}900 = 0.7\%$ with a true positive rate of 80%, which looks superb on an ROC plot. Its precision of 53% does not.

Saito and Rehmsmeier (PLOS ONE, 2015) examined exactly this and concluded that on strongly imbalanced data the visual interpretability of ROC plots "can be deceptive", while precision-recall plots give an accurate picture of future classification performance, because they measure the fraction of true positives among positive predictions.

Fit on training data, pick the threshold on validation data, and report once on test data.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (average_precision_score, roc_auc_score,
                             precision_recall_curve, classification_report)

clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
clf.fit(X_train, y_train)

val_scores = clf.predict_proba(X_val)[:, 1]
print("ROC-AUC:", roc_auc_score(y_val, val_scores))
print("Average precision:", average_precision_score(y_val, val_scores))

# highest threshold that still catches at least 90% of positives
prec, rec, thr = precision_recall_curve(y_val, val_scores)
t = thr[rec[:-1] >= 0.90].max()

test_pred = clf.predict_proba(X_test)[:, 1] >= t
print(classification_report(y_test, test_pred, digits=3))

precision_recall_curve returns one more precision and recall value than thresholds, hence rec[:-1] to align them. Choosing the threshold on the test set instead would leak exactly as feature selection does.

The costed threshold assumed the probabilities mean what they say: of all transactions scored 0.3, about 30% should be fraud. That property is calibration, and high accuracy does not imply it.

Niculescu-Mizil and Caruana (ICML 2005) compared the probabilities of seven learning methods across seven problems. Neural nets and bagged decision trees gave the best-calibrated probabilities, while SVMs and boosted trees were poorly calibrated despite excellent accuracy and ROC area. Both of their correction methods are standard today:

• Platt scaling: fit a logistic curve mapping raw scores to probabilities.
• Isotonic regression: fit a free-form increasing step function; more flexible, needs more data.

from sklearn.calibration import CalibratedClassifierCV
cal = CalibratedClassifierCV(base_model, method="isotonic", cv=5).fit(X_train, y_train)

Try it: Plot a reliability diagram with sklearn.calibration.CalibrationDisplay: bin the predictions, plot mean predicted probability against the observed positive rate, and see how far it strays from the diagonal. The Brier score, the mean of $(p - y)^2$, summarises the same thing in one number.

Pick the metric from the decision the model supports, before looking at any results, and write the choice down.

Key idea: Choosing the metric after seeing which one flatters the model is a form of overfitting to the evaluation. Declaring it first is what makes the reported number mean something.