Today's concept: how SVMs work
A Support Vector Machine can train on 10,000 examples, then throw 9,990 of them away and draw the exact same boundary. That is not a quirk. That is the design.
Picture two neighborhoods (two classes of data, say "spam" and "not spam") separated by a street. The SVM's job is to make that street as wide as it can, then run the decision boundary down the centerline. Houses deep inside either neighborhood are irrelevant. Only the ones on the property line, closest to the other side, decide where the road goes. Those are the support vectors, and they are the only points that survive into the finished model.
Why the widest? Thousands of lines separate the two classes perfectly on the data you already have. The one with the most clearance on both sides is the one most likely to still be right on the example you have not seen yet.
Two knobs you actually turn:
1) C, the soft margin. Real data overlaps, so you let some points sit on the road and charge a penalty for each. Low C: smoother boundary, tolerates mistakes. High C: contorts to get every training point right, and overfits. It is regularization wearing a different name.
2) The kernel. When no straight street separates them, you lift the data into a much higher dimensional space where one does, and the kernel trick gets you there without ever computing those coordinates. RBF is the usual default.
Where this lands in an agent stack: embeddings (the numeric fingerprints a model produces for text) are high dimensional, and you rarely have many labels. That is exactly the regime scikit-learn says SVMs hold up in, including when dimensions outnumber samples. So a linear SVM on embeddings is a cheap gatekeeper (prediction is one dot product, no GPU, no token bill): route a query, flag off-topic input, filter junk before it reaches your expensive model.
Not theory. Hugging Face's SetFit fine-tunes a sentence-embedding model on a handful of labeled examples, then trains a small head on those embeddings; logistic regression is the default, and their docs show swapping in scikit-learn's LinearSVC. LangChain ships an SVM retriever that fits a LinearSVC per query to rank documents instead of plain nearest-neighbor search.
Same instinct behind the model routing and cascade work everyone is doing right now: when a decision is really a boundary, do not spend a generation on it.
Quick check before you scroll: Why does an SVM only need a handful of "support vector" points to define its boundary, instead of using all the training data?
Full breakdown + the answer: frankduah.me/learnings/2026-08-16-how-svms-work
New here? I post a bite-size AI / ML concept like this every day, follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#SVM #AI #LLM #AIAgents #MachineLearning
The answer
Because the boundary is placed exactly halfway between the closest points of each class, moving a point that's already far from the edge doesn't change where that midpoint falls, so only the edge points matter.