Understanding Similarity Thresholds: Tuning Face Recognition Accuracy

The Number That Decides Everything

Every face recognition system ultimately reduces to one comparison: a similarity score between two face embeddings, checked against a threshold. Score above the threshold → same person. Below → different person.

That single number determines whether your attendance system lets an impostor clock in, or locks out a legitimate employee who grew a beard. Yet most integration guides treat the threshold as a magic constant. This article explains what the score means and how to tune it deliberately.

Where Similarity Scores Come From

A face recognition model converts a face image into an embedding — a vector of numbers where geometric distance encodes identity. Comparing two embeddings typically uses:

• Cosine similarity — the angle between the two vectors; higher means more similar

• Euclidean distance — straight-line distance; lower means more similar

Both measure the same underlying idea: photos of the same person land close together in embedding space, photos of different people land far apart. The detection, recognition, and verification stages all build on this comparison.

Crucially, scores are model-specific. A 0.6 cosine similarity from one vendor's model is not comparable to a 0.6 from another. Never port a threshold between systems — calibrate against the system you actually use.

The Two Errors You Trade Between

Every threshold choice balances two failure modes:

• False Accept (FA) — two different people score above the threshold. The impostor gets in. Measured as False Accept Rate (FAR).

• False Reject (FR) — the same person scores below the threshold. The legitimate user is locked out. Measured as False Reject Rate (FRR).

Raising the threshold lowers FAR but raises FRR; lowering it does the opposite. There is no threshold that minimizes both — only a trade-off you choose based on what each error costs you:

• KYC / payments / account recovery: a false accept is fraud. Bias toward a strict threshold and accept more retries. Pair with liveness detection — a strict threshold does nothing against a high-quality photo spoof.

• Attendance / visitor convenience: a false reject means an annoyed human and a manual fallback. A moderate threshold is usually right.

• Watchlist alerting: humans review every hit, so you can run looser and let reviewers filter.

1:1 vs. 1:N Changes the Math

Thresholds behave differently between verification and identification:

• 1:1 verification (is this selfie the same person as this ID photo?) makes one comparison — FAR applies once.

• 1:N identification (who is this, among 10,000 registered faces?) effectively makes N comparisons. With a database of 10,000 faces, even a 0.01% per-comparison FAR yields roughly a 1-in-1 chance of some false match if you accept any score above threshold. This is why 1:N systems need stricter thresholds than 1:1 systems, and why they should return the best match rather than any match.

Practical consequence: as your face database grows, revisit your threshold. What worked at 500 enrollees may produce false matches at 50,000.

How to Calibrate a Threshold Empirically

Don't guess — measure with your own population and capture conditions:

Collect genuine pairs: multiple photos of the same people, captured the way production will capture them (same cameras, lighting, angles)

Collect impostor pairs: photos of different people, including similar-looking pairs (siblings, colleagues) if you can

Score every pair through your API — with ARSA, the validate_faces endpoint returns the similarity for any two images

Plot the two score distributions. Genuine pairs cluster high, impostor pairs cluster low; the overlap region is where errors live

Pick the threshold that achieves your target FAR, then check the FRR you're accepting is tolerable

Monitor in production: log scores (not images) so you can see drift and re-calibrate

Even 30–50 people with 3–4 photos each produces a far better threshold than any default.

What Moves Scores in Production

Scores drop — and false rejects rise — when capture conditions degrade:

• Lighting: strong backlight and harsh shadows are the top real-world score killers

• Pose: beyond ~30° head rotation, embeddings drift; guide users to face the camera

• Resolution: faces smaller than ~100px across lose discriminative detail

• Occlusion: masks, sunglasses, and hands on faces remove signal

• Time: enrollment photos age; re-enroll periodically or register multiple reference photos per person

Registering 2–4 photos per person (ARSA's register_face accepts up to four images) widens the genuine score distribution's floor — the single cheapest accuracy improvement available.

Practical Defaults and a Warning

If you need a starting point before calibration: start strict for security flows and moderate for convenience flows, then calibrate within your first weeks of real traffic. And treat any vendor's "99.9% accuracy" claim with suspicion until you know the FAR/FRR pair and the dataset behind it — our API selection guide covers how to benchmark vendors honestly.

Test It Yourself

The fastest way to build intuition is to score real pairs: grab a free ARSA API key, call validate_faces on photo pairs of your own team, and watch how lighting and pose move the numbers. Then wire the same logic into your app with our Python or Node.js tutorials.