Login anomaly detection: why our 3 a.m. alarm was mostly wrong

Login anomaly detection based on a fixed 1 to 5 a.m. window judges every account by the same clock, so it fires on everyone whose normal life happens at night. Judging each login against that account's own history cut false alarms from 78.9 to 3.8 per 1,000 logins in our synthetic lab and caught more of the planted bad logins.

The fixed rule in our lab

The lab (labs/sd-60/run.mjs) generates 30 days of logins for 200 synthetic users with a fixed seed. Each user has a home country, usual hours and one or two devices, and ordinary life is mixed in: about 10% work night shifts, about 25% take a trip abroad, about 15% get a new phone mid-month, and 3% of honest logins happen at a random hour. Days 1 to 7 only build history. On days 8 to 30 it scores 8,457 logins, among which we planted 20 bad ones: 12 loud (about 3 a.m., new country, new device), 3 foreign-day (new country and device at the user's normal hour), 3 quiet (only the device is new) and 2 mimic (home country, usual hour, using an already-known device ID).

Hours are measured in each account's home time zone. The fixed rule alerts on any login from 1:00 to 4:59.

Detector	Bad logins caught	False alarms	Per 1,000 logins	Alerts that were real
Fixed 1 to 5 a.m. rule	13 of 20	667	78.9	13 of 680 (1.9%)
Per-user baseline, alert at 4	15 of 20	32	3.8	15 of 47 (31.9%)
Per-user baseline, alert at 2	19 of 20	985	116.5	1.9%

The rule caught all 12 loud attacks and one mimic login that happened to fall at 1:41 a.m. It missed every foreign-day and quiet login, because those came at the user's normal hour.

Why most alerts were wrong

Two things push the rule's precision (the share of alerts that are real) down to 1.9%.

The first is the base rate. Only 20 of 8,457 scored logins were bad, about 0.24%. When the thing you hunt is that rare, even a small false positive rate across the honest 99.76% outnumbers the true hits. Precision is caught / (caught + false alarms), so 13 / (13 + 667) = 1.9%.

The second is that the rule's slice of the clock is crowded with honest people. Of its 667 false alarms, 442 (66.3%) came from night shift users. For them the slice is the middle of the working day.

Frequent false alarms can obscure genuine incidents, a risk the OWASP Logging Cheat Sheet names when it discusses which events to log. Here the 13 real alerts sit in a queue of 680. NIST SP 800-92 covers how to plan log management, including the review work people do on the logs.

How per-user login anomaly detection scores a login

The baseline keeps a record per account, which the video calls a habit card: past login hours, countries seen with counts, devices seen, and the last login's country and time. Each new login is scored against that record only:

# score per login (labs/sd-60/run.mjs, weights set by hand)
unusual hour         +2   under 5% of past logins within 1 hour of this time
                           (circular distance, so 23:30 and 00:15 are 45 min apart;
                            only scored after 5 past logins)
new country          +3   country never seen for this account
new device           +2   device never seen for this account
rapid country change +3   country differs from the previous login's, it is not
                           the account's usual country, and under 8 hours passed
alert when score >= 4

Usual country means the country with the most previous logins. The rule only checks for a country change and never computes travel speed, so nearby borders and VPN exits can trigger it.

Two real rows from the lab show the arithmetic. A quiet attack at 17:09 from the home country on a new device scored 2 (new device only) and raised nothing. A loud one at 04:15 from a new country on a new device scored 5 and alerted. It was not flagged for the hour: that user's history already had logins near 4 a.m.

Every login joins the history, including bad ones, because the system cannot tell them apart when they arrive. This lab retains all history and scans every previous hour value on each login, so scoring takes O(n) time for n previous logins. A production design needs bounded retention and a defined update policy.

The baseline's 32 false alarms were 29 travellers (mostly the first login abroad) and 3 new phones. Only 5 involved night shift users, against 442 for the fixed rule. MITRE files logins with stolen credentials under T1078 Valid Accounts.

The threshold is a dial

When we lowered the alert line from 4 points to 2, recall (the share of planted bad logins caught) rose from 75% to 95% and precision fell from 31.9% to 1.9%. The lower line caught 19 of 20 bad logins, the 3 quiet ones and a mimic included, but it also flagged every login with a new device or an unusual hour on its own: 985 false alarms, 116.5 per 1,000 logins, more than the fixed rule's 78.9.

That is the usual precision versus recall trade-off (Wikipedia has the definitions). Pick it from the cost of a missed account takeover and the hours your team can spend per alert, and measure both after every change.

What these settings missed

Four planted logins got past both the fixed rule and the baseline at threshold 4: the 3 quiet ones and 1 mimic. A login from the user's own country at the user's usual hour, with only a new device, scores 2. The two mimic logins used a device ID the account already had; one scored 2 at 18:48 and the other 0 at 01:41.

Requiring multi-factor authentication on a new device adds protection against stolen-password logins (OWASP Authentication Cheat Sheet). Reuse of a session that is already signed in may produce no login event at all. Our lab tests neither case. Session reuse needs session controls: monitoring session activity, short session lifetimes, and reauthentication before sensitive actions such as changing the email address or creating an API token. The OWASP Session Management Cheat Sheet explains the limits of binding a session to device properties, and a compromised device passes that check anyway. Our PHP security checklist shows the session basics in code.

Poisoning is another gap. In this lab one login makes a country and device "seen"; repeated use is only needed to change the most frequent country, which then also turns off the rapid country change signal for that country.

Run the lab

cd labs/sd-60
node run.mjs          # prints the table above and writes three CSV files
node chart-data.mjs   # same run, plus per-hour counts used for the video's charts

It needs Node.js 22 and no packages or network, and it finishes in under a second. The CSVs list every alert with its score and reasons, and every planted login with whether each detector caught it.

What we saw when we ran it

We ran node run.mjs inside labs/sd-60 on a Mac on 2026-11-06. It printed one line per detector, then a summary of the bad logins missed by both, then the CSV files it saved, and last a RESULT summary.

Our lab run, with each detector's catches, misses and false alarms on 8457 scored logins.

The fixed 1 to 5 a.m. rule caught 13 of the 20 planted bad logins and raised 667 false alarms, 78.9 per 1,000 logins. The per-user baseline at score >= 4 caught 15 and raised 32, or 3.8 per 1,000 (both rates rounded to one decimal place). The line that surprised us was the night shift count: 442 of the fixed rule's false alarms landed on night shift users, against 5 for the baseline. Most of its alerts went to logins our synthetic data labelled as not bad.

In this synthetic run, the time window mostly flagged the users we set up to work nights. Scoring each login against the account's own habits gave us far fewer wrong alerts, though 4 bad logins still got past both detectors.

MITRE ATT&CK's official page for the Valid Accounts technique.

This page confirms that MITRE files logins made with stolen credentials under Valid Accounts.

Our take

A fixed time window is quick to write and easy to explain, and it did catch every loud attack. Its cost was the flood: 667 wrong alerts for 13 right ones, most of them on people doing their normal jobs. Scoring each login against the account's own habits kept the loud catches, added the three foreign-day ones, and cut false alarms about twentyfold in our data. The baseline at threshold 4 missed both mimic logins; the fixed rule caught one because it occurred inside its time window. Treat behaviour scoring as one layer next to MFA and session controls, the same layering we use for AI agents that read untrusted text. Our numbers come from synthetic data with an attack mix and weights we chose; they show the shape of the trade-off, not rates you would see in production. Our fake password leak test also uses synthetic inputs.

This article describes general monitoring patterns, public documentation and our own synthetic lab. It does not describe any company's private systems, and the diagrams are conceptual. Every number comes from labs/sd-60 (seed 60, 200 synthetic users, 30 days, our own attack mix and hand-set weights), run on one Apple M2 Pro with Node v22.13.1. No attack technique is shown.