What One Row Means in a VPN Dataset, and Why It Matters

Ask what one row of a VPN dataset represents and the answers multiply: a company, a listener, a public exit address, a reachability result. Each is legitimate, and mixing them without labels produces a total that sounds precise and means nothing. The unit is a design decision, and it has to be made before the first count. Most of the confusion in published VPN statistics comes from skipping that step.

A Row Is a Claim, Not a Thing

Every row asserts something about the world, and the assertion has a subject. Naming the subject is what makes the row comparable to another row.

A provider row asserts that an organisation offers a service. An endpoint row asserts that a particular address and port speak a protocol. An exit row asserts that traffic left from an address at a time. A measurement row asserts that a probe from somewhere observed a result. Four subjects, four schemas, and a dataset that mixes them without a discriminator cannot be aggregated at all.

The discriminator is the cheapest part of the schema and the one most often omitted.

Pick the Unit Before the Count

Choose the unit that answers the question you actually have. Counting endpoints answers how many listeners were advertised; counting providers answers how many organisations were involved; counting exit addresses answers how many apparent origins exist.

Those numbers are not interchangeable, and the differences are large. One provider can operate thousands of addresses, and one address can host several protocol listeners for different services. A count of addresses is therefore neither an upper nor a lower bound on a count of providers. State the unit in the same sentence as the number, every time. A number without a unit is an opinion with a decimal point.

A public directory supplies advertised participants in that service. It is not an inventory of every commercial, organisational and private deployment in the world, and treating it as one inflates every derived statistic.

Provenance Belongs in the Schema

A row without provenance cannot be rechecked. Record the source record, the collection time with its timezone, the observation method, and a last-seen status.

Keep advertised attributes separate from observed ones, because they carry different weight. A directory's country label is an advertised claim, and it is not proof of a server's physical location. An IP allocation identifies a network block, not every tenant inside it, and not the current operator of an application on top. Two columns and a note beat one merged field every time.

{"endpoint": "203.0.113.10:1194/udp", "unit": "endpoint", "advertised_country": "NL",
 "observed_at": "2026-11-03T02:15:00Z", "method": "udp_probe", "last_seen": "2026-11-03"}

A single JSON line makes the schema concrete and is easier to argue about than a paragraph of description. Note that the timestamp carries a timezone and the method is named rather than implied.

Deduplicate According to the Question

Duplicates depend on the unit. The same address with two protocol listeners is one endpoint pair and two services, and collapsing it silently changes the count.

Deduplicate at the level the question asks about, and record how many raw rows collapsed into each kept row. That ratio is itself a finding, because a large collapse usually means the source publishes per-listener records. A provider operating many addresses produces the opposite pattern. Both are correct, and only the recorded ratio lets a reader tell them apart.

Then keep the raw rows. A deduplicated table is a view, and the ability to rebuild it from the raw input is what makes the result reproducible.

Publish the Coverage Statement

State how sources were selected, which regions or networks they cover, and what is deliberately absent.

Private corporate endpoints and unadvertised deployments never appear in public lists, so absence in a directory proves only that the directory did not list it. Check the source terms and redistribution permissions before publishing, and strip fields that were never needed. An endpoint study rarely requires subscriber identities, credentials or browsing histories. Publishing those would be a different project with a different justification.

wc -l vpn-endpoints.jsonl
jq -r '.unit' vpn-endpoints.jsonl | sort | uniq -c
jq -r '.advertised_country' vpn-endpoints.jsonl | sort | uniq -c | sort -rn | head

Those three commands give a row count, a unit breakdown and a coverage sketch by advertised country. Keep dated snapshots when the population changes, and distinguish stale from confirmed absence. A bounded dataset with a clear population beats an unsupported claim to contain every VPN on the internet.