The VPN comparison industry is structurally broken. Affiliate revenue runs the rankings; opaque scoring runs the affiliate revenue. We built TheVPNMatrix.com to demonstrate that an evidence-based alternative is possible, and to document what that looks like.
This article explains the scoring system in enough detail that you could, in principle, reproduce it.
Five principles
The methodology is governed by five rules that override every individual scoring decision:
- Transparency. Every source is publicly accessible.
- Verifiability. Every claim can be independently checked.
- Reproducibility. The methodology can be replicated by others using the same sources.
- Independence. No pay-to-play; no scoring decision is influenced by affiliate relationships, and conflicts of interest reduce a record’s score.
- Timeliness. Evidence is dated, and stale evidence is downgraded by a fixed schedule, not by judgment.
These are not slogans. They are constraints on the source code that computes scores.
The Evidence Quality Score (EQS)
Every individual claim on the site — “ExpressVPN uses RAM-only servers”, “Surfshark is owned by Cayman Islands holding company X” — is backed by a record with an EQS, computed as follows:
EQS = Type + Grade + SQS + Ind + Scp + Rep + Age + COIEight components. Some additive, some subtractive. The full ranges:
Evidence Type (1–5)
- Primary (P, 5) — direct from the VPN provider (privacy policy, terms, audit report)
- Community-Replicable (CR, 4) — independent technical testing
- Secondary (S, 3) — reputable third-party analysis
- Tertiary (T, 2) — general information, summarised elsewhere
Evidence Grade (1–5)
A through E, awarded on completeness and rigor.
Source Quality Score (SQS, 1–5)
- 5 — academic, government, or verified independent
- 4 — reputable technology publication, established expert
- 3 — established technology blog
- 2 — general technology website
- 1 — unverified
Independence (Ind, 0–2)
- 2 — fully independent of the provider
- 1 — operationally independent, possibly affiliated
- 0 — affiliated or potentially biased
Scope (Scp, 0–2)
How comprehensive the analysis underlying the claim is.
Reproducibility (Rep, 0–2)
Whether someone else can re-run the test and check the result.
Timeliness (Age, 0 to -3)
- 0 — fresh (< 6 months)
- -1 — recent (6–12 months)
- -2 — older (12–24 months)
- -3 — stale (> 24 months)
Conflict of Interest (COI, 0 to -3)
- 0 — none disclosed
- -1 — soft conflict (e.g. disclosed affiliate)
- -3 — hard conflict (undisclosed relationship)
A perfect record scores 23. A typical strong record sits in the 14–18 band. Anything below 8 we treat as insufficient evidence and exclude from public scoring until corroborated.
What 28 criteria actually measure
The site evaluates each VPN provider across 28 criteria, grouped:
Trust & Jurisdiction (5) — country of incorporation, prior cooperation with state requests, ownership structure, court history, transparency reports.
Privacy Policy & Logs (5) — what is logged, retention windows, third-party data sharing, policy clarity, last update.
Security & Encryption (6) — protocol support, cipher suites, perfect forward secrecy, kill switch behaviour, DNS leak handling, RAM-only server architecture.
Audit & Independent Verification (4) — most recent audit, scope, auditor reputation, whether the report is public.
Performance & Reliability (3) — server coverage, throughput under standardised conditions, uptime.
Usability & Support (3) — clients across platforms, ease of secure defaults, support responsiveness on privacy-affecting questions.
Pricing & Value (2) — pricing transparency, payment options that preserve anonymity.
Each criterion has its own evidence record set. Most providers have 50–80 records on file; the largest have 200+.
The database
The evidence database is, at the time of writing, 3,469 records across 38 actively tracked providers. Each record carries the eight scoring components above plus:
provider— provider namecriterion— which of the 28 criteriasummary— one-sentence claimsource— title, URL, retrieval datelastAudited— when we last verified the source still resolves to the same contentnotes— methodology decisions, edge cases
Audit cadence is quarterly for primary sources, semi-annually for community-replicable, annually for the rest.
What this changes about the rankings
In affiliate-funded VPN comparisons, the variable that explains rank order best is the affiliate payout. In evidence-based ranking, rank order is a deterministic function of the database. If you disagree with a placement, you can examine the records that produced it. If you find a missing primary source, we’ll add it and the score will update.
Some providers fall further down our list than they do elsewhere on the internet. This is not because we dislike them. It is because the records we have don’t yet support a higher placement, and we don’t make up evidence to flatter a brand.
The wider point
VPNs are one product category. The same methodology can be applied to any market where consumers depend on technical claims they cannot themselves verify — password managers, secure messaging apps, hardware security keys, age-verification vendors, identity-proofing services.
The pattern is: write the criteria down, source every claim, compute the score from the sources, publish the database alongside the conclusion. If a competitor cannot or will not do this, that itself is information about the competitor.
This is the kind of work we do at Cosmo Codex. If you operate in a market where evidence-based comparison would change the conversation, talk to us.
TheVPNMatrix.com is operated by Cosmo Codex Ltd, Company No. 16627148. The methodology described here is implemented in TypeScript and runs on every page render; the source is in our private repository pending the next open-methodology release.