Why Most Rankings Are Wrong, And What We Do About It
A defense of publishing lists, written by people aware of the objections to them.
Ranking people is methodologically indefensible in several respects, and it is worth stating the objections plainly rather than pretending they do not apply.
The measurement problem
Influence is not a quantity. It cannot be measured on a single scale, and any ranking implicitly combines incommensurable things — capital controlled, decisions affected, institutions held — into an order that the underlying evidence does not support.
A list presenting a precise order therefore asserts more precision than exists.
The visibility problem
Rankings systematically over-represent the visible. People who speak publicly, whose organizations employ communications staff, and who are already covered by the press are easier to assess than people who are not.
The predictable result is that lists reflect who is documented as much as who is influential, and that bias operates against people and sectors that receive less coverage.
The circularity problem
Appearing on lists confers standing, which generates coverage, which supports appearing on future lists. Rankings can manufacture the reality they claim to describe.
The commercial problem
Rankings are commercially valuable to publishers. That creates an incentive to include the newsworthy and the well-connected, and to be responsive to people who might otherwise be advertisers. Readers are right to be suspicious of this, and the suspicion is earned by the industry rather than unfair to it.
What we do about it
We publish the criteria before the list. We order in bands rather than asserting false precision. We source every factual claim and publish no estimated figures. We separate list selection from commercial operations entirely, and we disclose paid features prominently wherever they appear.
Why publish at all
Because the alternative is not the absence of such judgements but their formation privately, among people who already know each other. A list published with its reasoning exposed can be argued with, corrected and held to its own stated standard.
That is a weaker claim than most rankings make for themselves, and it is one we can actually defend.
