This John Quiggin post at CT revisits the perennial question of why minor parties fare so poorly in the U.S.
I can’t model this in a convincing way, but looking at the comparative cases– much stronger two-party dominance in the US than in federal Australia or Canada or Germany, unitary France or (until recently) Britain, to say nothing of PR systems, it has always seemed to me that analyses in terms of either first-past-the-post voting or federalism/ centralism failed the at-first-blush test.
What marks the US as really distinctive in its political structures is complete presidentialism. The US has a separately elected unitary executive at both the federal and each state level; no other major developed liberal democracy does. The rest have prime ministers or their equivalents, or in France a pres-PM hybrid. And minor parties can credibly aspire to balance-tipping control of the determining house of the legislature, and hence to inclusion in coalition governments and a share of executive power, in a prime-ministerial system, even one elected on FPP. In the U.S., a share of executive power is effectively out of reach, and a share of federal executive power is completely out of reach.
And therefore the U.S., which seems like it should be a natural candidate for at least regional parties given its size and federal structure, doesn’t do what’s done in Canada or the UK and send regional parties to the national legislature. One occasionally gets a third-party governor, but never(*) a Senator and effectively never a Representative, even from the states with third-party governors. That’s anomalous, and I suspect has something to do with the strong executive-legislative separation and the impossibility of coalition governments.
(By the way: as far as I’m concerned an answer like “ballot access laws” is probably question-begging. Why do the two major parties have such overwhelming control that they can get away with cartelizing behavior and suffer no electoral consequences? It’s not as though parties elsewhere wouldn’t like to eliminate competitors.)
*Eugene rightly notes that James Buckley was elected to the Senate for one term (1971-77) as a Conservative, so “never” is a hair too strong.