Imagine someone puts together a site called gun-rights-vote-pledge.com. The site would urge voters to promise to vote for Gubernatorial Candidate X if this candidate publicly pledges to support gun rights (assume the term is suitably defined, for instance by focusing on one particular issue).
Voters who agree would record their promises, either on the site or in e-mail to the campaign. And the site would especially try to persuade voters who usually aren't single-issue voters, and who might have normally voted against candidate X because of his stands on other issues, but who are willing to pledge a single-issue vote just this once (perhaps to better demonstrate the power of pro-gun-rights voters).
I take it that this site would be quite proper, and perfectly constitutional, even though it solicits a deal: a pledge of citizen votes for a pledge of politician votes. Offering the candidate $10,000 (assume it's for his pocket, not even a campaign contribution) to vote for gun rights: a criminal bribe. Offering voters money to vote Candidate X: a criminal bribe. But offering the candidate valuable votes, and in exchange offering the voters valuable political victories: constitutionally protected.
This is further evidence, I think, that swapping political acts for political acts is often quite different from swapping political acts for money; the first should be legal and is like constitutionally protected, while the second is bribery. So the "vote swapping by voters is the same as vote buying" argument rests, it seems to me, on an unsound analogy.
Let's also look at it from another perspective. A legislator promising to vote a particular way if another legislator votes a particular way: ordinary log-rolling. A legislator promising to vote a particular way if voters elect him: ordinary and constitutionally protected (Brown v. Hartlage) campaign promises. Voters promising to vote for a legislator if the legislator promises to vote a particular way: the example above, which I think is quite proper. Voters promising to vote a particular way if other voters promise to vote a particular way: that's voteswap.com, and it seems to me hard to see why it should be a crime when the others are permissible and even constitutionally protected. If legislator-legislator, legislator-voter, and voter-legislator deals are permitted, why not voter-voter deals?