Commenter Luke faults the Webster's Dictionary of English Usage argument about times less and times more with this:

"Everyone" understands what I mean when I say "axe you a question," so presumably we can all start saying that in the courtroom and in business meetings, etc. with no problems, right? It's no different than this case. It would take "a good deal of effort" to not understand what I meant, after all.

The trouble with "axe" for "ask" isn't that the spelling is somehow "illogical." After all, everyone understands that the pronunciation "iern" means "iron"; that's perfectly proper. The pronunciation "kernel," in context, means "colonel"; that's proper, and we wouldn't phonetically say "colonel" instead.

The difference between "iern"/"kernel" and "axe" is that "axe"-for-"ask" has not established itself well enough in educated speech. Using it therefore makes people think of you as lower-class; that may not be fair, but there it is. To the extent that one can call certain usages "wrong," this one is "wrong," because it departs from standard usage among educated people -- not because it's somehow illogical.

"Times more" and "times less" is standard usage, from Gladstone on down, and has long been standard usage. Most educated English speakers, I suspect, aren't even aware of the controversy over it (unlike with "axe"). So I see no justification for describing it as "wrong" (as opposed to "inelegant" or "likely to cause annoyance among some listeners").

And, yes, if "axe" ever becomes common enough in educated speech, it will become right, just as "ice cream" and "iern" are now right.