An interesting dispute litigated in a Texas appellate court (partial dissent here) having to do with "a Mahr[, which] is an Islamic religious custom whereby the husband contracts to give the wife a sum of money, either at the time of the marriage or deferred in the event of a divorce." Some of the most interesting arguments -- "those regarding the Establishment Clause, public policy, and Islamic law" -- were found to be waived, and the decision rested on fairly technical questions. Still, it might give one a sense of the kinds of cases that are likely being litigated elsewhere, and are likely to be litigated in the future.
As I mentioned before, I think the right approach for the American legal system is to simply enforce these contracts as written, without regard to their religious character. Some such contracts might be unenforceable because they are supposedly substantively or procedurally "unconscionable" (a pretty hard standard to meet), or because they are somehow restrained by state family law, or because one of the parties can show duress under standard secular legal rules (again, a pretty hard standard to meet in the absence of express threats of illegal conduct). But generally speaking they should be enforceable, just as contracts are generally enforceable even when we think one party has less "bargaining power" (an ill-defined concept) than the other. We shouldn't nanny-state Muslims any more or any less than we nanny-state the Amish or evangelical Christians or Orthodox Jews or the secular.
I'd reject therefore any Establishment Clause challenges to the contracts, unless they by their terms call for theological judgment -- in which case the better solution would be for the contracts to expressly call for arbitration by tribunals that can make such theological judgments, and then for secular courts to generally enforce any arbitral property settlements or monetary awards. The one complicated question is what should be done if the contracts call for enforcement by arbitral tribunals that apply sex-, race-, or religion-discriminatory rules. It's possible that under the secular law having to do with enforcement of arbitration agreements, such discriminatory arbitration might be against public policy and thus the results of it might be unenforceable; I don't know what the rule ought to be. But again the basic principle should be to apply to Muslim contracting parties precisely the same rules that we apply to anyone else.