From a Washington Foreign Press Center briefing Donald R. Kennon, Chief Historian at the United States Capitol Historical Society:

There's an interesting thing about John Adams and John Quincy Adams -- they were both very religious men, and John Quincy Adams were so religious that he is one of probably only one or two American presidents who did not take the Oath of Allegiance on a Bible. Now, it's kind of ironic that John Quincy Adams, being such a religious man, would not have used the Bible, but he said that he thought the Bible should be reserved for strictly religious purposes. So he took the Oath of Office on a book of laws, the Constitution and American laws. That's really what he was swearing allegiance to was the Constitution, so he didn't use the Bible.

Thanks to reader Pennywit for the pointer.