From the federal prosecutor's brief in United States v. Translavina, 2003 WL 22716483 (signed by assistant U.S. Attorney Greg Addington):
In concluding that the movant in Duke had not been diligent in bringing his Rule 41(e) motion, the Court stated that "the victim of an alleged wrong [may not] postpone the running of the statute of limitations by willfully closing his eyes, ostrich-like, to a known probability that he has been injured, even if he is not certain."
[Footnote:] The analogy to ostrich behaviour reflects a myth which endures in popular culture. The popular conception that an ostrich will bury its head in the sand when confronted with danger or unpleasant stimuli has been thoroughly debunked. Various theories abound about the source of this ornithological myth, including the observation that ostriches will dig holes in the sand to bury their eggs and that they will occasionally place their heads on the ground when resting. The ostrich (Struthio camelus, of the group of flightless birds which includes emus, rheas, and kiwis) has been unfairly maligned as a symbol of cowardice and shirking of responsibilities. Once shorn of this erroneous negative association, the ostrich can take its proper place in the ornithological spectrum.
Curiously, the first use of the phrase "head in the sand" is in a poem by Elizabeth Barrett Browning in 1844. The text (stripped of its poetic context) is as follows: "this Ostrich age, which exposes its own eggs, and then hides its head in the sand..." Poems, Browning (1844). The myth endures through repetition in popular media as well as in obscure Circuit Court opinions.
All well and good, but what's with all this "behaviour"? Didn't we fight a couple of wars once upon a time about all that? Also, always be careful before saying something is the "first use of [a] phrase," even if you rely on the Oxford English Dictionary (which does give the 1844 reference as the earliest). Google Books -- which was introduced after the brief was filed -- points to several pre-Browning references.