Where does the phrase "do you catch my drift?" come from?

In trying to research your question, its been sort of tricky. Not nearly as easy as I first thought. So far I've found this:

"Anybody know what CATCH MY DRIFT means and were it comes from?

It means, follow my train of thought, or understand the direction my thoughts are leading you. Usually expressed as a conditional, as in "if you catch my drift..." to see if you're keeping up with the tour group (if you catch my drift.) Experience has shown that this phrase, and its close relatives, are used most often by stupid and/or contemptuous people who find it hard to believe that mere You can follow the dazzling intricacies of their logic. If you catch my drift, if you know what I'm saying, if you see what I mean. "
from http://www.phrases.org.uk/bulletin_board/1/messages/2440.html

And someone tried to extend it further in this post on the same site

"It's very unlikely that anyone could find out who said it first. The phrase uses this meaning of "drift": "General meaning or purport; tenor" (American Heritage Dictionary, 1969, sense 5b of "drift" as a noun; by the way, sense 5a is "A trend or general bearing; direction," and from that we can see how "drift" came to mean the meaning that a speaker is aiming at).
The earliest quotation in the Oxford English Dictionary for this sense of "drift" is dated 1526: "Harde it is . . . to perceyue the processe and dryfte of this treatyse.""
located at http://www.phrases.org.uk/bulletin_board/13/messages/274.html

Doing a bit more digging reveals this:
"if you catch/get my drift (informal, informal)
something that you say to suggest that you have left out information or your opinion from what you have just told someone. 'She always has to be the centre of attention, if you catch my drift.'
See also: catch, drift"

And that is about all I could dig up after a good half hour of google searching. If you interested in other word/phrase origins, this page offers some good links and tips http://www.ipl.org/div/pf/entry/48468

------
Author Edit
------

I came across this later on looking up the origin of another phrase... From http://www.etymonline.com/index.php?search=catch+my+drift&searchmode=none

drift
c.1300, lit. "a being driven" (of snow, etc.); not recorded in O.E., borrowed from O.N. or M.Du. drift, from P.Gmc. *driftiz, related to *dribanan "to drive." The verb is first attested c.1600. Sense of "what one is getting at" is from 1526. Drifter is first recorded 1864, as a mining term; meaning "a man following an aimless way of life" is from 1908. Driftwood first recorded 1633.

catch (v.)
c.1205, from Anglo-Fr. cachier "catch, capture" (animals), from V.L. *captiare "try to seize, chase," freq. of L. capere "to take, hold" (see capable). Sense shifted from original meaning of "chase, hunt." Past tense form caught is rare instance of Eng. strong verb with Fr. origin, probably infl. by latch, the cognate native verb, which this word replaced. Noun meaning "that which is caught or worth catching" (especially of spouses) is from 1596. Catchy was a colloquial word in 1831. To catch on "apprehend" is 1884, Amer.Eng. colloquial. To catch (someone's) eye is first attested 1813, in Jane Austen. Catchword (1730) was originally the first word of the following page inserted at the right-hand lower corner of each page of a book; extended to "word caught up and repeated" (especially in the political sense) by 1795. Catch as catch can first attested 1393.

In Drift it says "Sense of "what one is getting at" is from 1526." and in Catch it Says "Noun meaning "that which is caught or worth catching" (especially of spouses) is from 1596." So one could infer that someone as early as the 1500's could have coined this term.

Hope this helps.