For the GPS selection, a lot of that depends on your wallet and the features you "must" have. I don't know of any GPS receivers that are better in this specific area, but I could be wrong. I like the Garmin eTrex series as it's a basic unit with a wide selection of models and prices. (I suggest any Garmin eTrex with an "x" at the end like the Vista HCx, which means it has a magnetic compass in it.) Other people enjoy the Delorme PN-40. If you have more money to spend, you can check out one of the fancy Garmin models like the Oregon, Colorado, or Dakota.
For more info on geocaching, check out
http://www.geocaching.com/about/ and if you have further questions, feel free to ask. That link also takes you to the website where most geocachers get their caching information.
Anyone else can feel free to chime in and correct me or add to what I've said
