There's many small caves outside of Kampot city on the road to Kep. They're really small, but for Cambodia, they're really big. There's a sign from the main road directing you to the caves, but you'll probably have to ask directions, unless you can read Khmer.
If you continue straight down the road, you'll end up in a rock quarry. You'll see the first caves in back of the quarry.
It's also interesting to see the quarry workers smash big rocks into small rocks, and the small rocks into smaller rocks. They make about $1.50 for a long day's work there. Stop by and say hello.
If you backtrack out of the quarry, you can turn right on the first road (about 10 meters outside the quarry) and it's another 5 KM 'till you can find some more caves. Here, they ask for some money to see the caves, usually a dollar. It's a nice ride out in the countryside, but bring yourself some food and water for the ride. (They have water, but no ice)
A few kilometers out of Kampot, towards Kep, you'll find a Buddhist Wat, Wat Phnom Sorsia, which has a few nice caves at the top of a small hill
Climbodia has cave and rock climbing tours like no other in Kampot or Cambodia.