Across the River (West Side) the first big road to the right from either bridge, takes you to the Kampot Zoo (about 20 minute ride and $5 entrance fee), lions, tigers and bears (and maybe some monkeys), and the Kampot River rapids, Teuk Chhou, an informal park for eating, drinking, sleeping, and playing in the water. Take your own motorcycle or get a motorcycle taxi or Tuk Tuk for about $5-7 round trip.
Kampot is also known worldwide for it's Pepper Plantations (black, green, red and white table pepper) and locally for Durian farms (a giant, pointy fruit, weighing up to 12 kilograms). The Pepper from Kampot is the only food product in Cambodia exported worldwide, and has the Origin Specific label "Kampot Pepper". Pepper farm tours are a popular activity here.
There's also painball, zipline across the river, football (soccer), the secret lake, windsurfing, paddleboarding, kayaking, and day trips to Bokor Mountain.
Wat Phnom Sorsia is one of the caves about 10 kilometers outside of Kampot. The caves are small, and there is an ancient Buddhist shrine inside. You can find caves in random places in rice fields, if you turn off to the left, going from Kampot to Kep.
From Phnom Penh, a taxi or minibus for a 3 hour trip, about $5. From Kep, a half hour ride by car, tuk tuk or motorcycle. From SihanoukVille, a taxi for about $4/person, leaving from in front of the market, or $25 for the whole car for a 2 hour trip, or a 2 hour bus ride for about $7. Check the Sihanoukville site for departure times.
Now from Koh Kong, sometimes you can get a morning bus to Kampot for $10 and a 4 to 5 hour ride. From Vietnam, check out Andy's Oasis Bar in Ha Tien, Vietnam.
And don't forget the Train!!! Friday, Saturday, Sunday and Holidays, the Train runs between Phnom Penh, Kampot, and SihanoukVille!