Kampot's Central Market, downtown, is definitely the busiest place in town. Food, clothes, jewelry, fish, restaurants, household goods, pretty much everything you can buy in town is located in the market, and on the surrounding streets.
The market is at the intersection of the road to Phnom Penh, and the street from the new bridge. Morning, 8am to 10am is the busiest time. You can get a motodup (motorcycle taxi) from the riverside for about 1,000 Riel (25 cents U.S.) or walk about 5 to 10 minutes to get here.
Worth an hour walk through, and it's a great place for photography.