Many locations have marked swimmer zones. Not all swimmer zones exist throughout the year. Make sure before you rig your gear, you know where these are. Normally they are well marked with buoys or signs.
In some areas violating and ignoring those zones will lead to bans of the sport or expensive fines by the officials.
In areas without swimmer zones, common sense is needed. You will find areas with many swimmers, but no official swimmer zones. Here common sense comes into play, staying as far away as possible.
You should also not be riding upwind of swimmer zones or swimmers, since if something happens, you will be dragged into those zones and endanger the swimmers.
Never be too close to swimmers. If you leave or enter a beach, always ride slow, specially when swimmers are close, so they don't get hurt.
Observe the swimmers before you go out or come in, since sometimes swimmers play in the shore break or dive, so you won't see them.
Normally, swimmers are very scared by kitesurfers, so don't scare them even more. And they don't know your skills, control, and where you can go/fly. Keep this in mind.