I think you SHOULD be concerned about the possibility of a blown gasket, and the consequences are very serious. Though the risk may be small while riding, if you're in a hurry to get out (like most of us are), you may not notice a small tear happen while donning the suit. I'm pretty sure that's what happened here last Fall (or was it spring?). A relative newbie (unknown to some of us regulars) went out wearing a drysuit. The waves were choppy, about 3 ft. The wind died, kites came down and couldn't be relaunched. Most of the guys who didn't see the change in weather approaching managed to drift in without incident. The newbie in the drysuit had a seal fail, his drysuit filled up totally, and he nearly died of hypothermia while never more than 50 yards from the beach. He was too numb too quickly to realize his lines were getting all wrapped around him, and he was quickly so tangled up he couldn't swim in. A surfer swam him in. None of the kiters realized he was tangled or that his suit had failed.
A week later an experienced kiter had a lull catch him, too, but he got all tangled up while wrapping up his lines and had a very, very slow drift into the beach. He had no feeling in his feet or hands (cold water), and didn't realize he was getting tangled and couldn't feel the lines well enough to get himself untangled once it happened. His drysuit kept him the right side of hypothermic, but just barely.
So yes, drysuit seals can fail dramatically, and you need to be aware of the potential and take some safeguardsto prevent catastrophe.