I think you have a good idea but implementing that will be difficult to get an accurate reflection of the industry. No one gets to try all the gear, even the Magazine Test's do not. When the gear is "tested" really its just a bunch of riders(buddies of Editors,Paid in Kiteboarder Mags case etc) riding around on new gear. Some are better than you and me, some not. They form an opinion on a Kite,Bar and board in about a 1/2 an hour. In most cases they compare it to what they own, that is the benchmark.
I work in a shop, I get to ride most of the kites and boards we sell( I dont Test, the testing part was done by the people who designed it). When you come in to purchase one of our products, I can give you my experience with it. I always ride other brands we don't sell when I can. Thats when I would get my half hour to related to my benchmark kite or board.
Your idea is good, but how will it change anything? You can't try all the gear(and why would you want to?) so, you will get the same feedback we get now, industry reps, retailers(me), fan boys and people who just really like there kit. Read it, weed it, and go get the best gear. Set up a site I'll vote!
Oh, did I mention I have the best deal in Kitesurfing?...lol