Shouldn't the requirement that the one being dragged be pre-selected be requirement enough? That way it won't happen through accidently clicking it, because its already clicked.
And, actually, I seem to remember this threshold of dragging-distance which must be exceeded before an object is considered being dragged. Maybe that would be a solution to accidental draggage...