I do understand your point, and I think I was kind of writing along the wrong lines in my post, but I was thinking more of the societal side of things than what constitutes normative behavior. I'm sure there are plenty of people in this world who fully support gay marriage yet would be perfectly willing to lynch someone who makes the statement, "I've thought about children in a sexual light." As you say, since pedophilia isn't presumably fundamentally different from a structural standpoint than heterosexuality, homosexuality, or bisexuality, the only difference there is the moral social construct against it due to the implicit inability of children to give consent, which is something I also completely agree with. However, by the same token, there seems to be a dissociation present in the public as a whole between pedophilia and other sexual preferences by virtue of that fundamental difference, to the point where any reasonable discussion about the topic as a whole tends to be overwhelming drowned out by knee-jerk cries of protectionism.
I've seen this come into play most often during discussions about "virtual CP" in the anime/manga community and the individuals who've come into legal trouble by possessing such material. The majority forumite opinion, and one I generally tend to agree with, is that, since such material doesn't involve the abuse of actual children in any way, then it should fall under protected expression, as attempts to prosecute it veer disturbingly into the "thought crime" realm. However, the argument I've seen expressed far more commonly by public figures, including such organizations as UNESCO, is that such material is "damaging" to children everywhere by its very nature. The general counter-argument tends to run that, without scientific evidence either way, such material could prove beneficial to real children, as it provides a harmless outlet for pedophilic tendencies that might otherwise be turned on them...but that very lack of evidence represents the problem. There aren't many, or even any, researchers who'd delve into such a study, no matter how beneficial its conclusions might be, simply because of the hue and cry that would be raised against them. Along the same exact lines, you're not going to see politicians debating whether housing restrictions against registered sex offenders represent cruel and unusual punishment, because they'd be torn apart by their constituents for "supporting kid-touchers." The fact that this sort of discussion can never take place in a reasonable setting is at odds with the assertion of pedophilia as a natural (though immoral) form of sexual expression, which is I think where the real problem lies.
(Note that I'm not in any way trying to draw pedophilia into the whole discussion about whether or not hate-crime legislation is beneficial; since the topic kind of turned this way, I thought I'd throw down a few thoughts myself.)