If all you have is anecdotal evidence, how do you know it isn't just a fluke? A fluke is evidence of absolutely nothing.
And what if you have, say, ten people claiming that it works? Ten flukes? Possible. Now, a hundred? Sure, you can ask, out of how many? Still, at some point, you'll be looking at a considerable pile of "anecdotal evidence". If you're a good scientist, you'll start to ask "what is causing this"? If you're a Believer of Science (or a bad scientist, same thing, really), you'll dismiss them ex cathedra as "anecdotal evidence". Remember: the words "statistically significant", on their own, mean absolutely nothing
. Give me your p-value, your alpha and beta, name the test you're using, or go home. If you want, 100 cases out of million can be "statistically significant", or 50 out of 100 can be insignificant, just by fiddling with the significance level (the aforementioned alpha). Then you have the strength of your test (beta), which also influences the conclusion you arrive at. Not all tests are created equal, in some cases you can change your conclusion just by selecting a different mathematical method. And that's just the basics.
Statistical significance is a thing, of course, but it also depends on a surprising number of factors. It's not just a term to throw around. Also, outliers, no matter how few of them, have be caused by something
. Many a Nobel prize was awarded not for being the first to see something, but rather the first to realize what it could be (and promptly investigating further). Statistics are a way of accounting for too many factors to realistically account for in a deterministic way, but it's important to remember that outside of quantum effects, the world is deterministic to a very good precision.
For example, maybe an odd therapy has an obscure and uncommon prerequisite for working. If you test a random sample of people, you'll never get a definite result. However, if it occurs to you to check what the people who did
get better have in common, you might make a discovery. Even anecdotal evidence can help here, especially if you're trying to get by without running a staggeringly expensive
experiment that a full clinical trial is. It's another thing people often don't understand. You don't "just" run a clinical trial with an arbitrary sample size. Quite often, it's a choice between an inadequate sample size or not being able to afford to run it at all. Efficacy studies (a.k.a. phase III, what we're talking about here) are the worst of all.
Now, I'm actually an educated physicist (with some Nature exposure to boot), and I'm starting to get fed up with the nowadays scientifically induced idiocy - not related to this topic only. Filtering out what anecdotal evidence can hold factual observations is among the harder problems any practitioner of Science can face. But dismissing it entirely is more dangerous, because, sometimes the anecdotal evidence is correct. The result of this is doctors telling that you cannot possibly be feeling pain because we cannot find the source of it, so it must be psycho-somatic: "Would you like to see the psychiatrist next?"
I know what you're talking about. This, I feel is often related to the fact that lay people think "scientists are so smart that they know almost everything". We do have an enormous, nearly unimaginable amount of scientific knowledge, but what always amazes me is the amount of things we don't
know, which appears to be even greater. The laws of universe (and us, humans!) are just that complex. Disregarding this complexity is, I feel, one of the premier causes of both overt confidence in the current knowledge and mistrust for scientists and their discoveries.