I also think that what we are thinking when we utter "Dyson Sphere" is just a really cartoonish, simplified version of what it would actually be like. Despite the all correct problems that Phantom Hoover detects in the simple design we have in our heads, it's still true that the core principle of the Dyson Sphere is how to get the full power output of a Sun.
The only real question that makes the DS "impractible", "impossible" or "stupid" in my head is if there's a much better way to extract energy from the universe other than collecting it directly from stars. I'm not saying there is not, but if there isn't, then it would be built. It would probably not spin the way we think, it would probably be much more complex than we can even imagine, it would probably solve many "intractable" things with surprising efficiency and simplicity, etc.
Although I feel that any renderization of an actual Dyson Sphere in any sci-fi setting that I've seen never really captured the real scale of what it really means. ST:TNG's take on it is pityful, absolutely cartoonish (looks like a huge room with a strange light bulb at the center), without followup (I mean, for a civilization to have built up this thing millions of miles of radius long, it means this civ is by far the most advanced in ST universe, and still after the Enterprise goes away, not even one further mention of it is done. No one cares about any riches that can be extracted from such an advanced ruin! It's mind boggling).