Lots of heavy questions here, but I'm in a writing mood, so let's see what happens.
1) Like the previous responses to this, I think a mix of innovative ideas and more traditional gameplay is usually the best way to go. I haven't played BP3 yet (it's on my increasingly-long list, I swear!

), so I can't speak for what the team did there, but I could see how having brand-new mechanics thrown at you in every single mission could produce some mental overload. I know I only tried that capship-command mission from WiH once, and there was so much info being thrown at me I had no idea what I was doing, so I never went back to it. I've always liked the design philosophy that Valve has mentioned in a few of their gameplay commentaries: first introduce a new mechanic, then let the player mess around with it in a fairly-controlled environment, then turn them loose. I think more standard fare can serve well as a buffer between the segments that introduce more radical innovation, since it gives the player time to incorporate the new elements into their playstyle. And to touch on the last part, while I do like to see campaign designers trying new things, I don't think it's absolutely necessary: even today, it's completely possible to make a bog-standard retail campaign that's still a lot of fun.
2) I think NGTM-1R gets it right for me too. More traditional retail-style campaigns are very much a first-person experience for me: I'm Alpha 1, doing awesome stuff and blowing up Shivans all over the place, and my actions generally reflect what I feel like I'd do in that situation. I can still feel immersed by third-person campaigns, but it's a different sort of immersion: I'm controlling someone else, and viewing the universe through their eyes, instead of injecting myself right into the story. I don't think the inability to connect with a character is just a result of this style, but instead some sort of flaw in the writing: I may not always agree with what a character is doing, but I always want to feel like there's a good reason for it, that it makes sense in terms of what they've experienced and is consistent with their personality. Sometimes it's even refreshing when a player character makes decisions completely opposite to what I think I would have done, because then I'm really experiencing something new.
3) I would say that a story is important for keeping me invested in the gameplay (because let's face it, once you've blown up a few thousand Shivan fighters, it's not exactly a novel experience), but it doesn't have to be some deep elaborate narrative experience: even a simple story can be effective if it's told well. And not every story has to be about stopping the Third Shivan Incursion and saving humanity. I have to think of Homesick as an example: it largely consists of a few ships trying to get home, but I'd list it as one of the most compelling stories ever told in this game. And I daresay that most of us wouldn't have enjoyed your own offerings nearly as much if we felt that everything had to be srs bzns and make complete logical sense.

3.5) Considering that I've willingly played Second Great War Part 2 more than once, I think you have your answer to this one.

But seriously, if there's a good chunk of gameplay that I find enjoyable, I'm perfectly capable of turning my brain off and just rolling with it.
4) Like some previous answers, I think that the amount of setting work should be left up to whatever the author feels like writing, or at least what they feel is necessary to get their story across. I'm always up for some good world-building, and I've certainly spent my fair share of time sifting through Tech Room entries. (As you noted, though, the default game interface doesn't make this the most enjoyable thing in the world to do.) I don't think that huge lists of entries are absolutely necessary for a good campaign, but they are a nice bonus; for instance, I very much enjoyed the BP team's take on the capabilities of retail FS2 weaponry, versus their own enhancements. I think it's definitely a good thing to put as much world-building to the campaign itself as possible, without turning every mission into straight-up info-dumping of course. At the very least, adding a decent amount of material into campaign briefings is probably preferable to hiding it away in the Tech Room; the fiction viewer allows for a lot more flexibility in this department, too.
5) I was extremely impressed by the fact that the BP team essentially put together a whole list of in-theater ships and their activities for WiH, but I don't think it's necessary for every campaign to go quite that far: it's okay to have some random cruisers and corvettes pop up as-needed, as I don't think it disrupts the player's suspension of disbelief at all. However, you don't want to swing too far in the opposite direction, lest you wind up like an infamous campaign I mentioned earlier: these ships are huge expensive things crewed by lots of people, so they shouldn't be treated as disposable toys. I think it's fine to have some random "filler" missions in a campaign that don't necessarily advance the main story, provided they're enjoyable enough and don't get too numerous. After all, even in an active warzone, real military combat duty comes with a lot of down time. I can't remember if it was Derelict itself, or another campaign of that era, but there was one old patrol mission I enjoyed where absolutely nothing at all happens, and your fellow pilots turn it into something of a joke.
6) I'm mostly with NGTM-1R here, and I don't limit this to just gaming either. If you're going to bash me over the head with some over-arching theme to the point where it gets in the way of the narrative, I'm probably going to strongly dislike it. Focus on telling a good enjoyable story before all else, and then you can layer what you want on top of that.
7) Other than your own legendary offerings, I don't think I've ever come across a piece of music in a campaign that I previously recognized from somewhere else, so I haven't had this problem yet. I do feel like it's a good idea to generally avoid popular and well-known music, just because it can have the effect of ripping the player out of the moment.