Yes we're doing Space Center - you wanted to do it, and you're the visitor, so we'll go!
I'm all for meeting there, really I am, but honestly it all depends on a) which direction everyone's coming from, and b) which direction everyone needs to go (as in - whereabouts in Houston are you staying?). Those two questions we need to answer as soon as possible, so we can determine a place to meet for lunch, or, as you mentioned, whether it's best to just meet at the space center.
Space Center closes at 7:00PM. If we meet up for lunch and then head that direction (say, getting there at 2:00PM) there'll be plenty of time to see most all of the sites and exhibits. 5 hours at the space center is plenty