One big plot hole for me is bosch wants to make contact with the shivans, how does he know the shivans are on the other side, would the shivans really wait for someone to activate the portal if their goal was to get to capella
I think that makes several assumptions:
1. You're assuming that the Shivans actions at Capella weren't the result of a new initiative. It's entirely possible that new information acquired by the Shivans had them move towards Capella, rather than assuming they had that knowledge all along. You're also assuming that ships in FS2 were present in that location during the time of the ancient war. However the number of new designs and new technology suggests to me, the opposite.
2. Conversely I think it's safe to assume that Bosch learned that the portal is the route that the ancients took in their voyage. And in that voyage, they encountered the shivans. I would think he's not so much certain that shivans are on the other side, but rather that he's hoping they are.
I would think that Lucifer was cut off because the ancients closed the portal. Between the time of the portal being shut down, and the time of FS2, a new shivan fleet moved into the area beyond the knossos. Bosch, hoping that the shivans were still there activated the portal with the intent of making contact. The new shivans, with differing goals from the lucifer fleet, discovered through either scouting or analysis of captured vessels (or contact with Bosch) that Capella was ideal for the Sathanas operation. They then gathered their fleet, routing it through the lion's den and into Capella. Secondary elements engaged the GTVA while the Sathanas successfully enacted their operation.