It depends on what you want to happen. If the ship is supposed to become friendly to some other faction in the mission, you should use the change iff SEXP to switch the unknown ship to that side (friendly, hostile, etc) before giving its escorts guard orders. If they are supposed to come in as unknown escorts, then they should be set to unknown as well. You also may find it useful to make sure the "all teams at war" flag is turned off, as this could be complicating things a bit. The AI doesn't know what to do about protecting a ship of another IFF, particularly if the two aren't naturally friendly. "Friendly" and "Unknown" are actually hostile to one another by default, as are "Friendly" and "Neutral" so you'll need to be aware of that.
Another thing that might work is giving the escorts an ignore ship order on the escortee before actually telling them to protect it, so that they are not naturally inclined to open fire on it as they manuver around it. Ignore does tend to cause AI pilots to avoid the ship they are ignoring though, so you'll have to use some experimentation.
