Most likely DXT3 or DXT5 format DDS file would do... as to size, that depends on what size you want it to be in the mission!
I could calculate it accurately, but as a general rule you should pay attention to where the mission happens. Most likely key installations and events would be located on high orbit, likely on geosynchronous altitude, which are planet-specific. For earth, that altitude is about 35000 km... from which altitude Earth would appear to be about 17 degrees in apparent diameter.
Now, if you don't use any -fov X.XX command line, the default field of view in FS2 is 43 degrees horizontally. 17 degrees is about 40% of that. If the horizontal resolution of the game is 1280 pixels, 40% of that is 512 pixels...
So, if you would only have high orbit missions over some planet, you wouldn't need bigger planets than 512x512 resolution.
But it is likely that at some stage you or someone else would like to have a lower orbit mission around that planet, and in that case a bigger planet image would be required. 1024x1024 would offer possibility to do this, without being overly big. A planet of that size can be used to fill the screen from top to bottom, and quite frankly that is already big in FS2 mission. IMHO all missions closer to planet than that should be made with skyboxes anyway, so that would be a good resolution.
So... depending on the atmospheric blending, use either DXT3 or DXT5 compression DDS images. AS a rule of thumb, DXT5 has better alpha blending so if you have a thick atmosphere around your planet, blending to transparent space, use DXT5; if it's a lifeless rock or a gas giant with relatively thin atmosphere, use DXT3. Try both, though, and look which gives better results. If there's no noticeable difference, I would use DXT3 for planets.