Uh-oh. That's pretty messed up...
Okay, planets. Or backgrounds in general. There are two ways - the Retail way and the Better way. In Retail way, you need to use 256-colour (8-bit) PCX format, and on this case the colour (0,255,0) aka pure green will be interpreted as transparent (if you use the transparency mode in the table file, but more of that later). However, this has numerous downsides, the biggest of them being that you cannot blend things to transparency, since only pure green is transparent. This means that all the blended/aliased grey-green parts would be displayed as grey-green ugly mix and only the pure green parts of the image would show themselves transparent.
Now, the Better way, courtesy of friendly neighbourhood SCP coders, offers a lot more versatile way to use background textures, because it allows the use of 32-bit RGBA images, in either TGA or DDS format depending on which is better for the image in question. The inclusion of alpha channel allows you to blend things smoothly into transparency, which is a godsend with atmospheric glows, nebulae and other fancy schmancy stuff that are supposed to blend into transparency.
[There is a third way which is usable exclusively for nebulae, and that's additive blending, which uses black on the image as transparent and the transparency is defined by colour intensity of the image; these use just 24-bit RGB images without alpha channel. For planets though, that isn't really viable option since you hardly want the shadowed part of your planet to be transparent, do you? Now you don't, so let's get on with it...

]
So, you'll need to put your planet, including it's atmosphere and the shadowed side, on a transparent layer and save as TGA. Once you get the TGA one working, you can then see if DDS compression can be used without damaging the gradient-rich atmosphere portion of the image too severely, but while you're working on it, get the TGA working first.
Your planet should look something like this:

Save the image and open it in your image editing software of choice to compare with your planet.
---
As to how to actually make the game read the background, you need to make a newplanet-str.tbm modular table file, which tells to the game that there's new stuff to be used.
Make a mod folder called "
newplanets" or whatever you like to call it on your Freespace2 directory (
..\Freespace2\newplanets\). Then, create data directory on it. On it, create "tables" and "effects" -directories.
On the
..\Freespace2\newplanets\data\tables\ directory, create a new notepad document, type
$BitmapX: newplanet-01
#end
Then, save the file as
newplanets-str.tbm (or in general,
xxx-str.tbm, whatever you prefer to use as it's name; I suggest something descriptive

)
This tells the game to find a transparency-blended background image called newplanet-01.* from the data used by the game. Now, you need to put the planet into the game with that filename. Simply save your ready alpha-blended planet to
..\Freespace2\newplanets\data\effects\ with filename
newplanet-01.tga.
Now you will have that planet on your disposal when you start either FRED2_Open or FS2_Open with the command line
-mod newplanets. To use a mod command line with FRED2_Open, create an icon to FRED2_Open, then open it's properties and add the appropriate command line options to the command line activated by the icon.