An indexed image is a little more complicated than an RGB image. An indexed image includes a color map in the file that has a list of numbers associated with a list of colors. Then each pixel, instead of including three or more numbers for RGB(A), only has one, the index number that relates to the color for that pixel. This is a form of compression and is a good idea in the world of 256 colors or less. In the world of millions of colors that we have today, it's not very useful. Mainhall masks, is a clever use of it, though.
Here, take a look at
this. To do this in Photoshop, you need to go to Image -> Mode -> Color Table. You'll then see the indexed color map. If you change the colors there, then you'll see the colors on your image change wherever they are linked to that particular index.
Additionally, I have found that Photoshop maps it's color table slightly differently than what FSO expects to find. It took me a lot of trial and error to get an indexed image out of Photoshop that FSO agreed with and in the end, I think I had to use another program inbetween. I haven't tried since, because it's not worth it. It's
way easier to open an existing mask file, change it however I need and then save it as a new mask. You're creating the file you need without having to deal with the outdated, numbery/mathy stuff that goes on behind the scenes.
TLDR: It's definitely possible to create an RGB/8 image in Photoshop and then save it as an indexed .PCX for FSO to use as a mask file, but it's difficult and complicated. Just use an existing mask file edited to your heart's content.