Ok, so I did a bit more work. I tried the precompiled binary (19-RC1), and got a different error, but it similarly wasn't able to open. It says this binary was compiled in 10.11, and I'm running 10.14, so that's likely the issue.
So I went and got the source, to recompile it. Unfortunately, I had to install a bunch of dependencies to get it working, but was able to finally build a binary without error - it even runs!
Unfortunately, it throws the error that indicates I've not installed Freespace2! Which means that, while the FSOInstaller runs, it doesn't appear to do the job of installing freespace2
Lastly, I simply used the innoextract in the command line to extract the gog setup file into the same directory as the newly created binary - this did the trick! I was able to load the game, I think. I clicked around a bit and checked out some of the menus etc., but didn't play anything for now. Hopefully this is the end of the saga of getting the install to work on my computer!
Thanks for you help, let me know if there's something I can do to improve the installer. Hopefully us lowly mac users won't need to go through all this to get it working in the future!