As far as I know GPL allows to use (and even modify) the libraries if they are not distributed so building a binary for private use may actually not be against the rules of the GPL.
Yep, section 2 of the GPL states this, that you can use and modify, and allow others to modify, the work under you sole discretion, provided that none of the involved party intends to distribute the "modified work" without appropriate licensing under the GPL. If you have the GPL'd library on your machine already, or otherwise can get it separately, there's nothing legal stating that you can't link the GPL'd library to the program.
The SCP can't distribute such versions of FSO, since that's expressively forbidden by the GPL, but conveying a version of FSO that's linked to an unmodified LGPL'd library is permitted, and of course the source code of FSO is freely available so folks savvy with build systems can use whatever GPL flavor they desire--so long as they don't distribute them.
The safe bet here for us would be to stick with LGPL libraries and leave the custom GPL linking to users that wish to do so, and probably warn them somewhere in
bold friendly letters that they cannot distribute/convey copies of their GPL-linked version.