It should be *possible* to scale to more than 12 players;
Eve Online has had battles with thousands of players,
and NATO training simulators have had tens of thousands
of players I believe.
It's not really possible to have more than 12 players currently. We definitely would have done it by now if we could. Volition just designed the game around that limit and changing it breaks a LOT of stuff. Packets, loadout, UI ... just all sorts of stuff is designed around that limit. It would be quite an undertaking to change. Remember that, even now, the bulk of the game engine was designed in 1998, and the networking code was optimized for dial-up modems. We are looking into increasing the player limit in the future and have several projects in the works which will lay the foundation for that change, but that's a really long term goal.
FSO does support around 8000 ships in a mission for multiplayer, but only 12 can be player controlled, with the rest being AI. Also, a standalone server counts as a player, even though it's non-playable, so using that only allows for 11 other players.
Your best bet is probably to look into scripting. A lot can be done with the lua scripting in FSO and you could probably work up some interesting scenarios (e.g., a few "players" each controlling large numbers of drones). All ships, whether player or AI controlled, are handled largely the same by the networking code. But it's really going to depend on your exact goals as to whether such capabilities are sufficient.