I don't know how easy this kind of support would be and what would be the easiest way to implement this.
But I have an idea:
ships.tbl, weapons.tbl and other .tbl files are still used.
But now you can add references to another .tbl file.
So basically you can use just single ships.tbl and weapons.tbl file or you can add references into them to point to another file.
If you use references to another file, you don't need to use full pathnames. Instead FS2 looks for "ships" and "weapons" folders that are under "tables" folder.
In \data\tables\ships folder there are files like ulysses.tbl, orion.tbl, ravana.tbl and so on.
In \data\tables\weapons there are files like subach.tbl, prometheus-r.tbl, cyclops.tbl and so on.
If you use similar references in other main tbl files, FS2 looks for appropriate folder and file. For example, if you use references in music.tbl, FS2 looks for \data\tables\music\brief1.tbl for example.
Normal ships.tbl entry would look like this:
$Name: EA Aurora
$Short name: Aurora
$Species: Terran
+Type: XSTR("Heavy fighter", -1)
+Maneuverability: XSTR("Excellent", -1)
+Armor: XSTR("Light", -1)
+Manufacturer: XSTR("Mitchell-Hyundyne", -1)
+Description: XSTR("Short Range Defense Operations" , -1)
$end_multi_text
.
.
.
.
.
But if you want to use references, it could look like this:
$Name: EA Aurora
$Ref: aurora.tbl
Rest are in \data\ships\aurora.tbl file.
Regular table entry will take around 72-75 lines of space, this alternative method would take only two lines for every ship and weapon. References are in ships.tbl and weapons.tbl file, but if particular ship is not used in a mission, FS2 will skip it entirely. It will look for names that a mission uses.
If particular mission has only Orion, Ulysses, Medusa, Dragon, Demon and Lilith class ships. FS2 searches only for those references and loads them up.
Of course it can already do that. I really don't know anything about current FS2 mission loading methods.