Hard Light Productions Forums
Modding, Mission Design, and Coding => FS2 Open Tools => Topic started by: karajorma on August 05, 2010, 12:04:32 pm
-
Here's a tool I think we've needed for a while.
In order to translate the game into other languages it would be very useful if we had a program that was capable of parsing all the tables and missions used in a mod and generating a tstrings.tbl from them. While this might sound difficult I did once create a very simple of these in Java in a couple of hours (what I did with the code has proved to be more of a mystery).
Generating a table is actually pretty easy. Translatable strings are present in both the tables and the mission files in this format.
XSTR("Text", Index)
In the case of most of our mods and games the index will be -1 (or an invalid number resulting from a cut and paste) as keeping track of the indexes during development is a pain in the arse. In the case of FS2 each of those indexes correctly correspond to an entry in tstrings.tbl. The FS2 table looks like this.
#default
0, "The main FreeSpace 2 campaign."
1, "Special Operations Command has selected you for a dangerous covert assignment. Will you accept?"
2, "Special Operations Command requests your participation in a high-risk rescue mission. Will you volunteer?"
3, "Act 3"
4, "Shivan Gauntlet"
5, "A gauntlet module in which you face waves of Shivan fighters and warships."
<<etc>>
3419, "So where's the Iceni, Command? Let’s end this now."
3420, "Protect Tatenen"
#end
At a rudimentary level (which is what I coded previously) all a program has to do in order to generate a table is parse in each table and mission in sequence, look for XSTR, append the text to Tstrings.tbl and change the index in the table/mission to correspond to the one it just wrote to Tstrings.tbl.
The problem with this approach is that you end up with a Tstrings.tbl with lots of duplicate entries. A better solution (and one :v: obviously used since their table doesn't have so many duplicates) is to check that each entry to Tstrings.tbl is unique.
In addition any generator should have the ability to run in two modes, the first where it simply generates a completely new table is easy. The second however should be able to take an existing Tstrings.tbl and append any new entries to it (and preferably discard any that are no longer used too). This second mode would be the one that was useful when dealing with patches to existing campaigns etc.
From what I can see there are two ways this could work.
1) An independent stand alone program - This would be a very good project for any prospective coder looking to join the team as it doesn't even require looking at the FS2_Open codebase. All they need after all is a few example missions and tables. It has the advantage of being much simpler to code too.
2) An integral part of FRED. - This would be a lot more complicated. It would allow the user to add new messages or remove old ones and have the tables automatically updated to feature them.
On the other hand while the stand alone is something that is only run a few times during the dev cycle this feature would run constantly and therefore has much more of a chance of causing problems when different developers have different versions of the tables or missions.
Personally I'd rather see the standalone.
Anyway, I wanted to get this proposal written down. If everyone agrees with me on a standalone program then we have something we can turn over to the next prospective SCP coder we get. :D Comments and crits welcome.
-
Interesting. :D I definitely need to contact those who continued progress on the Italian translation and redirect them to this thread.
-
You could include a SQLite db file of the text that is generated from the initial data and could later verify all of your missions/tables. A db lends itself to indexed data with multiple languages quite nicely. It wouldn't even _need_ to be distributed, but could be kept in a mod's SVN/etc for development purposes. You could set it to scan the data for any discrepancies between the database, and maybe even edit them on the fly.
-
This sort of thing is long overdue. :yes: It's been on my to-do list for a long time, and I'd be extremely glad if someone who isn't me (or karajorma) could finish it before us. :) I'm in favor of the standalone version as well; it would have the advantage that it can process multiple files (missions and tables) at once.
The problem with this approach is that you end up with a Tstrings.tbl with lots of duplicate entries. A better solution (and one :v: obviously used since their table doesn't have so many duplicates) is to check that each entry to Tstrings.tbl is unique.
You must not have looked at tstrings very closely. Search for the line "Medium" (including quotes) and you'll see that there are plenty of duplicates. :p
-
Okay, for the most part it doesn't have duplicates though IIRC. I remember my program spit out a much larger strings table than the :v: one when given the same data.
Either way, getting rid of duplicate entries is definitely a good thing.
-
I wrote a program that implements the idea of this topic:
It currently supports the creation of a tstrings table though I plan to add a feature to update the table.
Using it on retail data the program generated a table which is by 90 smaller compared to the retail tstrings.tbl (though this is based on indexes which this program optimizes as well).
I've attached the Jar-file along with a windows batch-file.
Please use with caution because the program may destroy missions or tables (it creates backup of the files it writes to but still... :nervous:)
I'll upload the source later because it isn't documented in any way...
[attachment deleted by ninja]
-
Ummm. How exactly does it work?
-
It searches for all occurences of "XSTR" and searches the next ending bracket, then filters the string within " and finally parses the number behind the string.
Here's the code:
int tstringEnd = content.indexOf(')', index); // index is the index of the XSTR string
int nextOpenIndex = content.indexOf('(', index + 5);
while (nextOpenIndex < tstringEnd && nextOpenIndex > index)
{
tstringEnd = content.indexOf(')', nextOpenIndex);
tstringEnd = content.indexOf(')', tstringEnd + 1);
nextOpenIndex = content.indexOf('(',nextOpenIndex + 1);
}
String tstring = content.substring(index,tstringEnd + 1);
int stringBegin = tstring.indexOf('"') + 1;
int stringEnd = tstring.indexOf('"', stringBegin + 1);
if (stringEnd < 0)
{
stringEnd = stringBegin + 1;
}
String value = tstring.substring(stringBegin, stringEnd).trim();
int indexBegin = tstring.indexOf(',', stringEnd) + 1;
int indexEnd = tstring.indexOf(')', stringEnd);
String indexStr = tstring.substring(indexBegin,indexEnd).trim();
It's quite straight forward.
-
Would it deal with the following XSTR correctly?
XSTR(" This is an XSTR with text in brackets. (This is the text in Brackets)", -1)
-
Neat-o I like listening to programmers. :nod: It makes my feel smarter by association. Ya brainy freaks! :D
-
Would it deal with the following XSTR correctly?
XSTR(" This is an XSTR with text in brackets. (This is the text in Brackets)", -1)
The case of multiple brackets is already covered and the XSTR problem is dealt with a single line of code :P
Though it will cut of the begining whitespace of the string which will cause problems for FS because the strings don't match.
I'll post a fix tomorrow :nervous:
-
It seems like matching would be easier using
XSTR\("([^"]*)", ?(-?\d+)\)
-
Thank you for the help, matching the expression looks promising and I'll use the matching approach as soon as I figure out how to do it properly in Java (my first attempts failed miserably :nervous:).
But I finally finished documenting the code and here it is along with the fixes and an eclipse project that should import without warnings.
I'll focus on getting the matching and the update mode (which is already included but does nothing ATM :lol:) to work.
Regards,
m!m
[attachment deleted by ninja]
-
Actually I was asking more where do the files need to be. I gave it a try but I just couldn't get it to generate anything no matter where I put the table files.
-
Either use
java -jar tstrings.jar [modDirectoryHere]or if you're on windows use the batch-file with
tstrings [modDirectoryHere]
If you don't give parameters it should default to the current directory
The program searches the files inside data/tables and data/missions.
And I ****ed up multiple things while "optimizing" the generator :mad:. Updated Jar-file is attached...
I hope it works :nervous:
[attachment deleted by ninja]
-
before we speak about translating to other languages, is the text finalized at all? because if so we should start the voice acting ASAP as its a pretty long process... and takes priority to translating...
-
before we speak about translating to other languages, is the text finalized at all? because if so we should start the voice acting ASAP as its a pretty long process... and takes priority to translating...
I'm pretty sure that this doesn't belong in this thread ;)
-
Some ninja has deleted all the attachments. Any chance of putting up a link to somewhere?
-
Here it is: http://www.mediafire.com/?x7pu4qg3wccy4qz (http://www.mediafire.com/?x7pu4qg3wccy4qz)
I also added a windows exe launcher for easier usage.
-
Should something like this be included with FSO by default, sort of like how FRED is?
-
Thanks for the link!
Well, if/when wxFred starts getting off the ground, actually including this capability into it (via the Campaign Editor, maybe?) might not be a half-bad idea. We already do have a fair number of externalized tools though that building a common framework to run them all in would be nice in the meantime. (This, the ModPack Checker that FSF wrote, PCS2, Mjn's ANI Tools, Contemplates (interface Generator) my proposed HUD builder, etc).
So, maybe a nice all-in-one wrapper for a Community ToolKit/Toolbox/whateveryouwanttocallit.
-
I'm working on some kind of modding IDE which should contain all sorts of modding tools from a table editor to a mod exporter which would package the files inside a VP. I already got a working EFF viewer/editor and an ANI viewer. I'd also like to integrate a visual HUD_gauges.tbl editor where you could create a custom HUD layout by using drag-and-drop and then export it to the desired resolutions.
-
2) An integral part of FRED. - This would be a lot more complicated. It would allow the user to add new messages or remove old ones and have the tables automatically updated to feature them.
On the other hand while the stand alone is something that is only run a few times during the dev cycle this feature would run constantly and therefore has much more of a chance of causing problems when different developers have different versions of the tables or missions.
That's the reason why I wouldn't want to see it in wxFRED. It makes more sense as a standalone or at least something you must explicitly choose to run from FRED.
-
A binary that wxFRED can call, sort of how VPView32 can call other things such as TblView?
-
Uhhhmm, really useful, thanks
-
Sorry for the necro post but I wanted to make this available to anyone who could need this tool.
I created a github repository for this project here: https://github.com/asarium/TStringGenerator (https://github.com/asarium/TStringGenerator)
The repository contains the source code and I also uploaded binary packages.
-
Hmmm, this is neat.. I'll have to try it out.
I'm working on some kind of modding IDE which should contain all sorts of modding tools from a table editor to a mod exporter which would package the files inside a VP. I already got a working EFF viewer/editor and an ANI viewer. I'd also like to integrate a visual HUD_gauges.tbl editor where you could create a custom HUD layout by using drag-and-drop and then export it to the desired resolutions.
EFF Viewer/editor? Really? I would like that if it exists.
-
EFF Viewer/editor? Really? I would like that if it exists.
I actually have an application that is capable of reading and displaying EFFs and ANIs but it is unpolished, full of bugs and pretty much unusable right now.
-
EFF Viewer/editor? Really? I would like that if it exists.
I actually have an application that is capable of reading and displaying EFFs and ANIs but it is unpolished, full of bugs and pretty much unusable right now.
:(
-
That's why apng will be awesome.
-
That's why apng will be awesome.
Quoted for truth. Cannot wait for that.
-
I found a minor bug in TStringGenerator: the first XSTR structure that the program finds becomes the last tstrings.tbl entry for some reason. This happens in both versions 1.0 and 1.1.
-
Do you have a mod where this happens? I could not reproduce this here but I may be simply lacking example data.
-
Do you have a mod where this happens? I could not reproduce this here but I may be simply lacking example data.
Yes. It happens with the attached file (which is FSPort's fs1_tech-wep.tbm with all index numbers changed to -1) if translated by itself and no tstrings.tbl file exists.
[attachment deleted by ninja]
-
Ok, I can reproduce the bug and also have found the cause of it. I documented it in a github issue here: https://github.com/asarium/TStringGenerator/issues/1 (https://github.com/asarium/TStringGenerator/issues/1)
I will however wait until I have finished the my current refactoring of the code as I will not change anything in the current code base which is simply horrible :nono:
-
Ok everything is done I just pushed version 2.0 to github and also added binary distributions. Please add bugs you find to the github issue tracker, thank you.
-
Is there a readme or something? I assumed I could place it in the mod folder and run the jar.. but no such luck.
-
You have to execute one of the scripts in the bin folder with the -r or -root option to specify the mod folder:
> bin/tstrings -r "/path/to/your/mod/root"