[karajorma]

Here's a tool I think we've needed for a while.

In order to translate the game into other languages it would be very useful if we had a program that was capable of parsing all the tables and missions used in a mod and generating a tstrings.tbl from them. While this might sound difficult I did once create a very simple of these in Java in a couple of hours (what I did with the code has proved to be more of a mystery).

Generating a table is actually pretty easy. Translatable strings are present in both the tables and the mission files in this format.

Code: [Select]

XSTR("Text", Index)

In the case of most of our mods and games the index will be -1 (or an invalid number resulting from a cut and paste) as keeping track of the indexes during development is a pain in the arse. In the case of FS2 each of those indexes correctly correspond to an entry in tstrings.tbl. The FS2 table looks like this.

Code: [Select]

#default
0, "The main FreeSpace 2 campaign."

1, "Special Operations Command has selected you for a dangerous covert assignment. Will you accept?"

2, "Special Operations Command requests your participation in a high-risk rescue mission. Will you volunteer?"

3, "Act 3"

4, "Shivan Gauntlet"

5, "A gauntlet module in which you face waves of Shivan fighters and warships."

<<etc>>

3419, "So where's the Iceni, Command? Let’s end this now."

3420, "Protect Tatenen"

#end

At a rudimentary level (which is what I coded previously) all a program has to do in order to generate a table is parse in each table and mission in sequence, look for XSTR, append the text to Tstrings.tbl and change the index in the table/mission to correspond to the one it just wrote to Tstrings.tbl.

The problem with this approach is that you end up with a Tstrings.tbl with lots of duplicate entries. A better solution (and one obviously used since their table doesn't have so many duplicates) is to check that each entry to Tstrings.tbl is unique.

In addition any generator should have the ability to run in two modes, the first where it simply generates a completely new table is easy. The second however should be able to take an existing Tstrings.tbl and append any new entries to it (and preferably discard any that are no longer used too). This second mode would be the one that was useful when dealing with patches to existing campaigns etc.


From what I can see there are two ways this could work.


1) An independent stand alone program - This would be a very good project for any prospective coder looking to join the team as it doesn't even require looking at the FS2_Open codebase. All they need after all is a few example missions and tables. It has the advantage of being much simpler to code too.

2) An integral part of FRED. - This would be a lot more complicated. It would allow the user to add new messages or remove old ones and have the tables automatically updated to feature them.
 On the other hand while the stand alone is something that is only run a few times during the dev cycle this feature would run constantly and therefore has much more of a chance of causing problems when different developers have different versions of the tables or missions.

Personally I'd rather see the standalone.

Anyway, I wanted to get this proposal written down. If everyone agrees with me on a standalone program then we have something we can turn over to the next prospective SCP coder we get. Comments and crits welcome.

[Mobius]

Interesting. I definitely need to contact those who continued progress on the Italian translation and redirect them to this thread.

[chief1983]

You could include a SQLite db file of the text that is generated from the initial data and could later verify all of your missions/tables.  A db lends itself to indexed data with multiple languages quite nicely. It wouldn't even _need_ to be distributed, but could be kept in a mod's SVN/etc for development purposes.  You could set it to scan the data for any discrepancies between the database, and maybe even edit them on the fly.

[Goober5000]

This sort of thing is long overdue.   It's been on my to-do list for a long time, and I'd be extremely glad if someone who isn't me (or karajorma) could finish it before us.   I'm in favor of the standalone version as well; it would have the advantage that it can process multiple files (missions and tables) at once.

The problem with this approach is that you end up with a Tstrings.tbl with lots of duplicate entries. A better solution (and one obviously used since their table doesn't have so many duplicates) is to check that each entry to Tstrings.tbl is unique.

You must not have looked at tstrings very closely.  Search for the line "Medium" (including quotes) and you'll see that there are plenty of duplicates.

[karajorma]

Okay, for the most part it doesn't have duplicates though IIRC. I remember my program spit out a much larger strings table than the one when given the same data.

Either way, getting rid of duplicate entries is definitely a good thing.

[m!m]

I wrote a program that implements the idea of this topic:
It currently supports the creation of a tstrings table though I plan to add a feature to update the table.

Using it on retail data the program generated a table which is by 90 smaller compared to the retail tstrings.tbl (though this is based on indexes which this program optimizes as well).

I've attached the Jar-file along with a windows batch-file.
Please use with caution because the program may destroy missions or tables (it creates backup of the files it writes to but still...  )

I'll upload the source later because it isn't documented in any way...

[attachment deleted by ninja]

[karajorma]

Ummm. How exactly does it work?

[m!m]

It searches for all occurences of "XSTR" and searches the next ending bracket, then filters the string within " and finally parses the number behind the string.

Here's the code:

Code: [Select]

int tstringEnd = content.indexOf(')', index); // index is the index of the XSTR string
int nextOpenIndex = content.indexOf('(', index + 5);
while (nextOpenIndex < tstringEnd && nextOpenIndex > index)
{
tstringEnd = content.indexOf(')', nextOpenIndex);
tstringEnd = content.indexOf(')', tstringEnd + 1);
nextOpenIndex = content.indexOf('(',nextOpenIndex + 1);
}
String tstring = content.substring(index,tstringEnd + 1);
int stringBegin = tstring.indexOf('"') + 1;
int stringEnd = tstring.indexOf('"', stringBegin + 1);
if (stringEnd < 0)
{
stringEnd = stringBegin + 1;
}
String value = tstring.substring(stringBegin, stringEnd).trim();
int indexBegin = tstring.indexOf(',', stringEnd) + 1;
int indexEnd = tstring.indexOf(')', stringEnd);
String indexStr = tstring.substring(indexBegin,indexEnd).trim();

It's quite straight forward.

[The E]

Would it deal with the following XSTR correctly?

XSTR(" This is an XSTR with text in brackets. (This is the text in Brackets)", -1)

[jr2]

Neat-o  I like listening to programmers.   It makes my feel smarter by association. Ya brainy freaks!    

[m!m]

Would it deal with the following XSTR correctly?

XSTR(" This is an XSTR with text in brackets. (This is the text in Brackets)", -1)

The case of multiple brackets is already covered and the XSTR problem is dealt with a single line of code
Though it will cut of the begining whitespace of the string which will cause problems for FS because the strings don't match.
I'll post a fix tomorrow

[Spicious]

It seems like matching would be easier using

Code: [Select]

XSTR\("([^"]*)", ?(-?\d+)\)

[m!m]

Thank you for the help, matching the expression looks promising and I'll use the matching approach as soon as I figure out how to do it properly in Java (my first attempts failed miserably  ).
But I finally finished documenting the code and here it is along with the fixes and an eclipse project that should import without warnings.

I'll focus on getting the matching and the update mode (which is already included but does nothing ATM ) to work.

Regards,
m!m

[attachment deleted by ninja]

[karajorma]

Actually I was asking more where do the files need to be. I gave it a try but I just couldn't get it to generate anything no matter where I put the table files.

[m!m]

Either use

Code: [Select]

java -jar tstrings.jar [modDirectoryHere]or if you're on windows use the batch-file with

Code: [Select]

tstrings [modDirectoryHere]
If you don't give parameters it should default to the current directory
The program searches the files inside data/tables and data/missions.

And I ****ed up multiple things while "optimizing" the generator . Updated Jar-file is attached...
I hope it works 

[attachment deleted by ninja]

[Nighteyes]

before we speak about translating to other languages, is the text finalized at all? because if so we should start the voice acting ASAP as its a pretty long process... and takes priority to translating...

[m!m]

before we speak about translating to other languages, is the text finalized at all? because if so we should start the voice acting ASAP as its a pretty long process... and takes priority to translating...

I'm pretty sure that this doesn't belong in this thread

[Zacam]

Some ninja has deleted all the attachments. Any chance of putting up a link to somewhere?

[m!m]

Here it is: http://www.mediafire.com/?x7pu4qg3wccy4qz
I also added a windows exe launcher for easier usage.

[jr2]

Should something like this be included with FSO by default, sort of like how FRED is?