[Yarn]

Fonts appear to be the only thing it does support actually. I'll see if I can expand that so that it can find German voice files in both data\voice\special and data\voice\special\gr. Doing that with the tables as well would allow you to have a properly multilingual install where the only requirement to switch languages is a change in the registry.

I would further expand this to all voices (including command briefings, briefings, debriefings, and personas), fiction, and HUD graphics (since kills1.ani in the German version is much wider to fit the text).

[karajorma]

Yeah, I meant to say that the special folder was an example and I'd be doing it with other folders too.

I'll also need to look into tables and missions since the German versions of these are different and I could see the need for this with other languages. Help.tbl is the worst offender and the question there is whether it's better to make the game parse table files separately for a different language (which would have a knock on effect in multiplayer) or simply make it possible to have multiple languages in the tables themselves.

EDIT : Well I looked into it but there is one major issue that is worth talking about. There is a problem with altering cfile to look for localised versions in this way.

Suppose Diaspora had German language files in the gr folders. Then we notice a bug with one of the English language files and release a patch to fix a few of the voices. Since these files are now in a VP with higher priority than the main VP, the English voice files will be found first. The game will then look for a German version in the patch VP file fail to find it and return the English version as the correct one to use. This results in a game with mostly German voice acting but the patched voices will be in English.*

The only way to solve this is to include all the German voices in the patch even though they didn't change or to change CFILE so that it checks through all the VPs for German versions first and only then revert to English. Changing CFILE looks pretty simple, merely requiring that I reorder a for loop but it opens a whole new can of worms. If instead of a voice file the issue is with a table file which has both English and German versions then now we have maintenance issues. Fixing the English table doesn't solve the problem because the German version would be found first and would still have the bug. This doesn't sound like too big a problem (simply release German versions of the table too) until you consider what happens if someone decides to add a German table to the media VPs. Doing so would break every single mod that has it's own version of that table.


So basically I though I'd better post this and ask for some suggestions on what to do before I start altering anything. 


*It should be noted that this scenario is a problem right now. A patched voice would break the current German version of FS2 too. Even though it doesn't use /gr/ folders.

[AdmiralRalwood]

The only way to solve this is to include all the German voices in the patch even though they didn't change or to change CFILE so that it checks through all the VPs for German versions first and only then revert to English. Changing CFILE looks pretty simple, merely requiring that I reorder a for loop but it opens a whole new can of worms. If instead of a voice file the issue is with a table file which has both English and German versions then now we have maintenance issues. Fixing the English table doesn't solve the problem because the German version would be found first and would still have the bug. This doesn't sound like too big a problem (simply release German versions of the table too) until you consider what happens if someone decides to add a German table to the media VPs. Doing so would break every single mod that has it's own version of that table.

How about look for any German version of the file in any VP, but only in that specific folder? In your example, that would give the mod's modified English table priority over the German MediaVPs table.

Basically, for loop hierarchy like this:

Code: [Select]

for folder in folders:
    for language in languages:
        for vp_file in vp_files:
            blah

instead of like this:

Code: [Select]

for language in languages:
    for folder in folders:
        for vp_file in vp_files:
            blah

(Apologies for horrible psuedocode and also not having looked at the source to see what it's actually doing.)

[karajorma]

Adding support for this is proving to be a bigger problem than I had expected.

I can add support for loose folders pretty easily, the issue is .vp files. The game only adds files into its map of what the vp file contains if the folder it's in matches one of the hardcoded pathtypes. If the folder doesn't match one of them, the file is ignored. That means that all the foreign language folders will be completely ignored. I can't even add pathtypes on the fly because the strings.tbl that tells the game which languages are supported could very easily be within the vp!

The only solution I can think of is to say that strings.tbl must be outside of the vp for it to work (or to create a new table that always lives outside the vp and does the same thing).

If anyone has a better idea, I'm all ears.

[Goober5000]

I would suggest emailing Taylor and see what he has to say.  He has worked extensively on the CFILE system (like just about everything else) in the past.

[karajorma]

I'm not certain the issue is actually CFILE. It's more a chicken and egg situation. CFILE can't index the extra language folders to the vp unless it can read the strings.tbl folder but it can only read once it's indexed the vp files.

Short of doing more than one pass of all the vp files I can't see any way round that.

I don't mind speaking to Taylor but I don't have his email address on my phone and I won't have net access on my PC for a few days.

[Goober5000]

Maybe do more than one pass at first, and from then on (once strings.tbl has been read) do only one pass.

Alternatively, you could make a list of every single two-character language code and always index all of them.  But that would probably be too inefficient.

[karajorma]

I'd rather avoid hardcoding the support for languages. I don't much like the idea of needing a new build should we forget one.

Similarly there is an issue with parsing until we find a string.tbl, we have no way of knowing if somewhere else in the vp files there is a string .tbm which adds another language.

At least if we insist on loose files we can find them all before we start parsing. Or we could do loose files + vp files with a certain name.

[chief1983]

Is a multi-pass approach that big of a deal?  It's only done at load time, and checking to see if a certain filename is in a .vp isn't that big of a performance hit is it?

[karajorma]

It seems like a bit of a waste to index all the vps in the game, find an instance of only one or two files and then throw the whole lot away and do it again.

But if that's the approach everyone else is happy with, I don't mind doing it. I just didn't want to implement it that way and then have everyone complain at me.

[chief1983]

My only thought was that perhaps a multi-pass approach implemented now might solve other similar future problems too.

[karajorma]

Well it's probably worth seeing how long the game takes to index the vps, I definitely don't want to add to the start up time for something most users won't ever see (since they use English) and which even the foreign users will only see once mods start supporting it.

[FUBAR-BDHR]

Where exactly are the non-english voice files going to be stored?  It seems that if they are put anywhere under data/voice that it will cause issues as from my experience the entire voice folder including all subdirectories are searched for any voice file.  I've seen files found in a non standard folder like data/special/music be found and play. 

Since this needs removed from the registry (lets face it one mod/TC having language support and another not has and is going to cause things to blow up) why not create a new data/localization directory and add the language subdirectoires there.  Make the directory names whatever the language is.  For instance data/localization/german for German.  The language would be passed by the launcher or command line and that directory would be set in cfile based on the name passed.  If no name or an invalid name is passed it's skipped.  Other languages in the data/localizaton folder would be ignored just like things in data/docs, data/freddocs, data/whatever that isn't read by cfile.  Of course if the localization folder exists it would need to look there first and ignore the data/voice folder entry if it already exists.  If a specified file doesn't exist in localization then the question becomes should it fall back to English?  My opinion would be yes as there are a lot of cases where voice files aren't really voices (things like radio static). 

Now to take this a step further I would suggest a data/localization/<language>/voice data/localization/<language>/sounds data/localization/<language>/music data/localization/<language>/subtitles (this one doesn't exist yet but I was planning on adding a folder for subtitles since there is no default location for them they get thrown into maps, effects, interface etc).  This would allow you not only the ability to localize voice acting but subtitles, have translated music if there is voice in that, and any sound effects that contain voices. 

[karajorma]

The game is actually already set up to have voice files for german in Data/Voice/Special/Gr etc. It doesn't work currently in the case of loose files cause of a stupid bug in the code. VP files are similarly crippled by the bug described above.

If the game is finding files in subfolders of voice/special. It really shouldn't be. In fact as far as VP files go it won't even index them as being part of the VP at all. If it is finding files in the wrong folders this is actually a bug.

As for having a localisation folder. I did think about it but in the end I wasn't much of a fan. It require rewriting all the current localisation code and I really don't see much of a point since I can't see how it would prevent any bugs which aren't currently an issue.

That said, a localisation folder might be a solution for the issue I'm facing now that avoids the need to reparse every VP file. So I will look into it.

[FUBAR-BDHR]

Well a quick search does turn up the use of CF_TYPE_ANY in audiostr.cpp.  Not sure if that would cause it. 

In any case while having the voice files under a subdirectroy might take care of voice localization it doesn't do anything for other things that also could need localization.  Wouldn't it be better to set it up to handle that now before people start widely using it then to try to shoehorn it in later? 

[karajorma]

The problem is that you'd need to completely rewrite the Pathtypes code since it assumes everything is in data. You'd need to teach it that data/localisation has the same path structure. And I don't think that would be a trivial change.

Whereas the gr/fr/ etc folders simply require altering the code to consider looking in the subfolders of the existing pathtypes.

Basically as far as I can see it's much, much easier to implement things the way wanted to than completely rewrite the way Pathtypes works.

[FUBAR-BDHR]

I really don't see the difference in altering the code to look in data/voice/special/gr vs /data/localizaion/voice/special/gr.  Did even transloate anything besides the special voices and text?  What about briefings, debriefings, etc? 

[karajorma]

I'm not sure what they translated but we shouldn't be limited by that since their translation isn't in localisation folders, but simply replaces the English version like a mod.

I think ideally you should be able to replace any English in the game with another language. Both in sound and text.

I really don't see enough difference between having the German version of a table in data/tables/gr or data/localisation/tables to justify rewriting the entire pathtypes code (and risking the resulting bugs). If I'm missing something, I really can't see what it is.

[Yarn]

I've got some important things to say about this effort, none of which are related to localized folders:


1) Currently, the "Non-English Character Index" field (which, as I said earlier, is for special symbols and not accented letters) takes only one parameter. However, as I pointed out here, font03.vp has the special characters at a unique offset, so there's no way for a HUD gauge that uses special characters to use font03.vp and still display correctly. The "Non-English Character Index" field should somehow be expanded to take three parameters, one for each font.


2) When displaying non-ASCII characters, the game does some remapping from ISO Latin-1 to its own mapping. However, it doesn't do this with the fiction viewer. Because of this, if a default font that supports accented characters is used there, then non-ASCII characters will not display correctly.

3) For this last problem, I'll quote myself on something that we're adding to the next version of FSPort:

Most of the FreeSpace 2 multiplayer missions are still being parsed in the mission simulator. This is a problem because, with the new tstrings.tbl, the XSTR structures for many of their names now point to strings that are longer than 32 characters, causing a boatload of warnings to be thrown in the mission simulator if the language is set to Spanish.

We're going to have to block all the FS2 missions somehow. Right now, we can block single-player campaigns and their missions, but that's it. Perhaps this can be expanded somehow to block single (non-campaign) missions as well as multiplayer missions and campaigns. (If that can't be done, then we could use dummy files...)

So, yeah, the ignore-campaign feature should be expanded to cover single missions and multiplayer missions and campaigns. Also, any XSTR structures found in ignored campaigns and missions should not be translated.

[karajorma]

Since I'm on my phone i'll keep thing brief.

I don't much like the idea of splitting the special characters field into 3 in strings.tbl. I think that making it a font property defined in fonts.tbl is a better idea. If the setting isn't found, then it falls back to the language default.