[karajorma]

FS2 has always had support for German, French and Polish. We've often heard requests for other languages and today I decided it was time to add support for any language we can fit in under 255 characters (Chinese, Japanese, etc will have to wait unfortunately).

Support works like this. You'll need to add a new section to the start of Strings.tbl which looks like this.

Code: [Select]

#Supported Languages

$Language: German
+Extension: gr
+Non English Character Index: 164

$Language: French
+Extension: fr
+Non English Character Index: 164

$Language: Spanish
+Extension: sp
+Non English Character Index: 164

#End
the extension is used by the game to have extra folders which contain the foreign language files (i.e German voice acting etc.). The Non English Character Index is the point in the font where characters like the accents etc start appearing (at least as far as I can tell, that's what it's for).

At the moment, the only way to change languages is by manually editing the registry but hopefully the wx_Launcher guys will make it easier once this starts being used.

Here's the code changes.

Code: [Select]

Index: code/fred2/management.cpp
===================================================================
--- code/fred2/management.cpp (revision 9729)
+++ code/fred2/management.cpp (working copy)
@@ -325,7 +325,7 @@
  // NOTE : Fred should ALWAYS run in English. Otherwise it might swap in another language
  // when saving - which would cause inconsistencies when externalizing to tstrings.tbl via Exstr
  // trust me on this :)
- lcl_init(LCL_ENGLISH);
+ lcl_init(FS2_OPEN_DEFAULT_LANGUAGE);
 
  // Goober5000 - force init XSTRs (so they work, but only work in English, based on above comment)
  extern int Xstr_inited;
Index: code/freespace2/freespace.cpp
===================================================================
--- code/freespace2/freespace.cpp (revision 9729)
+++ code/freespace2/freespace.cpp (working copy)
@@ -8378,14 +8378,6 @@
 // Language autodetection stuff
 //
 
-// this layout *must* match Lcl_languages in localize.cpp in order for the
-// correct language to be detected
-int Lang_auto_detect_checksums[LCL_NUM_LANGUAGES] = {
- 589986744, // English
- -1132430286, // German
- 0, // French
-};
-
 // default setting is "-1" to use registry setting with English as fall back
 // DO NOT change that default setting here or something uncouth might happen
 // in the localization code
@@ -8415,8 +8407,8 @@
  detect = NULL;
 
  // now compare the checksum/filesize against known #'s
- for (idx=0; idx<LCL_NUM_LANGUAGES; idx++) {
- if (Lang_auto_detect_checksums[idx] == (int)file_checksum) {
+ for (idx=0; idx < (int)Lcl_languages.size(); idx++) {
+ if (Lcl_languages[idx].checksum == (int)file_checksum) {
  mprintf(( "AutoLang: Language auto-detection successful...\n" ));
  return idx;
  }
Index: code/localization/localize.cpp
===================================================================
--- code/localization/localize.cpp (revision 9729)
+++ code/localization/localize.cpp (working copy)
@@ -28,27 +28,29 @@
 // general language/localization data ---------------------
 
 // current language
-int Lcl_current_lang = LCL_ENGLISH;
+int Lcl_current_lang = FS2_OPEN_DEFAULT_LANGUAGE;
+SCP_vector<lang_info> Lcl_languages;
 
-// language info table
-typedef struct lang_info {
- char lang_name[LCL_LANG_NAME_LEN + 1]; // literal name of the language
- char lang_ext[LCL_LANG_NAME_LEN + 1]; // for adding to names on disk access
-} lang_info;
+// These are the original languages supported by FS2. The code expects these languages to be supported even if the tables don't
 
-lang_info Lcl_languages[LCL_NUM_LANGUAGES] = {
- { "English", "" }, // english
- { "German", "gr" }, // german
- { "French", "fr" }, // french
- { "Polish", "pl" }, // polish
+#define NUM_BUILTIN_LANGUAGES 4
+lang_info Lcl_builtin_languages[NUM_BUILTIN_LANGUAGES] = {
+ { "English", "", 0, 589986744}, // English
+ { "German", "gr", 164, -1132430286 }, // German
+ { "French", "fr", 164, 0 }, // French
+ { "Polish", "pl", 127 }, // Polish
 };
 
+/*
 #define DEFAULT_LANGUAGE "English"
+*/
 
+/*
 // following is the offset where special characters start in our font
 #define LCL_SPECIAL_CHARS_FR 164
 #define LCL_SPECIAL_CHARS_GR 164
 #define LCL_SPECIAL_CHARS_PL 127
+*/
 #define LCL_SPECIAL_CHARS 127
 int Lcl_special_chars;
 
@@ -148,7 +150,10 @@
 // initialize the pointer array into tstrings.tbl (call from lcl_ext_open() ONLY)
 void lcl_ext_setup_pointers();
 
+// parses the string.tbl and reports back only on the languages it found
+void parse_stringstbl_quick(char *filename);
 
+
 // ------------------------------------------------------------------------------------------------------------
 // LOCALIZE FUNCTIONS
 //
@@ -160,28 +165,51 @@
 
  char lang_string[128];
  char *ret;
- int lang, idx;
+ int lang, idx, i;
+ int rval;
 
  // initialize encryption
  encrypt_init();
 
+ // setup English
+ Lcl_languages.push_back(Lcl_builtin_languages[FS2_OPEN_DEFAULT_LANGUAGE]);
+
+ // check string.tbl to see which languages we support
+ if ( (rval = setjmp(parse_abort)) != 0 ) {
+ mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "strings.tbl", rval));
+ }
+ else {
+ parse_stringstbl_quick("strings.tbl");
+ }
+
+ parse_modular_table(NOX("*-lcl.tbm"), parse_stringstbl_quick);
+
+ // if the only language we have at this point is English, we need to setup the builtin languages as we might be dealing with an old style strings.tbl
+ // which doesn't support anything beyond the builtin languages. Note, we start at i = 1 because we added English above.
+ if ((int)Lcl_languages.size() == 1) {
+ for (i=1; i<NUM_BUILTIN_LANGUAGES; i++) {
+ Lcl_languages.push_back(Lcl_builtin_languages[i]);
+ }
+ }
+
  // read the language from the registry
  if(lang_init < 0){
  memset(lang_string, 0, 128);
  // default to DEFAULT_LANGUAGE (which should be English so we don't have to put German text
  // in tstrings in the #default section)
- ret = os_config_read_string(NULL, "Language", DEFAULT_LANGUAGE);
+ ret = os_config_read_string(NULL, "Language", Lcl_languages[FS2_OPEN_DEFAULT_LANGUAGE].lang_name);
 
  if(ret == NULL){
- Int3();
- strcpy_s(lang_string, DEFAULT_LANGUAGE);
- } else {
- strcpy_s(lang_string, ret);
- }
+ Warning(LOCATION, "Default language not found. Defaulting to %s", Lcl_languages[FS2_OPEN_DEFAULT_LANGUAGE].lang_name);
+ ret = Lcl_languages[FS2_OPEN_DEFAULT_LANGUAGE].lang_name;
+ }
 
+ strcpy_s(lang_string, ret);
+
+
  // look it up
  lang = -1;
- for(idx=0; idx<LCL_NUM_LANGUAGES; idx++){
+ for(idx = 0; idx < (int)Lcl_languages.size(); idx++){
  if(!stricmp(Lcl_languages[idx].lang_name, lang_string)){
  lang = idx;
  break;
@@ -191,7 +219,7 @@
  lang = 0;
  }
  } else {
- Assert((lang_init >= 0) && (lang_init < LCL_NUM_LANGUAGES));
+ Assert((lang_init >= 0) && (lang_init < (int)Lcl_languages.size()));
  lang = lang_init;
  }
 
@@ -221,6 +249,48 @@
  return Lcl_current_lang;
 }
 
+// parses the string.tbl to see which languages are supported. Doesn't read in any strings.
+void parse_stringstbl_quick(char *filename)
+{
+ lang_info language;
+ int lang_idx;
+ int i;
+
+ // make sure localization is NOT running
+ lcl_ext_close();
+
+ read_file_text(filename, CF_TYPE_TABLES);
+ reset_parse();
+
+ if (optional_string("#Supported Languages")) {
+ while (required_string_either("#End","$Language:")) {
+ required_string("$Language:");
+ stuff_string(language.lang_name, F_NAME, LCL_LANG_NAME_LEN + 1);
+ required_string("+Extension:");
+ stuff_string(language.lang_ext, F_NAME, LCL_LANG_NAME_LEN + 1);
+ required_string("+Non English Character Index:");
+ stuff_ubyte(&language.special_char_offset);
+
+ lang_idx = -1;
+
+ // see if we already have this language
+ for (i = 0; i < (int)Lcl_languages.size(); i++) {
+ if (!strcmp(Lcl_languages[i].lang_name, language.lang_name)) {
+ strcpy_s(Lcl_languages[i].lang_ext, language.lang_ext);
+ Lcl_languages[i].special_char_offset = language.special_char_offset;
+ lang_idx = i;
+ break;
+ }
+ }
+
+ // if we have a new language, add it.
+ if (lang_idx == -1) {
+ Lcl_languages.push_back(language);
+ }
+ }
+ }
+}
+
 void parse_stringstbl(char *filename)
 {
  char chr, buf[4096];
@@ -395,35 +465,16 @@
 
  nprintf(("General", "Setting language to %s\n", Lcl_languages[lang].lang_name));
 
+ Assertion((Lcl_current_lang >= 0) && (Lcl_current_lang < (int)Lcl_languages.size()), "Attempt to set language to an invalid language");
+
  // flag the proper language as being active
- Lcl_fr = 0;
- Lcl_gr = 0;
- Lcl_pl = 0;
- Lcl_english = 0;
- switch(lang){
- case LCL_ENGLISH:
- Lcl_english = 1;
- Lcl_special_chars = LCL_SPECIAL_CHARS;
- break;
- case LCL_FRENCH:
- Lcl_fr = 1;
- Lcl_special_chars = LCL_SPECIAL_CHARS_FR;
- break;
- case LCL_GERMAN:
- Lcl_gr = 1;
- Lcl_special_chars = LCL_SPECIAL_CHARS_GR;
- break;
- case LCL_POLISH:
- Lcl_pl = 1;
- Lcl_special_chars = LCL_SPECIAL_CHARS_PL;
- break;
- }
+ Lcl_special_chars = Lcl_languages[Lcl_current_lang].special_char_offset;
 
  // set to 0, so lcl_ext_open() knows to reset file pointers
  Lcl_pointer_count = 0;
 
  // reset file pointers to the proper language-section
- if(Lcl_current_lang != LCL_DEFAULT_LANGUAGE){
+ if(Lcl_current_lang != FS2_OPEN_DEFAULT_LANGUAGE){
  lcl_ext_setup_pointers();
  }
 }
@@ -504,7 +555,7 @@
  Assert(Lcl_ext_file == NULL);
 
  // if we're running in the default language, do nothing
- if(Lcl_current_lang == LCL_DEFAULT_LANGUAGE){
+ if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){
  return;
  }
 
@@ -524,7 +575,7 @@
  }
 
  // if we're running in the default language, do nothing
- if(Lcl_current_lang == LCL_DEFAULT_LANGUAGE){
+ if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){
  return;
  }
 
@@ -673,7 +724,7 @@
  }
 
  // if the localization file is not open, or we're running in the default language, return the original string
- if ( (Lcl_ext_file == NULL) || (str_id < 0) || (Lcl_current_lang == LCL_DEFAULT_LANGUAGE) ) {
+ if ( (Lcl_ext_file == NULL) || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) {
  if ( strlen(text_str) > (uint)max_len )
  error_display(0, "Token too long: [%s].  Length = %i.  Max is %i.\n", text_str, strlen(text_str), max_len);
 
@@ -759,7 +810,7 @@
  }
 
  // if the localization file is not open, or we're running in the default language, return the original string
- if ( (Lcl_ext_file == NULL) || (str_id < 0) || (Lcl_current_lang == LCL_DEFAULT_LANGUAGE) ) {
+ if ( (Lcl_ext_file == NULL) || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) {
  out = text_str;
 
  if (id != NULL)
@@ -1274,7 +1325,7 @@
  // seek to the currently active language
  memset(language_string, 0, 128);
  strcpy_s(language_string, "#");
- if(!stricmp(DEFAULT_LANGUAGE, Lcl_languages[Lcl_current_lang].lang_name)){
+ if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){
  strcat_s(language_string, "default");
  } else {
  strcat_s(language_string, Lcl_languages[Lcl_current_lang].lang_name);
@@ -1348,7 +1399,7 @@
 
 void lcl_get_language_name(char *lang_name)
 {
- Assert(LCL_NUM_LANGUAGES == 3);
+ Assert(Lcl_current_lang < (int)Lcl_languages.size());
 
  strcpy(lang_name, Lcl_languages[Lcl_current_lang].lang_name);
 }
Index: code/localization/localize.h
===================================================================
--- code/localization/localize.h (revision 9729)
+++ code/localization/localize.h (working copy)
@@ -18,21 +18,35 @@
 // LOCALIZE DEFINES/VARS
 //
 
+
+#define FS2_OPEN_DEFAULT_LANGUAGE 0
+
+/*
 // language defines
 #define LCL_NUM_LANGUAGES 4 // keep this up to date
-#define LCL_ENGLISH 0
 #define LCL_GERMAN 1
 #define LCL_FRENCH 2
 #define LCL_POLISH 3
+*/
 
-#define LCL_DEFAULT_LANGUAGE LCL_ENGLISH
+// for language name strings
+#define LCL_LANG_NAME_LEN 32
 
+// language info table
+typedef struct lang_info {
+ char lang_name[LCL_LANG_NAME_LEN + 1]; // literal name of the language
+ char lang_ext[LCL_LANG_NAME_LEN + 1]; // the extension used for adding to names on disk access
+ ubyte special_char_offset; // where in the font do we have the special characters for this language
+ int checksum; // used for language auto-detection
+} lang_info;
+
+extern SCP_vector<lang_info> Lcl_languages;
+
+//#define LCL_DEFAULT_LANGUAGE LCL_ENGLISH
+
 // following is the offset where special characters start in our font
 extern int Lcl_special_chars;
 
-// for language name strings
-#define LCL_LANG_NAME_LEN 32
-
 // use these to replace *_BUILD values
 // only 1 will be active at a time
 extern int Lcl_fr;
And you can find the builds here.

[Phantom Hoover]

Unicode support would be a character set too far, I assume?

[karajorma]

I was under the impression that unicode was already being worked on. This was something I could do in a few hours which was necessary for when we have unicode support anyway, so why not do it sooner rather than later?

[Iss Mneur]

I a few concerns.

It looks like this patch adds about 12 lines of commented out code.

This Warning doesn't really make sense. I am assuming its more like, "Preferred language not found" but if that is the case I don't know why this is a warning.

Code: [Select]

+   Warning(LOCATION, "Default language not found. Defaulting to %s", Lcl_languages[FS2_OPEN_DEFAULT_LANGUAGE].lang_name);


[Dragon]

Maybe "Selected language not found"? That'd make more sense (unless I misunderstood what it does, that is).

[karajorma]

This Warning doesn't really make sense. I am assuming its more like, "Preferred language not found" but if that is the case I don't know why this is a warning.

Code: [Select]

+   Warning(LOCATION, "Default language not found. Defaulting to %s", Lcl_languages[FS2_OPEN_DEFAULT_LANGUAGE].lang_name);


This is the result of changing a rather weird little from an Int3() in the original code into something a little less crashy. It really is doing what it says, failing to find the default language and therefore setting it to the default language. The original code did this

Code: [Select]

ret = os_config_read_string(NULL, "Language", DEFAULT_LANGUAGE);

and would then Int3() if ret was NULL. As far as I can tell, that is actually a legal return from the os_config_read_string() function but you really shouldn't be getting it at that point in the game. I'll change the error message to reflect that there is some sort of registry / ini file error going on.


BTW, people who use foreign language versions of FS2, can you please check this out. I especially want to know if I've caused any problems for the Polish version of Freespace cause V seem to have done something rather strange there (There are no Polish entries in Strings.tbl or Tstrings.tbl).

[Yarn]

For the sake of not requiring improper English in tables, "+Non English Character Index" should be changed to "+Non-English Character Index" (with a hyphen between "Non" and "English").

[karajorma]

Yeah. I noticed that earlier and it's already changed in my local copy. I just couldn't be bothered to build everything again before posting.

[Iss Mneur]

This is the result of changing a rather weird little from an Int3() in the original code into something a little less crashy. It really is doing what it says, failing to find the default language and therefore setting it to the default language. The original code did this

Code: [Select]

ret = os_config_read_string(NULL, "Language", DEFAULT_LANGUAGE);

and would then Int3() if ret was NULL. As far as I can tell, that is actually a legal return from the os_config_read_string() function but you really shouldn't be getting it at that point in the game. I'll change the error message to reflect that there is some sort of registry / ini file error going on.

Okay.

No.  The only way for os_config_read_string to return NULL is if the default is NULL.  Since we specify the default the only way that we could have a NULL in the default is NULL, and since the default comes from the global initialization, we would have rampant memory corruption, which should probably warrant more than just a Warning, I would think.

[karajorma]

Actually there is one other way. It does a quick return if a certain variable hasn't been set. I didn't chase down that variable and see what can cause it to be unset though.

[Rodo]

Sorry to butt in, but I'm kinda interested in this

the extension is used by the game to have extra folders which contain the foreign language files (i.e German voice acting etc.).

Which files exactly should those folders contain?, could we have a list of the possible files as an example or maybe a list of types of files supported/expected by fso?

Also, thanks for taking the time to do this

[karajorma]

To be honest I'm not certain (I don't have the code in front of me at the moment). I'd like to hear from the Germany and French users on this one.

[Goober5000]

Oh, well done with this.

[Iss Mneur]

Actually there is one other way. It does a quick return if a certain variable hasn't been set. I didn't chase down that variable and see what can cause it to be unset though.

I see.  You are still pretty much screwed if you are trying to read config data when the registry code was not initialized.  The SCP_UNIX equivalent doesn't even check.

[karajorma]

It probably should error out in those situations then.

[Flaser]

Just a quick question, is the engine still limited to the built in Bank Gothic font for all displayed text messages?

[Iss Mneur]

Just a quick question, is the engine still limited to the built in Bank Gothic font for all displayed text messages?

No. You can substitute any font you want.  Search the forum for "font builder".  There is also work on TTF font support though I am not sure of the status of that (check the test builds forum).  IIRC Diaspora has a custom font that is similar to the one used on the show.

However, as far as I know, any font that you build is still restricted to the strict ASCII latin characters (that is, the 26 upper case, the 26 lowercase, the 10 digits and about 10 symbols) as far a character set goes, which is what the subtitle of this thread is referring to.

[karajorma]

I don't think it is limited actually. Just that the characters over a certain value are reserved for other languages. That's what +Non-English Character Index: is for, setting where those characters start from.

[Yarn]

Does this currently provide a way for different languages to use different sets of fonts? If the FreeSpace font format doesn't support more than 256 characters, then this feature would be needed for a mod to support, say, both German and Russian. Also, I believe the font files included with non-English versions of FS2 are different from the ones in the English version, seeing as how playing the English version in German causes accented letters to display improperly. With this feature, mods could include those those altered fonts specifically for non-English languages.

[mjn.mixael]

This would go nicely with the TTF support...