Modding, Mission Design, and Coding > Test Builds

Unicode text support

(1/11) > >>

m!m:
The text encoding FSO is using has always been a bit weird. In order to actually display special characters like the German umlauts FSO needed a special font file which had support for these characters. These font files also weren't the same for all localizations which lead to a lot of problems if you wanted to play the game in German but only had the English font files. The TrueType font rendering feature improved that situation a bit by adding the ability to render UTF-8 encoded Unicode strings but FSO still assumed that every string was in the special encoding FSO used so no one could use that ability.

That is what these test builds are about. They introduce the ability to let FSO process UTF-8 encoded files and handle them properly in the entire engine.

Test builds for all platforms: http://swc.fs2downloads.com/builds/test/unicodeSupport/
Pull request: https://github.com/scp-fs2open/fs2open.github.com/pull/1416

Since this feature changes the way the engine handles text pretty extensively I chose to introduce a new "Unicode mode" for FSO. In this mode FSO expects that every text data it handles must be UTF-8 encoded Unicode strings. It also completely disables support for the old VFNT bitmap fonts.

You can enable Unicode mode by using the $Unicode mode option in the mod table. It has to appear after the location where the engine expects $Window title.

Now that you have enabled unicode mode you will probably encounter some issues. The standard retail files are all Latin1 encoded and FSO can't read that encoding anymore (it expects UTF-8). Since FSO is a nice program it will automagically detect if a file uses Latin1 encoding and then convert that data to UTF-8. Since that is not the desired data format FSO will show a warning to let you know that you should really convert the encoding.

Converting the encoding can either be done with a command line utility like iconv or a text editor like Notepad++ which has support for reencoding a file. The files you must convert from a retail install are string.tbl (this one requires a few other small changes), tstrings.tbl and weapons.tbl. The weapons table is only an issue because it uses some special characters in a comment but FSO still reads those characters and doesn't understand them.

The string table contains one entry that previously used a special character. This is the entry with the number 385. Replace the %c with © symbol to restore the old behavior. Another issue with this table is that it contains syntax errors that were not recognized by the retail parsing code. These parser errors were corrected in FSO but since we never break retail compatibility there is a workaround which fixes that for retail data but that workaround does not work with UTF-8 encoded files since it assumes that the text data is Latin-1 encoded. I already fixed these issues for my tests and uploaded those files in the test mod below.


While I was testing these changes I needed a test mod which should be a good starting point for your tests: http://www.mediafire.com/file/pw5788071fljm1m/unicodeTest.7z

Theoretically, this should also allow translations into languages with radically different characters like Japanese.


Please test these changes and let me know if you find something that breaks the new code.

Novachen:
Oh yes... that is very interesting for my german translations, especially for modifications that uses TTFs. I will take a look into this tomorrow.

Novachen:
So, my first report.

With the supplemented mod everything seems to work with the german language setting in my playtests. Did not encounter an error in a playthrough of one mission.


After that, i started some further tests with a full fleshed out modification that uses some additional FSO features. So i have adapted this to my Between the Ashes: Slaves of Chaos translation.

The good news:
- The special characters are shown correctly in the menus. Even in the fiction viewer, news terminal and the sector map. It seems that the fiction viewer texts have to be converted into UTF-8 without BOM instead of pure UTF-8, otherwise there is a visible rectangle at the start of the text.


The bad news:
- After i start a mission, my game crashes after a few seconds. So i can not check if this one would work with custom and mission specific hud gauges also. The error message is not helpful in my case to see, where the actual problem is.



--- Quote ---Error: Caught std::exception in main(): 'Not enough space'!
File: freespace.cpp
Line: 7947

ntdll.dll! ZwWaitForSingleObject + 12 bytes
KERNELBASE.dll! WaitForSingleObject + 18 bytes
fs2_open_3_8_1_unicodeSupport_SSE2.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2.exe! <no symbol>
KERNEL32.DLL! BaseThreadInitThunk + 36 bytes
ntdll.dll! RtlGetAppContainerNamedObjectPath + 253 bytes
ntdll.dll! RtlGetAppContainerNamedObjectPath + 205 bytes

--- End quote ---

In Debug Build i got also this one:

--- Quote ---Assert: "(size_t)handle < GL_buffer_objects.size()"
File: gropengltnl.cpp
Line: 210

ntdll.dll! ZwWaitForSingleObject + 12 bytes
KERNELBASE.dll! WaitForSingleObject + 18 bytes
fs2_open_3_8_1_unicodeSupport_SSE2-FASTDBG.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2-FASTDBG.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2-FASTDBG.exe! <no symbol>

[ This info is in the clipboard so you can paste it somewhere now ]


Use Debug to break into Debugger, Exit will close the application.

ntdll.dll! ZwWaitForSingleObject + 12 bytes
KERNELBASE.dll! WaitForSingleObject + 18 bytes
fs2_open_3_8_1_unicodeSupport_SSE2-FASTDBG.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2-FASTDBG.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2-FASTDBG.exe! <no symbol>
fs2_open_3_8_1_unicodeSupport_SSE2-FASTDBG.exe! <no symbol>
--- End quote ---

m!m:

--- Quote from: Novachen on November 25, 2017, 08:29:50 am ---The good news:
- The special characters are shown correctly in the menus. Even in the fiction viewer, news terminal and the sector map. It seems that the fiction viewer texts have to be converted into UTF-8 without BOM instead of pure UTF-8, otherwise there is a visible rectangle at the start of the text.

--- End quote ---
FSO should handle the BOM properly. Well, I added code for handling it but apparently that doesn't work.


--- Quote from: Novachen on November 25, 2017, 08:29:50 am ---The bad news:
- After i start a mission, my game crashes after a few seconds. So i can not check if this one would work with custom and mission specific hud gauges also. The error message is not helpful in my case to see, where the actual problem is.

--- End quote ---
Yeah, that error message isn't very useful (although I know where the exception is generated which is a start).

What changes did you make to your BtA mod for this? I have downloaded your BtA translation mod so I only need the files you changed.

Novachen:

--- Quote ---What changes did you make to your BtA mod for this? I have downloaded your BtA translation mod so I only need the files you changed.
--- End quote ---

I created a special character version of this translation back before the 3.8 release.

Extract that into the "FreiRaum_BtA SoC" folder.

It includes a sector map with special characters (only the sector view itself, not the invidiual planet views), one headline in the news room with special characters as well all the Hud gauges and mission specific gauges in the mission files itself with special characters.

I think the best missions to test should be "Murmeltiertag" (bta1_m3_03) and "Mit einem Knall erlöschen" (bta1_m3_04) because these show special character gauges right after the beginning of the mission. Even every one i have tried crashed.

[attachment stolen by Russian hackers]

Navigation

[0] Message Index

[#] Next page