Author Topic: Unicode text support  (Read 3547 times)

0 Members and 1 Guest are viewing this topic.

The newest nightly now contains a fix to how FSO passes the speech text to the WIndows API which should fix the issue with Japanese text.
thank you!

 
Have you made progress? The major problem seems like he presented.

I'd like to report a couple of problems I could find so far. I understand there's a workaround for each of these but it would be great if FS2 Open can handle them.

“Token too long” error is a little too unforgiving. There are only 14 letters in the line but the error message says “Length = 42”. I believe this is UTF-8 related problem... Some Japanese characters (including hiragana, katakana and kanji) take 3 bytes in UTF-8.

Code: [Select]
Warning: Training-2.fs2(line 832):
Warning: Token too long: [プライマリウェポンを選択しろ].  Length = 42.  Max is 31.

File: parselo.cpp
Line: 301

I don't know what “MAX_BRIEF_LINE_LEN” means but it seems Japanese briefing text is too long to display (actually they are not that long). Cutting them in half can fix this error.



FS2 automatically starts a new line when it encountered space (the one which appears when you hit “space” key) in Japanese text. Obviously you can avoid this by not using space at all.



 

Offline m!m

  • 211
Unfortunately I did not have time to work on resolving this issue.

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
“Token too long” error is a little too unforgiving. There are only 14 letters in the line but the error message says “Length = 42”. I believe this is UTF-8 related problem... Some Japanese characters (including hiragana, katakana and kanji) take 3 bytes in UTF-8.

Code: [Select]
Warning: Training-2.fs2(line 832):
Warning: Token too long: [プライマリウェポンを選択しろ].  Length = 42.  Max is 31.

File: parselo.cpp
Line: 301
In this particular instance (assuming line 832 is in the same as in the original mission file, and/or that I've correctly retranslated your too-long token back into English), there's no real reason for objective text to be limited to NAME_LENGTH since they're dynamically allocated anyway (and the final buffer they get put into for display purposes is 256 bytes long). So, in this specific situation, the maximum allowed length could theoretically be increased without really affecting anything else. The one potential problem is that the code doesn't work right for more than two lines of text, but given that we're dealing with text that takes up more bytes without taking up significantly more width, some sort of allowance could probably be made.

FS2 automatically starts a new line when it encountered space (the one which appears when you hit “space” key) in Japanese text. Obviously you can avoid this by not using space at all.
Well, yes, if there's only one space in a long string of text, wordwrapping will generally force a linebreak there. Now, granted, I'm no Japanese expert, but surely you can just... add more spaces, so there are other, more natural places for wordwrapping to break lines? If not, well, can you maybe use a non-breaking space (U+00A0)?
« Last Edit: January 29, 2018, 12:31:39 pm by AdmiralRalwood »
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 
Have you made progress? The major problem seems like he presented.

As I said before, these “problems” can be easily avoided. There is workaround for each of them.

“Token too long” error is a bit annoying, though. Sometimes it is very difficult to explain mission objectives in a 10-letter text...

Well, yes, if there's only one space in a long string of text, wordwrapping will generally force a linebreak there. Now, granted, I'm no Japanese expert, but surely you can just... add more spaces, so there are other, more natural places for wordwrapping to break lines? If not, well, can you maybe use a non-breaking space (U+00A0)?

I'd like to use existing old Japanese localization patch. Sadly it contains hundreds of “spaces” and it will take some time to remove/replace all of them.
However, if you guys believe it is not easy to let FSO handle them properly, of course I can do this task manually. I guess there are other Japanese players who want to join.

I've been busy working on my own project (I'm writing a blog about Orbiter simulator). I'll be back on this localization project as soon as my current project is finished (Hopefully it will be done by April).

 

Offline jr2

  • The Mail Man
  • 212
  • It's prounounced jayartoo 0x6A7232
    • Steam
Have you made progress? The major problem seems like he presented.

As I said before, these “problems” can be easily avoided. There is workaround for each of them.

“Token too long” error is a bit annoying, though. Sometimes it is very difficult to explain mission objectives in a 10-letter text...

Well, yes, if there's only one space in a long string of text, wordwrapping will generally force a linebreak there. Now, granted, I'm no Japanese expert, but surely you can just... add more spaces, so there are other, more natural places for wordwrapping to break lines? If not, well, can you maybe use a non-breaking space (U+00A0)?

I'd like to use existing old Japanese localization patch. Sadly it contains hundreds of “spaces” and it will take some time to remove/replace all of them.
However, if you guys believe it is not easy to let FSO handle them properly, of course I can do this task manually. I guess there are other Japanese players who want to join.

I've been busy working on my own project (I'm writing a blog about Orbiter simulator). I'll be back on this localization project as soon as my current project is finished (Hopefully it will be done by April).

Should be able to do a Find and Replace All (Use Notepad++, and do a Search > Replace > Replace All).  With a bit of fiddling you should be able to find something that will work to do what you want.

Np++ should have localization for your (and most) languages.

 
Re: Unicode text support
Unicode text doesn't show up when I use recent Nightly. Is there anything I should add to mod file?


I'm using Japanese localization mod and unicodeTest mod.

I have tried some of the Nightly builds and this is the result:

fs2_open_3_8_1_20180113_830fd70_x64_SSE2.exe or older – OK
fs2_open_3_8_1_20180116_2779629_x64_SSE2.exe or newer – doesn't work

 
Re: Unicode text support
I don't know what “MAX_BRIEF_LINE_LEN” means but it seems Japanese briefing text is too long to display (actually they are not that long). Cutting them in half can fix this error.



I found that I can avoid this error by starting a new line.

It seems FSO doesn't like lengthy sentence which contains no space (use of “space” is not common in most Asian languages. Usually we don't separate words in a sentence). All I have to do is separate them by adding space or CR/LF.
If you don't do this, briefing text doesn't appear properly or the game will crash immediately.

 

Offline m!m

  • 211
Re: Unicode text support
Unicode text doesn't show up when I use recent Nightly. Is there anything I should add to mod file?
<snip>

I'm using Japanese localization mod and unicodeTest mod.

I have tried some of the Nightly builds and this is the result:

fs2_open_3_8_1_20180113_830fd70_x64_SSE2.exe or older – OK
fs2_open_3_8_1_20180116_2779629_x64_SSE2.exe or newer – doesn't work
I can not reproduce this issue with the latest master version (which should also be the newest nightly). Please post your fs2_open.log file.  Instructions on how to do this can be found in this post.

 
Re: Unicode text support
Attached fs2_open.log is created when I encountered the problem above.

That being said, I believe I found a solution... Because everything is working fine now (with the latest Nightly). :confused:

This is all I did:

delete cache folder from mod folder.
run the game with "Run in window" option.
Unicode text is back!

I guess I did something wrong. Sorry.

 

Offline m!m

  • 211
Re: Unicode text support
That is very odd. Maybe there was something wrong with the shaders in the cache folder but "Run in window" should have no effect on that. Does the text disappear if you remove that option again?

 
Re: Unicode text support
  • delete cache folder
  • run the game with debug build ("Run in window" is on)

This is what seems to be needed to use Japanese text mod. Maybe "Run in window" is irrelevant?

Once Unicode text is back, everything works fine without debug build or "Run in window" option.
However, if I delete cache folder one more time, Unicode text will disappear again...