Author Topic: Unicode text support  (Read 1777 times)

0 Members and 1 Guest are viewing this topic.

The newest nightly now contains a fix to how FSO passes the speech text to the WIndows API which should fix the issue with Japanese text.
thank you!

 
Have you made progress? The major problem seems like he presented.

I'd like to report a couple of problems I could find so far. I understand there's a workaround for each of these but it would be great if FS2 Open can handle them.

“Token too long” error is a little too unforgiving. There are only 14 letters in the line but the error message says “Length = 42”. I believe this is UTF-8 related problem... Some Japanese characters (including hiragana, katakana and kanji) take 3 bytes in UTF-8.

Code: [Select]
Warning: Training-2.fs2(line 832):
Warning: Token too long: [プライマリウェポンを選択しろ].  Length = 42.  Max is 31.

File: parselo.cpp
Line: 301

I don't know what “MAX_BRIEF_LINE_LEN” means but it seems Japanese briefing text is too long to display (actually they are not that long). Cutting them in half can fix this error.



FS2 automatically starts a new line when it encountered space (the one which appears when you hit “space” key) in Japanese text. Obviously you can avoid this by not using space at all.



 

Offline m!m

  • 210
Unfortunately I did not have time to work on resolving this issue.

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
“Token too long” error is a little too unforgiving. There are only 14 letters in the line but the error message says “Length = 42”. I believe this is UTF-8 related problem... Some Japanese characters (including hiragana, katakana and kanji) take 3 bytes in UTF-8.

Code: [Select]
Warning: Training-2.fs2(line 832):
Warning: Token too long: [プライマリウェポンを選択しろ].  Length = 42.  Max is 31.

File: parselo.cpp
Line: 301
In this particular instance (assuming line 832 is in the same as in the original mission file, and/or that I've correctly retranslated your too-long token back into English), there's no real reason for objective text to be limited to NAME_LENGTH since they're dynamically allocated anyway (and the final buffer they get put into for display purposes is 256 bytes long). So, in this specific situation, the maximum allowed length could theoretically be increased without really affecting anything else. The one potential problem is that the code doesn't work right for more than two lines of text, but given that we're dealing with text that takes up more bytes without taking up significantly more width, some sort of allowance could probably be made.

FS2 automatically starts a new line when it encountered space (the one which appears when you hit “space” key) in Japanese text. Obviously you can avoid this by not using space at all.
Well, yes, if there's only one space in a long string of text, wordwrapping will generally force a linebreak there. Now, granted, I'm no Japanese expert, but surely you can just... add more spaces, so there are other, more natural places for wordwrapping to break lines? If not, well, can you maybe use a non-breaking space (U+00A0)?
« Last Edit: January 29, 2018, 12:31:39 pm by AdmiralRalwood »
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 
Have you made progress? The major problem seems like he presented.

As I said before, these “problems” can be easily avoided. There is workaround for each of them.

“Token too long” error is a bit annoying, though. Sometimes it is very difficult to explain mission objectives in a 10-letter text...

Well, yes, if there's only one space in a long string of text, wordwrapping will generally force a linebreak there. Now, granted, I'm no Japanese expert, but surely you can just... add more spaces, so there are other, more natural places for wordwrapping to break lines? If not, well, can you maybe use a non-breaking space (U+00A0)?

I'd like to use existing old Japanese localization patch. Sadly it contains hundreds of “spaces” and it will take some time to remove/replace all of them.
However, if you guys believe it is not easy to let FSO handle them properly, of course I can do this task manually. I guess there are other Japanese players who want to join.

I've been busy working on my own project (I'm writing a blog about Orbiter simulator). I'll be back on this localization project as soon as my current project is finished (Hopefully it will be done by April).

 

Offline jr2

  • The Mail Man
  • 212
  • It's prounounced jayartoo 0x6A7232
    • Steam
Have you made progress? The major problem seems like he presented.

As I said before, these “problems” can be easily avoided. There is workaround for each of them.

“Token too long” error is a bit annoying, though. Sometimes it is very difficult to explain mission objectives in a 10-letter text...

Well, yes, if there's only one space in a long string of text, wordwrapping will generally force a linebreak there. Now, granted, I'm no Japanese expert, but surely you can just... add more spaces, so there are other, more natural places for wordwrapping to break lines? If not, well, can you maybe use a non-breaking space (U+00A0)?

I'd like to use existing old Japanese localization patch. Sadly it contains hundreds of “spaces” and it will take some time to remove/replace all of them.
However, if you guys believe it is not easy to let FSO handle them properly, of course I can do this task manually. I guess there are other Japanese players who want to join.

I've been busy working on my own project (I'm writing a blog about Orbiter simulator). I'll be back on this localization project as soon as my current project is finished (Hopefully it will be done by April).

Should be able to do a Find and Replace All (Use Notepad++, and do a Search > Replace > Replace All).  With a bit of fiddling you should be able to find something that will work to do what you want.

Np++ should have localization for your (and most) languages.