I'd like to report a couple of problems I could find so far. I understand there's a workaround for each of these but it would be great if FS2 Open can handle them.As you correctly stated, the issue here is that the amount of characters you see is less than the amount of bytes it takes to store the characters. Unfortunately, there is no general solution for this apart from reducing the length of the string. There are a lot of places in the parsing code where a fixed size array is used for storing a string and there simply isn't enough space to store the entire string. That was no problem before since ASCII text is stored pretty efficiently but if you need characters that are encoded using 3 or 4 bytes then you run out of space pretty fast.
“Token too long” error is a little too unforgiving. There are only 14 letters in the line but the error message says “Length = 42”. I believe this is UTF-8 related problem... Some Japanese characters (including hiragana, katakana and kanji) take 3 bytes in UTF-8.Code: [Select]
Warning: Training-2.fs2(line 832):
Warning: Token too long: [プライマリウェポンを選択しろ]. Length = 42. Max is 31.
There is a way of fixing this by using a variable size string but that type is sufficiently different from the current code that it doesn't work everywhere without major changes.
I don't know what “MAX_BRIEF_LINE_LEN” means but it seems Japanese briefing text is too long to display (actually they are not that long). Cutting them in half can fix this error.FSO probably doesn't know how to properly find word separators in Japanese text since that code is only written with Western languages in mind. It is possible to fix this by using an external library which knows how to split non-English text properly but that requires adding that library to FSO and integrating it into the code which is additional work. I hope that I can get around to that eventually but for now you probably need to help FSO a bit by adding spaces at the right places even if that makes the text look weird.
FS2 automatically starts a new line when it encountered space (the one which appears when you hit “space” key) in Japanese text. Obviously you can avoid this by not using space at all.
At least the first issue should be possible to fix without extensive changes to FSO (I hope) so could you upload your translation mod here so I can take a look at it?