Hard Light Productions Forums

Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: Yarn on March 05, 2016, 09:52:15 pm

Title: NOX(s) macro
Post by: Yarn on March 05, 2016, 09:52:15 pm
This appears countless times throughout FSO's code, but all it appears to do is return whatever it's given. Is this macro intended to serve any purpose? I'm asking because I'm going to start making a PR that adds translation support to more text. One string that looks like it can safely be translated is "Nothing" when describing cargo contents, but some instances of it are surrounded with NOX().
Title: Re: NOX(s) macro
Post by: niffiwan on March 05, 2016, 10:09:54 pm
I think :v: had another program that searched the Freespace2 code for strings, and dumped them all into a list for translation, except it ignored strings that had NOX(s) around them. I guess that list was then used to insert XSTR around the strings they did want translated.

I don't know why :v: wouldn't want "Nothing" as cargo translated though.
Title: Re: NOX(s) macro
Post by: AdmiralRalwood on March 05, 2016, 11:22:02 pm
I think :v: had another program that searched the Freespace2 code for strings, and dumped them all into a list for translation, except it ignored strings that had NOX(s) around them.
There's also:
Code: [Select]
//XSTR:OFF
[bunch of strings]
//XSTR:ON

But since AFAIK nobody has that tool lying around (or really any reason to use it anymore), it's pretty vestigial.
Title: Re: NOX(s) macro
Post by: Goober5000 on March 05, 2016, 11:24:02 pm
What niffiwan and AdmiralRalwood said.

I wonder if "Nothing" remains untranslated for cargo because some of the code explicitly uses that to search for the cargo index.  (That's poor design; instead, the index should always be set to 0 -- as happens in some places, but not all -- and the 0 index be explicitly initialized with the XSTR for "Nothing".)  As for why some have NOX and some don't, I'm guessing they all should have NOX, but due to code drift this wasn't universally applied.
Title: Re: NOX(s) macro
Post by: Yarn on March 06, 2016, 11:59:17 pm
Setting index 0 to the translation of "Nothing" sounds like a good idea to me. In addition, I think that whenever a ship's cargo is either "Nothing" or its translation (since most missions don't use XSTRs), the cargo index should be 0. If this sounds good to you, then I'll put it into my upcoming PR.

EDIT: I noticed just now that the is-cargo and set-cargo SEXPs, like the old tech-add-intel SEXP, can't accept an XSTR index and are thus unable to cope with translation. It looks like is-cargo-xstr and get-cargo-xstr should be added to remedy this.
Title: Re: NOX(s) macro
Post by: karajorma on March 07, 2016, 05:50:03 am
I'm a little confused as to why that would matter if the cargo name in the model entry and in the SEXP are in the same language. If cargo names are translated and then stored then surely it would be a better idea to change that so that they were stored in English (or whatever the default is) and then translated whenever needed rather than coming up with new SEXPs just for the rarer edge cases. Or have I missed something?
Title: Re: NOX(s) macro
Post by: Goober5000 on March 07, 2016, 10:12:17 am
Setting index 0 to the translation of "Nothing" sounds like a good idea to me. In addition, I think that whenever a ship's cargo is either "Nothing" or its translation (since most missions don't use XSTRs), the cargo index should be 0. If this sounds good to you, then I'll put it into my upcoming PR.

EDIT: I noticed just now that the is-cargo and set-cargo SEXPs, like the old tech-add-intel SEXP, can't accept an XSTR index and are thus unable to cope with translation. It looks like is-cargo-xstr and get-cargo-xstr should be added to remedy this.

Both of these sound good. :yes:


I'm a little confused as to why that would matter if the cargo name in the model entry and in the SEXP are in the same language. If cargo names are translated and then stored then surely it would be a better idea to change that so that they were stored in English (or whatever the default is) and then translated whenever needed rather than coming up with new SEXPs just for the rarer edge cases. Or have I missed something?

You've missed one of two things: a) Yarn is talking about cargo contents, not cargo ship names or subsystems; b) translation happens when the mission is parsed and therefore access to the original untranslated name is lost.

This problem was discovered in FSPort when we called ( tech-add-intel "Shivans" ) but someone was running a German translation where the intel entry was "Die Shivaner".  The solution was to add a new sexp and change the mission to call ( tech-add-intel-xstr "Shivans", 3197 ).

What do you think would happen if someone wrote a mission with a sexp ( is-cargo "Iceni" "Bosch Beer" ) and the cargo was translated to "Bosch Bier"?
Title: Re: NOX(s) macro
Post by: karajorma on March 07, 2016, 12:07:51 pm
Actually I missed neither of those things.

As I said, it seems to me that it is a better solution is to not translate the cargo names on mission load but instead translate them on the fly than to come up with new SEXPs for really rare scenarios. Expecting the mission designer to make missions with the idea of translation a factor is somewhat silly. The game should be handling that for you.

Asking a mission designer to add the index from a table that might be neither finished or not even started is a rather massive disruption to the flow of mission design. I don't know how you did it but I don't worry about translations until a mod is very close to release because it makes sense to have all the messages in one mission in the same part of the table rather than scattered around the table based on when they were added. And that's before we get into the readability aspect. If I'm dealing with multiple ships, I now have to remember what all these indexes are.

Is their some reason why we can't translate the name on the fly instead? Or simply store cargo name AND an XSTR index and then just get the code to simply retrieve the translated string whenever it needs to be displayed? Cause then it wouldn't matter what the name was translated as the mission code would simply call it Bosch Beer all the way through.
Title: Re: NOX(s) macro
Post by: mjn.mixael on March 07, 2016, 12:16:00 pm
Asking a mission designer to add the index from a table that might be neither finished or not even started is a rather massive disruption to the flow of mission design. I don't know how you did it but I don't worry about translations until a mod is very close to release because it makes sense to have all the messages in one mission in the same part of the table rather than scattered around the table based on when they were added. And that's before we get into the readability aspect. If I'm dealing with multiple ships, I now have to remember what all these indexes are.

Those were my thoughts reading this thread earlier. I intend for my mod to have a complete XSTR table, should someone want to translate it. But this thread made me think it might not be worth it. I'd have to go change a whole score of SEXPs (I've no idea how many of those we've used at this point.).. and then as I'm working on the last mission now, I suddenly felt like I'd need to stop and figure this all out before moving on. So I thought to myself.. "nah, nevermind. Not worth it."
Title: Re: NOX(s) macro
Post by: Goober5000 on March 09, 2016, 09:05:30 pm
Actually I missed neither of those things.

Then why did you say you were confused as to why it would matter?

Quote
As I said, it seems to me that it is a better solution is to not translate the cargo names on mission load but instead translate them on the fly than to come up with new SEXPs for really rare scenarios. Expecting the mission designer to make missions with the idea of translation a factor is somewhat silly. The game should be handling that for you.

Perhaps so, but it's not a trivial problem.  Any localized string would have to be handled using something like the message list in the Event Editor rather than explicitly specifying the string in the sexp.  Message localization is transparent to the mission designer, but only because a certain amount of bookkeeping is involved.

Quote
Asking a mission designer to add the index from a table that might be neither finished or not even started is a rather massive disruption to the flow of mission design. I don't know how you did it but I don't worry about translations until a mod is very close to release because it makes sense to have all the messages in one mission in the same part of the table rather than scattered around the table based on when they were added. And that's before we get into the readability aspect. If I'm dealing with multiple ships, I now have to remember what all these indexes are.

You don't have to worry about the indexes at all.  Just set them all to -1 (which is the default of add-intel-xstr anyway) and then enumerate them later on.  You don't add the XSTR indexes for messages and briefings as you write them, so don't add them for this sexp either.

Quote
Is their some reason why we can't translate the name on the fly instead? Or simply store cargo name AND an XSTR index and then just get the code to simply retrieve the translated string whenever it needs to be displayed? Cause then it wouldn't matter what the name was translated as the mission code would simply call it Bosch Beer all the way through.

See above re the message list.  At minimum, you'd need to define a new OPF_LOCALIZED_STRING sexp parameter type and present the FREDder with a dialog box listing the strings and their localization names.


Those were my thoughts reading this thread earlier. I intend for my mod to have a complete XSTR table, should someone want to translate it. But this thread made me think it might not be worth it. I'd have to go change a whole score of SEXPs (I've no idea how many of those we've used at this point.).. and then as I'm working on the last mission now, I suddenly felt like I'd need to stop and figure this all out before moving on. So I thought to myself.. "nah, nevermind. Not worth it."

It's only slightly different from setting XSTR indexes for briefings, messages, and main hall text.  If you have a script that enumerates XSTR tags, modify the script to recognize the sexps as well.  (Or you could just translate everything except cargo names.  These hypothetical cargo-name-xstr sexps don't even exist yet.)

Yarn has done extensive work in the FSPort with translation support, including the ST:R translations.  He has a utility which can run through tables and missions and enumerate XSTR indexes.  I'm sure he'd be willing to share his insight with you for use in BTA.
Title: Re: NOX(s) macro
Post by: karajorma on March 10, 2016, 02:51:19 am
You don't have to worry about the indexes at all.  Just set them all to -1 (which is the default of add-intel-xstr anyway) and then enumerate them later on.  You don't add the XSTR indexes for messages and briefings as you write them, so don't add them for this sexp either.

If you don't see the idiocy involved in asking a mission designer to deliberately make events which won't actually work until just before release when you are ready to do work on localisation, then I'm at a loss to explain it.


Quote
Perhaps so, but it's not a trivial problem.  Any localized string would have to be handled using something like the message list in the Event Editor rather than explicitly specifying the string in the sexp.  Message localization is transparent to the mission designer, but only because a certain amount of bookkeeping is involved.

No it wouldn't. I don't see any sensible reason to localise cargo names in a way that requires changes to FRED or new SEXPs when every other thing in FRED is in English. We'd simply keep the current system (FRED in English) and change the FS2_Open side of things to handle the localisation for when the language isn't English.


Making new SEXPs just to handle the translation of cargo on a ship is a complete over reaction to a problem that is

1) Pretty minor
2) Should be solved without requiring the actions of the FREDder if translation is done by a second person.
3) Is massively overshadowed by the fact you can't translate the name of the ship that has the cargo in the first place!
Title: Re: NOX(s) macro
Post by: Dragon on March 10, 2016, 07:48:16 am
Note that usually, ship names (or any names for that matter) aren't supposed to be translated, so this would be a non-issue unless the name was descriptive (which happens, but is a rarer case). Cargo, on the other hand, usually describes items that are in the cargo bay, which usually should be translated.

I agree that this shouldn't require fiddling with missions, though.
Title: Re: NOX(s) macro
Post by: AdmiralRalwood on March 10, 2016, 11:43:25 am
You don't have to worry about the indexes at all.  Just set them all to -1 (which is the default of add-intel-xstr anyway) and then enumerate them later on.  You don't add the XSTR indexes for messages and briefings as you write them, so don't add them for this sexp either.

If you don't see the idiocy involved in asking a mission designer to deliberately make events which won't actually work until just before release when you are ready to do work on localisation, then I'm at a loss to explain it.
Er... why wouldn't they work, exactly? You're still supplying the default text, aren't you? They just wouldn't be translated when the language isn't set to English until you're ready to do localization, which isn't all that different from "never being translated", i.e. current behavior.
Title: Re: NOX(s) macro
Post by: Yarn on March 10, 2016, 02:34:14 pm
Yarn has done extensive work in the FSPort with translation support, including the ST:R translations.  He has a utility which can run through tables and missions and enumerate XSTR indexes.  I'm sure he'd be willing to share his insight with you for use in BTA.
While I did work on FSPort's translation support, it was m!m who made the tool that you're referring to.
Title: Re: NOX(s) macro
Post by: Goober5000 on March 10, 2016, 11:18:52 pm
Karajorma, if you're going to make pronouncements about the design and implementation of these sexps, at least do some research first so that you don't end up looking like an utter moron.


You don't have to worry about the indexes at all.  Just set them all to -1 (which is the default of add-intel-xstr anyway) and then enumerate them later on.  You don't add the XSTR indexes for messages and briefings as you write them, so don't add them for this sexp either.

If you don't see the idiocy involved in asking a mission designer to deliberately make events which won't actually work until just before release when you are ready to do work on localisation, then I'm at a loss to explain it.
Er... why wouldn't they work, exactly? You're still supplying the default text, aren't you? They just wouldn't be translated when the language isn't set to English until you're ready to do localization, which isn't all that different from "never being translated", i.e. current behavior.
Exactly so.  This is the same way that any other XSTR with an index of -1 behaves.

And although the is-cargo-name-xstr sexp is still theoretical at this point, the tech-add-intel-xstr sexp has been in the code for two and a half years and used in [a released version of] FSPort for one year.


While I did work on FSPort's translation support, it was m!m who made the tool that you're referring to.

Ah, okay.  I'll contact him and ask him about tweaking the script to handle this sexp.
Title: Re: NOX(s) macro
Post by: karajorma on March 11, 2016, 04:42:09 am
I still reckon you're solving the problem at the wrong end by dealing with it at the SEXP stage. So this is going to keep biting us on the arse. For instance this isn't going to stop lua scripts which work on cargo from breaking.

But what the hell, if people want to treat the symptoms, as long as I'm not the one having to do the nursing I don't care.
Title: Re: NOX(s) macro
Post by: Yarn on March 11, 2016, 01:42:53 pm
Karajorma does have a point, actually. The problem here is that translated strings are being used as identifiers, requiring everything that checks these strings to take an XSTR index to work properly.

The only solution seems to be to somehow store both the original and translated strings. The original string would be used as the identifier, while the translated string would be the one that the player sees. This would allow things like ships and medals to be translated without causing problems. The tech-add-intel-xstr SEXP can also be deprecated in favor of the old tech-add-intel, and that lame "feature" of tying each pilot to a single language can be abolished. (Something would still have to be done about SEXPs and scripts that add new text, though, such as the set-cargo SEXP.)
Title: Re: NOX(s) macro
Post by: AdmiralRalwood on March 11, 2016, 03:45:14 pm
and that lame "feature" of tying each pilot to a single language can be abolished.
I am particularly in favour of this outcome.
Title: Re: NOX(s) macro
Post by: Goober5000 on March 11, 2016, 06:01:13 pm
Karajorma does have a point, actually. The problem here is that translated strings are being used as identifiers, requiring everything that checks these strings to take an XSTR index to work properly.

The only solution seems to be to somehow store both the original and translated strings. The original string would be used as the identifier, while the translated string would be the one that the player sees.

But for XSTRs, the identifier is the index, not the string preceding it.  The string shown to the player is the string found in strings.tbl or tstrings.tbl, even for English.  It is only when the index is -1 that the preceding string is shown to the player.  In this sense, the preceding string is more like a comment than an "original" string.

And as was pointed out to me years ago by someone, if a translation is updated, even to fix spelling or grammar, the XSTR string and the tstrings.tbl string will no longer be in sync.  That will break the string lookup.  Relying on the string instead of the index is the programming equivalent of a faux ami.

So yes, the fact that strings are being used as identifiers is a problem, but storing the original string is the wrong approach.  The actual solution, as you yourself discovered with tech-add-intel-xstr, is to either translate both the lookup string and its target string or neither of them.  This is mathematically consistent with relying on the index.

Quote
This would allow things like ships and medals to be translated without causing problems. The tech-add-intel-xstr SEXP can also be deprecated in favor of the old tech-add-intel, and that lame "feature" of tying each pilot to a single language can be abolished. (Something would still have to be done about SEXPs and scripts that add new text, though, such as the set-cargo SEXP.)

The reason each pilot is tied to a single language is simply to ensure that both the lookup string and the target string are translated in the same way (per the previous point).  See this Mantis ticket (http://scp.indiegames.us/mantis/view.php?id=2920) for background.
Title: Re: NOX(s) macro
Post by: Yarn on March 14, 2016, 04:06:00 am
I was actually thinking of using something like a struct to store a pair of pointers: one pointer to the original string, and another to the translated string (or the original string if a translated one doesn't exist). Only strings that are 1) looked up and 2) displayed to the player (such as ship and intel names) would be handled this way; others (such as briefing text) would continue to be handled as they are now (as a char* or SCP_string).

I understand that this would require, at the very least, additional versions of stuff_string and lcl_ext_localize, but with clever programming it may be possible to avoid duplicating much code.

But for XSTRs, the identifier is the index, not the string preceding it.  The string shown to the player is the string found in strings.tbl or tstrings.tbl, even for English.  It is only when the index is -1 that the preceding string is shown to the player.  In this sense, the preceding string is more like a comment than an "original" string.
Actually, if the language is English, the text from tstrings.tbl isn't used at all (although the text from strings.tbl still is); the string preceding the index is used instead. This is true in both retail and current SCP builds (yes, I tested it just now).

And as was pointed out to me years ago by someone, if a translation is updated, even to fix spelling or grammar, the XSTR string and the tstrings.tbl string will no longer be in sync.  That will break the string lookup.  Relying on the string instead of the index is the programming equivalent of a faux ami.
I don't quite understand what you're getting at here. A typical translation is going to be different from the original anyway; how would changing such a translation suddenly cause string lookup to start failing?

So yes, the fact that strings are being used as identifiers is a problem, but storing the original string is the wrong approach.  The actual solution, as you yourself discovered with tech-add-intel-xstr, is to either translate both the lookup string and its target string or neither of them.  This is mathematically consistent with relying on the index.
How is storing the original string the wrong approach if you're also storing the translated string (and you've properly implemented the functions needed to deal with this)?

Quote
The reason each pilot is tied to a single language is simply to ensure that both the lookup string and the target string are translated in the same way (per the previous point).  See this Mantis ticket (http://scp.indiegames.us/mantis/view.php?id=2920) for background.
If I'm not mistaken, that was done mainly because the original names of intel entries aren't available for lookup if they're translated. If they were made available, then that safeguard wouldn't be necessary (at least for new pilots).
Title: Re: NOX(s) macro
Post by: Goober5000 on March 16, 2016, 09:16:21 pm
Actually, if the language is English, the text from tstrings.tbl isn't used at all (although the text from strings.tbl still is); the string preceding the index is used instead. This is true in both retail and current SCP builds (yes, I tested it just now).

:wtf:

Well, phooey.  I tested this back in the day for a strings.tbl entry and assumed tstrings.tbl handled it the same way.  A foolish inconsistency is the hobgoblin of code maintainers.

Quote
I don't quite understand what you're getting at here. A typical translation is going to be different from the original anyway; how would changing such a translation suddenly cause string lookup to start failing?

In other words, if you use the untranslated string as a string lookup, you're relying on something which is not actually the intended key of the table.  If you are using the untranslated string to obtain the XSTR index using the translated English string, this relies on the equivalence of the untranslated and English strings.

Quote
How is storing the original string the wrong approach if you're also storing the translated string (and you've properly implemented the functions needed to deal with this)?

See the previous paragraph.  It's not clear to me how you would link the untranslated string with a translated string (of any sort) without using the XSTR index.  Your proposed dual-string struct seems like a transformation of the problem rather than a solution to it.
Title: Re: NOX(s) macro
Post by: karajorma on March 17, 2016, 08:06:27 pm
Storing the index isn't the best idea either though. While it works fine during a single run of the code, if someone changes the tables it can result in all the indexes being wrong. From a quick glance at the pilot code I suspect that is what would happen if you insert an entry to species.tbl rather than appending to it.
Title: Re: NOX(s) macro
Post by: Goober5000 on March 19, 2016, 01:35:30 pm
The best idea that I've come across is to have a string identifier instead of a numeric identifier, but have that string be a short token rather than the entire untranslated string.  Then the English string would be a "translation" just like any other, but there would be no risk of mismatch.  This is the strategy Java uses in its language bundles, and consequently is what is used in the FSO Installer.

(It does have the disadvantage that the string is not immediately visible when maintaining the code.)