Hard Light Productions Forums

Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: m!m on April 05, 2017, 04:00:52 am

Title: The new Lua SEXP interface
Post by: m!m on April 05, 2017, 04:00:52 am
Dynamic SEXPs is a feature I implemented recently which allows to add new SEXPs at runtime and have them properly integrated into FSO and FRED. The code for that is done (here is the pull request (https://github.com/scp-fs2open/fs2open.github.com/issues/1320)) but now I need to decide how that feature should be integrated into the modding workflow. There are multiple possible ways of exposing this functionality to modders and I would like to know which version would be preferred by the modding community:


These are the solutions I can currently think of. If you have any suggestions for improvements or alternative solutions I would like to hear them.


Code: (Lua API example) [Select]
local params = {
help = "This is a dynamic parameter test SEXP",
min_args = 1,
max_args = math.huge,
return_type = "void",
params = {
["..."] = {
description = "Some arguments",
type = "boolean"
},

{
description = "Test",
type = "number"
},
{
description = "Test2",
type = "ship"
};
},
}
mn.addSEXP("dynamic-sexp", params, function(arg1, arg2, ...)
ba.error(tostring(arg1) .. " " .. arg2)
end)
Title: Re: The new Lua SEXP interface
Post by: X3N0-Life-Form on April 05, 2017, 05:06:30 am
Hmm, I'm not sure what would be best, from a modder's standpoint.

I would be OK with a pure lua solution, since we could shove the new SEXPs into a *-sct.tbm.

However, I really like the way tables neatly structure their data, are automatically parsed when the game/FRED starts. I would also make it easier to read for modder unfamiliar with lua cracking open a mod looking for interesting stuff to grab. Incidentally, I'm not fond of having inline lua code, I feel it would unnecessarily crowd the file.


Table spec could look something like this
Code: [Select]
#Dynamic SEXPs
;; spec
$Name:           String
$Min Args:       Integer
$Max Args:      Integer --> Optional, default to any number
$Return type:  Type --> Optional, default to action SEXP
$Description:   String
$Function:       String --> name of the lua function that executes the SEXP
$Parameter
    +Description: String
    +Type:           Type

;; example
$Name:   my-awesome-sexp
$Min Args:       1
$Max Args:      3
$Description:   That's an awesome SEXP
$Function:       myAwesomeSexp
$Parameter
    +Description: That's a string param
    +Type:           String
$Parameter
    +Description: That's a bool
    +Type:           Boolean
$Parameter
    +Description: That's a number
    +Type:           Number

#End
Title: Re: The new Lua SEXP interface
Post by: m!m on April 05, 2017, 05:14:34 am
That's a good specification but $Function: will not work that way. I dislike the usage of global Lua values in general and I will not build a system which requires global values to function. Also, due to how Lua works, it's impractical to specify which function to call that way since the named function does not exist at the time when the table is parsed which means that the name must be resolved every time the SEXP is called which is unnecessary overhead.
Title: Re: The new Lua SEXP interface
Post by: xenocartographer on April 05, 2017, 07:05:03 am
Anything stopping you from caching the value, m!m?

I don't really think that avoiding globals is a worthwhile concern here, but if you really care, you could use a table as a namespace, analogous to static classes in Java. It doesn't entirely eliminate the risk of name collisions, but if you request (or enforce) naming functions after the SEXP itself, it's no worse than it would be anyway.
Title: Re: The new Lua SEXP interface
Post by: m!m on April 05, 2017, 08:26:15 am
Caching could be done but I am opposed to using global functions in the first place because it's bad design to use it like that if we have a system that is perfectly capable of handling functions as parameters which allows us to use something like my example above (mn.DynamicSEXPs["my-awesome-sexp"].Action = function() ...).

If everyone agrees that the global function name is the only solution that will ever work then I'll use it but at this point I don't see why it would be. Global functions complicate the implementation tremendously and lead to bad design decisions so I would like to avoid them wherever possible.
Title: Re: The new Lua SEXP interface
Post by: xenocartographer on April 05, 2017, 11:06:31 am
Okay, I'm not getting the hatred of globals, but what about something like this?

Code: [Select]
$Function: [
function someReservedName(foo, bar, baz)
-- implementation
end
]

? After evaluating the inline snippet, you'd register the function with that reserved name and set that name to nil. It doesn't require a global anything (well, okay, I guess it does for a handful of machine cycles, but eh), and you can immediately validate that the function exists (and possibly even that it has the correct arity).



Oh, would these dynamic SEXPs have access to the rest of the SEXP tree? I can think of some use cases for defining custom conditionals, stuff like that.



EDIT: @X3NO: I like your table spec as well, but I think you overlooked truly varardic SEXPs. How'd I specify something like is-destroyed-delay (which can take a theoretically infinite number of string arguments) or send-message-list (which can take a theoretically infinite number of arguments, but only in a certain pattern)? Maybe a +Repeat: suboption...? If there are fewer $Parameters: than specified in $Max Args:, any +Repeat: parameters are repeated, in order, to fill it out?
Title: Re: The new Lua SEXP interface
Post by: Axem on April 05, 2017, 12:26:52 pm
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.

Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)
Title: Re: The new Lua SEXP interface
Post by: Phantom Hoover on April 05, 2017, 12:39:28 pm
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.

Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)

I think table-with-Lua-API is actually quite good in that regard: you define the SEXP interface in a table and present an interface which FRED-and-SEXPers can use without touching Lua at all, then put the backend scripts somewhere else.
Title: Re: The new Lua SEXP interface
Post by: Axem on April 05, 2017, 01:04:32 pm
But shouldn't making that bridge be the responsibility of the scripter? Ideally the FREDder wouldn't have to do anything with these files except plop them in their modding directory.
Title: Re: The new Lua SEXP interface
Post by: Phantom Hoover on April 05, 2017, 01:07:12 pm
Well the scripter would provide both the backend script files and the table files defining the interface, the idea is that a FREDder would just have to look at the table to read which SEXPs were provided and what they did without having to try to parse that out of the Lua code.
Title: Re: The new Lua SEXP interface
Post by: xenocartographer on April 05, 2017, 01:26:37 pm
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.

Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)

Hm, I can see this, actually (especially with how controversial implementing a hybrid system looks like it might be :P ). Another side benefit is that it may be easier to convert existing scripts into SEXPs if desired. I think PH's concern is handled pretty well by FRED's in-editor documentation, so as long as the script author writes good docs, you shouldn't have to look at the Lua source even if that is where the interface is defined.
Title: Re: The new Lua SEXP interface
Post by: Axem on April 05, 2017, 01:37:59 pm
PH: Going by with what I've seen in that initial github discussion and PR, the dynamic sexps get put into FRED's sexp menu, complete with the documentation provided in the definition. So FREDders using the dynamic sexps wouldn't need to open up any files to see what's available, it'd look like any other sexp to them. So that's why in this case I think it's okay to be a little more "technical" than "friendly".
Title: Re: The new Lua SEXP interface
Post by: m!m on April 06, 2017, 03:12:36 am
Okay, I'm not getting the hatred of globals, but what about something like this?
My primary reason why I dislike globals is that they can cause conflicts with other scripts. If I could, I would refactor the entire scripting system so that each scripting block has its own namespace but that's not going to happen anytime soon. I don't want to put the script writers into a position where they have to think about the name of their function because it might overwrite a function from a different script. When I wrote the scripting SEXPs, that was my reason for not using globals in any way since I think that they only cause issues later on.

Oh, would these dynamic SEXPs have access to the rest of the SEXP tree? I can think of some use cases for defining custom conditionals, stuff like that.
What do you mean by that? If you mean that you can use other SEXPs as parameters for the dynamic SEXP then the answer is yes but more advanced features like custom conditional nodes are not included in the current version.

I am currently in favor of using the second alternative in my list above since a table based solution would improve error handling tremendously.
Title: Re: The new Lua SEXP interface
Post by: The E on April 06, 2017, 03:25:10 am
Solution 2 does sound like the best of both worlds.
Title: Re: The new Lua SEXP interface
Post by: m!m on April 08, 2017, 03:52:06 am
I implemented the second solution now and it's ready for testing. Here are some test builds (https://bintray.com/scp-fs2open/FSO/download_file?file_path=luaSEXP.7z) (64-bit SSE2) and a test mod (https://bintray.com/scp-fs2open/FSO/download_file?file_path=LuaSEXPs.7z) which shows how the table works.

I also added support for specifying a category and subcategory which includes support for adding new subcategories but that feature is not included in the test builds yet. I'll compile new builds later today.
Title: Re: The new Lua SEXP interface
Post by: Axem on April 09, 2017, 09:29:05 am
I just want to say this is amazing and I'm going to be playing with this a lot. :)

Is there a list of what parameter types are allowed?
Title: Re: The new Lua SEXP interface
Post by: m!m on April 09, 2017, 10:28:42 am
The source is the only reliable list right now, I think the list should be easy enough to read: https://github.com/asarium/fs2open.github.com/blob/feature/scriptedSEXPs/code/parse/sexp/LuaSEXP.cpp#L13

Once this gets merged I will write a detailed wiki article about how to use it properly which will include that type list.
Title: Re: The new Lua SEXP interface
Post by: m!m on April 10, 2017, 05:41:16 am
Here are some new test builds (https://bintray.com/scp-fs2open/FSO/download_file?file_path=LuaSEXPs.7z), now with support for specifying the category and subcategory the SEXP should be shown in. I also uploaded a new test mod (https://bintray.com/scp-fs2open/FSO/download_file?file_path=testMod.7z).

Categories must be chosen from the existing list in FRED. New categories can't be added due to how the FRED category system works at the moment. The subcategory can be one of the existing ones or a new one. Just chose an unused name if you want to add a new subcategory. Again, due to how the subcategories are implemented in FSO, you can not reuse the name of a subcategory under another parent category.
Title: Re: The new Lua SEXP interface
Post by: m!m on April 15, 2017, 10:36:07 am
Another batch of test builds (https://bintray.com/scp-fs2open/FSO/download_file?file_path=test.7z)! I added support for SEXP variable parameters. Now the type "variable" can be selected as a parameter type which allows to accept a variable reference. The scripting functions is passed a SEXP variable handle for this parameter which can be used to read and write the value of the variable. The test mod stays the same since I accidentally already included the variable data in that version.
Title: Re: The new Lua SEXP interface
Post by: Axem on April 16, 2017, 05:48:24 pm
I've been playing around with this and I must confirm: this is pretty cool. This is going to make scripts so much easier for FREDders to use. And now I don't need coders to make my sexps, I can make them for me!

*Axem immediately adds 11MB worth of new sexps
Title: Re: The new Lua SEXP interface
Post by: Bryan See on May 06, 2017, 06:41:08 am
Excuse me, guys.

What about the ability to override existing SEXPs?

For example, I want to augment the cutscene SEXPs so that camera movements will be reminiscent to recent movies and games - thus improving cinematic experience of in-mission cutscenes.
Title: Re: The new Lua SEXP interface
Post by: m!m on May 06, 2017, 06:44:00 am
Existing SEXPs will always override SEXPs added by this new system. You cannot replace a built-in SEXP with a Lua SEXP since that could cause a lot of problems.
Title: Re: The new Lua SEXP interface
Post by: Bryan See on May 06, 2017, 07:04:50 am
That means if I want to override them without actually replacing built-in SEXPs with Lua SEXPs, I just have to create them to complement them.
Title: Re: The new Lua SEXP interface
Post by: m!m on May 06, 2017, 07:06:09 am
If you mean by that that you need to create a new SEXP with a different name and then use that in your missions then yes, that's how you need to do it.
Title: Re: The new Lua SEXP interface
Post by: Bryan See on May 06, 2017, 02:12:24 pm
I got it.

Can these SEXPS may include built-in SEXPs as well as some Lua code?
Title: Re: The new Lua SEXP interface
Post by: xenocartographer on May 06, 2017, 02:27:52 pm
mn.evaluateSEXP springs to mind.
Title: Re: The new Lua SEXP interface
Post by: m!m on December 21, 2017, 05:13:24 am
This feature has been merged into the master branch and the newest nightly already contains the new code. I added documentation for how this works to the wiki: http://wiki.hard-light.net/index.php/Dynamic_SEXPs
Title: Re: The new Lua SEXP interface
Post by: Axem on December 21, 2017, 07:13:49 am
Quote
ship
This parameter is the name of a ship. The script will receive this value as a string which is the ship name. mn.Ships can be used for getting a ship handle from the name. The name may be invalid if the ship does not exist or is no longer present in the mission.
...
team
This parameter is a handle to a specific team. The script will receive this value as a team handle.
waypointpath
This parameter is a reference to a specific waypoint path. The script will receive this as a waypointlist handle.
variable
This is a reference to a SEXP variable. This is a reference to the actual variable and not its contents. The script will receive this as a sexpvariable handle.

Is there a reason Ship gets passed as a string, but everything else is passed as their handle?
Title: Re: The new Lua SEXP interface
Post by: m!m on December 21, 2017, 08:14:05 am
Quote
The name may be invalid if the ship does not exist or is no longer present in the mission.
That is the reason :p

Ships are (so far) the only parameter type that can be invalid when the SEXP function is called. All other types will always be valid which means that they can be converted to their Lua handles without worrying about producing an invalid handle. Also, when the script receives the ship name as a string it can do more with an invalid name than with an invalid handle. The name can at least be used for displaying a helpful message which includes the ship name. If the script only received the invalid ship handle then all it could do would be to determine that the handle is invalid and nothing more since the invalid ship handle only has that one bit of useful information.

I hope that clears up my reason for passing this as a string.

EDIT: I changed the description of the ship type a bit to explain why it is a string.
Title: Re: The new Lua SEXP interface
Post by: Axem on December 21, 2017, 05:21:35 pm
Hmmm, okay. Buuuttt...

Through use of arguments/variables and unfortunate typos/cross mod files without paying attention, one could throw an invalid handle for teams or waypoint paths.

And also the standard sexp behavior for being given an invalid ship reference is just to silently fail. "self-destruct a ship that's departed? It couldn't be helped." The scripter has the power with :isValid() to check ship object validity and if it can't work they should either output a debug print saying "this function couldn't work" or just silently fail.

Now I super agree that it would be great to be able to know which object failed so it can be narrowed down a lot faster, but I also think things should be consistent. Intuitive behavior is more important I think than syntactical sugar. Either method is fine by me, just as long as its the same for every parameter. Just my two cents anyway!
Title: Re: The new Lua SEXP interface
Post by: AdmiralRalwood on December 21, 2017, 05:26:33 pm
Intuitive behavior is more important I think than syntactical sugar.
Seconded. If being able to know the name of the object is so useful in one case, it might be useful in all cases.
Title: Re: The new Lua SEXP interface
Post by: m!m on December 22, 2017, 02:09:30 am
Well, ok then. I though it would be better to have more information in case of an invalid ship name but I don't really have a preference here so I will just change it to a ship handle.

EDIT: I made the changes. PR: https://github.com/scp-fs2open/fs2open.github.com/pull/1539
Title: Re: The new Lua SEXP interface
Post by: xenocartographer on December 22, 2017, 09:25:21 am
I read that more as "why not pass everything though strings, cos someone's going to mistakenly refer to Gramma wing sometime".
Title: Re: The new Lua SEXP interface
Post by: m!m on January 01, 2018, 12:50:00 pm
In case people look here for documentation, I made a wiki article about how to use this new system: http://wiki.hard-light.net/index.php/Dynamic_SEXPs

EDIT: Oops, I though I hadn't posted that link here yet. Well, it's better to have it twice I guess :nervous:
Title: Re: The new Lua SEXP interface
Post by: karajorma on January 01, 2018, 10:55:37 pm
For the naming section, you might want to strongly recommend prefixing the names of SEXPs with lua- so that there is no chance of a SEXP being added later by a coder which has the same name. The example you use of cloak-disable is a good example of something which could later be added by a coder and if the lua version has different parameters to the script one it would break all the missions using the scripted version.
Title: Re: The new Lua SEXP interface
Post by: xenocartographer on January 02, 2018, 09:29:05 am
There seems to be a mismatch between the documentation and the Lua example:

Quote
ship
This parameter is the name of a ship. The script will receive this value as a ship handle. If the name provided by the mission is invalid (if the ship has not arrived yet or if it is not present anymore) then the script will receive an invalid ship handle.
[...]
variable
This is a reference to a SEXP variable. This is a reference to the actual variable and not its contents. The script will receive this as a sexpvariable handle.

But, the Lua snippet looks up the ship/variable by name:

Code: [Select]
ba.print(mn.Ships[arg2].Name)
ba.print(tostring(mn.SEXPVariables[arg3].Value))

It seems like one of these sections needs to be corrected to make them line up.
Title: Re: The new Lua SEXP interface
Post by: m!m on January 02, 2018, 12:31:06 pm
I forgot to fix that when the changes were merged. I don't know how the SEXP variable lookup even got in there. That was never necessary but I fixed that as well...