Author Topic: The new Lua SEXP interface  (Read 885 times)

0 Members and 1 Guest are viewing this topic.

Offline m!m

  • 210
The new Lua SEXP interface
Dynamic SEXPs is a feature I implemented recently which allows to add new SEXPs at runtime and have them properly integrated into FSO and FRED. The code for that is done (here is the pull request) but now I need to decide how that feature should be integrated into the modding workflow. There are multiple possible ways of exposing this functionality to modders and I would like to know which version would be preferred by the modding community:

  • A pure Lua solution: This only exposes a Lua API for adding the new SEXPs. This has the advantage that we don't need to introduce a new table format just form specifying how the new system works. Since this new system requires knowledge of Lua coding it doesn't limit the viability of the system since anyone who want to add a new SEXP also has to know how to write Lua code. You can find an example of this API below. Since this Lua code has to be executed by FRED I needed to add Lua scripting support to FRED which requires some additional care from script writers since FRED works differently than FSO. In order to not break existing scripts, I added a mod_table flag for enabling Lua scripting in FRED.
  • A table with a Lua API: This solution would use a new table format for specifying which dynamic SEXPs are available (I don't have an example since this hasn't been implemented yet). Then, a Lua script would use a new API for setting the actual function that should be executed for that SEXP. That might look like this: mn.DynamicSEXPs["my-awesome-sexp"].Action = function() ba.error("It's Axems fault!") end. That keeps the required Lua code to a minimum and would probably also improve error handling since the table validation of the first solution is pretty bad at the moment.
  • A table with inline Lua code: This is similar to the second solution but instead of adding the action code through a Lua API it would be specified in the table file. I think this is not a good solution since it requires that more Lua values are stored in the global namespace which can lead to conflicts between different scripts. Parsing the Lua function inside the table is also not easy to do since it would require some special structure from the executed Lua code which would be easy to get wrong. The reason for that is that every piece of Lua code gets compiled to a function with no parameters. In order to get the actual action function out of that, it needs to be returned from that function which requires special error handling code which is why I don't like this solution.

These are the solutions I can currently think of. If you have any suggestions for improvements or alternative solutions I would like to hear them.


Code: (Lua API example) [Select]
local params = {
help = "This is a dynamic parameter test SEXP",
min_args = 1,
max_args = math.huge,
return_type = "void",
params = {
["..."] = {
description = "Some arguments",
type = "boolean"
},

{
description = "Test",
type = "number"
},
{
description = "Test2",
type = "ship"
};
},
}
mn.addSEXP("dynamic-sexp", params, function(arg1, arg2, ...)
ba.error(tostring(arg1) .. " " .. arg2)
end)

 
Re: The new Lua SEXP interface
Hmm, I'm not sure what would be best, from a modder's standpoint.

I would be OK with a pure lua solution, since we could shove the new SEXPs into a *-sct.tbm.

However, I really like the way tables neatly structure their data, are automatically parsed when the game/FRED starts. I would also make it easier to read for modder unfamiliar with lua cracking open a mod looking for interesting stuff to grab. Incidentally, I'm not fond of having inline lua code, I feel it would unnecessarily crowd the file.


Table spec could look something like this
Code: [Select]
#Dynamic SEXPs
;; spec
$Name:           String
$Min Args:       Integer
$Max Args:      Integer --> Optional, default to any number
$Return type:  Type --> Optional, default to action SEXP
$Description:   String
$Function:       String --> name of the lua function that executes the SEXP
$Parameter
    +Description: String
    +Type:           Type

;; example
$Name:   my-awesome-sexp
$Min Args:       1
$Max Args:      3
$Description:   That's an awesome SEXP
$Function:       myAwesomeSexp
$Parameter
    +Description: That's a string param
    +Type:           String
$Parameter
    +Description: That's a bool
    +Type:           Boolean
$Parameter
    +Description: That's a number
    +Type:           Number

#End

 

Offline m!m

  • 210
Re: The new Lua SEXP interface
That's a good specification but $Function: will not work that way. I dislike the usage of global Lua values in general and I will not build a system which requires global values to function. Also, due to how Lua works, it's impractical to specify which function to call that way since the named function does not exist at the time when the table is parsed which means that the name must be resolved every time the SEXP is called which is unnecessary overhead.

 
Re: The new Lua SEXP interface
Anything stopping you from caching the value, m!m?

I don't really think that avoiding globals is a worthwhile concern here, but if you really care, you could use a table as a namespace, analogous to static classes in Java. It doesn't entirely eliminate the risk of name collisions, but if you request (or enforce) naming functions after the SEXP itself, it's no worse than it would be anyway.

 

Offline m!m

  • 210
Re: The new Lua SEXP interface
Caching could be done but I am opposed to using global functions in the first place because it's bad design to use it like that if we have a system that is perfectly capable of handling functions as parameters which allows us to use something like my example above (mn.DynamicSEXPs["my-awesome-sexp"].Action = function() ...).

If everyone agrees that the global function name is the only solution that will ever work then I'll use it but at this point I don't see why it would be. Global functions complicate the implementation tremendously and lead to bad design decisions so I would like to avoid them wherever possible.

 
Re: The new Lua SEXP interface
Okay, I'm not getting the hatred of globals, but what about something like this?

Code: [Select]
$Function: [
function someReservedName(foo, bar, baz)
-- implementation
end
]

? After evaluating the inline snippet, you'd register the function with that reserved name and set that name to nil. It doesn't require a global anything (well, okay, I guess it does for a handful of machine cycles, but eh), and you can immediately validate that the function exists (and possibly even that it has the correct arity).



Oh, would these dynamic SEXPs have access to the rest of the SEXP tree? I can think of some use cases for defining custom conditionals, stuff like that.



EDIT: @X3NO: I like your table spec as well, but I think you overlooked truly varardic SEXPs. How'd I specify something like is-destroyed-delay (which can take a theoretically infinite number of string arguments) or send-message-list (which can take a theoretically infinite number of arguments, but only in a certain pattern)? Maybe a +Repeat: suboption...? If there are fewer $Parameters: than specified in $Max Args:, any +Repeat: parameters are repeated, in order, to fill it out?
« Last Edit: April 05, 2017, 12:12:10 pm by xenocartographer »

 

Offline Axem

  • Administrator
  • 211
Re: The new Lua SEXP interface
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.

Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)

 
Re: The new Lua SEXP interface
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.

Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)

I think table-with-Lua-API is actually quite good in that regard: you define the SEXP interface in a table and present an interface which FRED-and-SEXPers can use without touching Lua at all, then put the backend scripts somewhere else.
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.

 

Offline Axem

  • Administrator
  • 211
Re: The new Lua SEXP interface
But shouldn't making that bridge be the responsibility of the scripter? Ideally the FREDder wouldn't have to do anything with these files except plop them in their modding directory.

 
Re: The new Lua SEXP interface
Well the scripter would provide both the backend script files and the table files defining the interface, the idea is that a FREDder would just have to look at the table to read which SEXPs were provided and what they did without having to try to parse that out of the Lua code.
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.

 
Re: The new Lua SEXP interface
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.

Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)

Hm, I can see this, actually (especially with how controversial implementing a hybrid system looks like it might be :P ). Another side benefit is that it may be easier to convert existing scripts into SEXPs if desired. I think PH's concern is handled pretty well by FRED's in-editor documentation, so as long as the script author writes good docs, you shouldn't have to look at the Lua source even if that is where the interface is defined.

 

Offline Axem

  • Administrator
  • 211
Re: The new Lua SEXP interface
PH: Going by with what I've seen in that initial github discussion and PR, the dynamic sexps get put into FRED's sexp menu, complete with the documentation provided in the definition. So FREDders using the dynamic sexps wouldn't need to open up any files to see what's available, it'd look like any other sexp to them. So that's why in this case I think it's okay to be a little more "technical" than "friendly".

 

Offline m!m

  • 210
Re: The new Lua SEXP interface
Okay, I'm not getting the hatred of globals, but what about something like this?
My primary reason why I dislike globals is that they can cause conflicts with other scripts. If I could, I would refactor the entire scripting system so that each scripting block has its own namespace but that's not going to happen anytime soon. I don't want to put the script writers into a position where they have to think about the name of their function because it might overwrite a function from a different script. When I wrote the scripting SEXPs, that was my reason for not using globals in any way since I think that they only cause issues later on.

Oh, would these dynamic SEXPs have access to the rest of the SEXP tree? I can think of some use cases for defining custom conditionals, stuff like that.
What do you mean by that? If you mean that you can use other SEXPs as parameters for the dynamic SEXP then the answer is yes but more advanced features like custom conditional nodes are not included in the current version.

I am currently in favor of using the second alternative in my list above since a table based solution would improve error handling tremendously.

 

Offline The E

  • He's Ebeneezer Goode
  • Global Moderator
  • 213
  • Nothing personal, just tech support.
    • Skype
    • Steam
    • Twitter
Re: The new Lua SEXP interface
Solution 2 does sound like the best of both worlds.
**** every cause that ends in murder and children crying. ― Iain Banks
Join the fun at the HLP IRC channel. Get the latest spam and gossip as long as it's fresh!

 

Offline m!m

  • 210
Re: The new Lua SEXP interface
I implemented the second solution now and it's ready for testing. Here are some test builds (64-bit SSE2) and a test mod which shows how the table works.

I also added support for specifying a category and subcategory which includes support for adding new subcategories but that feature is not included in the test builds yet. I'll compile new builds later today.

 

Offline Axem

  • Administrator
  • 211
Re: The new Lua SEXP interface
I just want to say this is amazing and I'm going to be playing with this a lot. :)

Is there a list of what parameter types are allowed?
« Last Edit: April 09, 2017, 09:36:32 am by Axem »

 

Offline m!m

  • 210
Re: The new Lua SEXP interface
The source is the only reliable list right now, I think the list should be easy enough to read: https://github.com/asarium/fs2open.github.com/blob/feature/scriptedSEXPs/code/parse/sexp/LuaSEXP.cpp#L13

Once this gets merged I will write a detailed wiki article about how to use it properly which will include that type list.

 

Offline m!m

  • 210
Re: The new Lua SEXP interface
Here are some new test builds, now with support for specifying the category and subcategory the SEXP should be shown in. I also uploaded a new test mod.

Categories must be chosen from the existing list in FRED. New categories can't be added due to how the FRED category system works at the moment. The subcategory can be one of the existing ones or a new one. Just chose an unused name if you want to add a new subcategory. Again, due to how the subcategories are implemented in FSO, you can not reuse the name of a subcategory under another parent category.

 

Offline m!m

  • 210
Re: The new Lua SEXP interface
Another batch of test builds! I added support for SEXP variable parameters. Now the type "variable" can be selected as a parameter type which allows to accept a variable reference. The scripting functions is passed a SEXP variable handle for this parameter which can be used to read and write the value of the variable. The test mod stays the same since I accidentally already included the variable data in that version.

 

Offline Axem

  • Administrator
  • 211
Re: The new Lua SEXP interface
I've been playing around with this and I must confirm: this is pretty cool. This is going to make scripts so much easier for FREDders to use. And now I don't need coders to make my sexps, I can make them for me!

*Axem immediately adds 11MB worth of new sexps