Hard Light Productions Forums
Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: m!m on April 05, 2017, 04:00:52 am
-
Dynamic SEXPs is a feature I implemented recently which allows to add new SEXPs at runtime and have them properly integrated into FSO and FRED. The code for that is done (here is the pull request (https://github.com/scp-fs2open/fs2open.github.com/issues/1320)) but now I need to decide how that feature should be integrated into the modding workflow. There are multiple possible ways of exposing this functionality to modders and I would like to know which version would be preferred by the modding community:
- A pure Lua solution: This only exposes a Lua API for adding the new SEXPs. This has the advantage that we don't need to introduce a new table format just form specifying how the new system works. Since this new system requires knowledge of Lua coding it doesn't limit the viability of the system since anyone who want to add a new SEXP also has to know how to write Lua code. You can find an example of this API below. Since this Lua code has to be executed by FRED I needed to add Lua scripting support to FRED which requires some additional care from script writers since FRED works differently than FSO. In order to not break existing scripts, I added a mod_table flag for enabling Lua scripting in FRED.
- A table with a Lua API: This solution would use a new table format for specifying which dynamic SEXPs are available (I don't have an example since this hasn't been implemented yet). Then, a Lua script would use a new API for setting the actual function that should be executed for that SEXP. That might look like this: mn.DynamicSEXPs["my-awesome-sexp"].Action = function() ba.error("It's Axems fault!") end. That keeps the required Lua code to a minimum and would probably also improve error handling since the table validation of the first solution is pretty bad at the moment.
- A table with inline Lua code: This is similar to the second solution but instead of adding the action code through a Lua API it would be specified in the table file. I think this is not a good solution since it requires that more Lua values are stored in the global namespace which can lead to conflicts between different scripts. Parsing the Lua function inside the table is also not easy to do since it would require some special structure from the executed Lua code which would be easy to get wrong. The reason for that is that every piece of Lua code gets compiled to a function with no parameters. In order to get the actual action function out of that, it needs to be returned from that function which requires special error handling code which is why I don't like this solution.
These are the solutions I can currently think of. If you have any suggestions for improvements or alternative solutions I would like to hear them.
local params = {
help = "This is a dynamic parameter test SEXP",
min_args = 1,
max_args = math.huge,
return_type = "void",
params = {
["..."] = {
description = "Some arguments",
type = "boolean"
},
{
description = "Test",
type = "number"
},
{
description = "Test2",
type = "ship"
};
},
}
mn.addSEXP("dynamic-sexp", params, function(arg1, arg2, ...)
ba.error(tostring(arg1) .. " " .. arg2)
end)
-
Hmm, I'm not sure what would be best, from a modder's standpoint.
I would be OK with a pure lua solution, since we could shove the new SEXPs into a *-sct.tbm.
However, I really like the way tables neatly structure their data, are automatically parsed when the game/FRED starts. I would also make it easier to read for modder unfamiliar with lua cracking open a mod looking for interesting stuff to grab. Incidentally, I'm not fond of having inline lua code, I feel it would unnecessarily crowd the file.
Table spec could look something like this
#Dynamic SEXPs
;; spec
$Name: String
$Min Args: Integer
$Max Args: Integer --> Optional, default to any number
$Return type: Type --> Optional, default to action SEXP
$Description: String
$Function: String --> name of the lua function that executes the SEXP
$Parameter
+Description: String
+Type: Type
;; example
$Name: my-awesome-sexp
$Min Args: 1
$Max Args: 3
$Description: That's an awesome SEXP
$Function: myAwesomeSexp
$Parameter
+Description: That's a string param
+Type: String
$Parameter
+Description: That's a bool
+Type: Boolean
$Parameter
+Description: That's a number
+Type: Number
#End
-
That's a good specification but $Function: will not work that way. I dislike the usage of global Lua values in general and I will not build a system which requires global values to function. Also, due to how Lua works, it's impractical to specify which function to call that way since the named function does not exist at the time when the table is parsed which means that the name must be resolved every time the SEXP is called which is unnecessary overhead.
-
Anything stopping you from caching the value, m!m?
I don't really think that avoiding globals is a worthwhile concern here, but if you really care, you could use a table as a namespace, analogous to static classes in Java. It doesn't entirely eliminate the risk of name collisions, but if you request (or enforce) naming functions after the SEXP itself, it's no worse than it would be anyway.
-
Caching could be done but I am opposed to using global functions in the first place because it's bad design to use it like that if we have a system that is perfectly capable of handling functions as parameters which allows us to use something like my example above (mn.DynamicSEXPs["my-awesome-sexp"].Action = function() ...).
If everyone agrees that the global function name is the only solution that will ever work then I'll use it but at this point I don't see why it would be. Global functions complicate the implementation tremendously and lead to bad design decisions so I would like to avoid them wherever possible.
-
Okay, I'm not getting the hatred of globals, but what about something like this?
$Function: [
function someReservedName(foo, bar, baz)
-- implementation
end
]
? After evaluating the inline snippet, you'd register the function with that reserved name and set that name to nil. It doesn't require a global anything (well, okay, I guess it does for a handful of machine cycles, but eh), and you can immediately validate that the function exists (and possibly even that it has the correct arity).
Oh, would these dynamic SEXPs have access to the rest of the SEXP tree? I can think of some use cases for defining custom conditionals, stuff like that.
EDIT: @X3NO: I like your table spec as well, but I think you overlooked truly varardic SEXPs. How'd I specify something like is-destroyed-delay (which can take a theoretically infinite number of string arguments) or send-message-list (which can take a theoretically infinite number of arguments, but only in a certain pattern)? Maybe a +Repeat: suboption...? If there are fewer $Parameters: than specified in $Max Args:, any +Repeat: parameters are repeated, in order, to fill it out?
-
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.
Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)
-
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.
Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)
I think table-with-Lua-API is actually quite good in that regard: you define the SEXP interface in a table and present an interface which FRED-and-SEXPers can use without touching Lua at all, then put the backend scripts somewhere else.
-
But shouldn't making that bridge be the responsibility of the scripter? Ideally the FREDder wouldn't have to do anything with these files except plop them in their modding directory.
-
Well the scripter would provide both the backend script files and the table files defining the interface, the idea is that a FREDder would just have to look at the table to read which SEXPs were provided and what they did without having to try to parse that out of the Lua code.
-
At first I was in favor of a more traditional table based solution, but I think I'm more in favor of a lua based one now.
Scripting is a whole new level of FS modding, so it might be more confusing to have something that's half FS2 table/half lua instead of a pure lua solution. Keeping things consistent is think is key. The scripters making the "interface" are going to have to know enough about lua and how scripting works in FSO anyway. (So option 1 or 2 is cool with me)
Hm, I can see this, actually (especially with how controversial implementing a hybrid system looks like it might be :P ). Another side benefit is that it may be easier to convert existing scripts into SEXPs if desired. I think PH's concern is handled pretty well by FRED's in-editor documentation, so as long as the script author writes good docs, you shouldn't have to look at the Lua source even if that is where the interface is defined.
-
PH: Going by with what I've seen in that initial github discussion and PR, the dynamic sexps get put into FRED's sexp menu, complete with the documentation provided in the definition. So FREDders using the dynamic sexps wouldn't need to open up any files to see what's available, it'd look like any other sexp to them. So that's why in this case I think it's okay to be a little more "technical" than "friendly".
-
Okay, I'm not getting the hatred of globals, but what about something like this?
My primary reason why I dislike globals is that they can cause conflicts with other scripts. If I could, I would refactor the entire scripting system so that each scripting block has its own namespace but that's not going to happen anytime soon. I don't want to put the script writers into a position where they have to think about the name of their function because it might overwrite a function from a different script. When I wrote the scripting SEXPs, that was my reason for not using globals in any way since I think that they only cause issues later on.
Oh, would these dynamic SEXPs have access to the rest of the SEXP tree? I can think of some use cases for defining custom conditionals, stuff like that.
What do you mean by that? If you mean that you can use other SEXPs as parameters for the dynamic SEXP then the answer is yes but more advanced features like custom conditional nodes are not included in the current version.
I am currently in favor of using the second alternative in my list above since a table based solution would improve error handling tremendously.
-
Solution 2 does sound like the best of both worlds.
-
I implemented the second solution now and it's ready for testing. Here are some test builds (https://bintray.com/scp-fs2open/FSO/download_file?file_path=luaSEXP.7z) (64-bit SSE2) and a test mod (https://bintray.com/scp-fs2open/FSO/download_file?file_path=LuaSEXPs.7z) which shows how the table works.
I also added support for specifying a category and subcategory which includes support for adding new subcategories but that feature is not included in the test builds yet. I'll compile new builds later today.
-
I just want to say this is amazing and I'm going to be playing with this a lot. :)
Is there a list of what parameter types are allowed?
-
The source is the only reliable list right now, I think the list should be easy enough to read: https://github.com/asarium/fs2open.github.com/blob/feature/scriptedSEXPs/code/parse/sexp/LuaSEXP.cpp#L13
Once this gets merged I will write a detailed wiki article about how to use it properly which will include that type list.
-
Here are some new test builds (https://bintray.com/scp-fs2open/FSO/download_file?file_path=LuaSEXPs.7z), now with support for specifying the category and subcategory the SEXP should be shown in. I also uploaded a new test mod (https://bintray.com/scp-fs2open/FSO/download_file?file_path=testMod.7z).
Categories must be chosen from the existing list in FRED. New categories can't be added due to how the FRED category system works at the moment. The subcategory can be one of the existing ones or a new one. Just chose an unused name if you want to add a new subcategory. Again, due to how the subcategories are implemented in FSO, you can not reuse the name of a subcategory under another parent category.
-
Another batch of test builds (https://bintray.com/scp-fs2open/FSO/download_file?file_path=test.7z)! I added support for SEXP variable parameters. Now the type "variable" can be selected as a parameter type which allows to accept a variable reference. The scripting functions is passed a SEXP variable handle for this parameter which can be used to read and write the value of the variable. The test mod stays the same since I accidentally already included the variable data in that version.
-
I've been playing around with this and I must confirm: this is pretty cool. This is going to make scripts so much easier for FREDders to use. And now I don't need coders to make my sexps, I can make them for me!
*Axem immediately adds 11MB worth of new sexps
-
Excuse me, guys.
What about the ability to override existing SEXPs?
For example, I want to augment the cutscene SEXPs so that camera movements will be reminiscent to recent movies and games - thus improving cinematic experience of in-mission cutscenes.
-
Existing SEXPs will always override SEXPs added by this new system. You cannot replace a built-in SEXP with a Lua SEXP since that could cause a lot of problems.
-
That means if I want to override them without actually replacing built-in SEXPs with Lua SEXPs, I just have to create them to complement them.
-
If you mean by that that you need to create a new SEXP with a different name and then use that in your missions then yes, that's how you need to do it.
-
I got it.
Can these SEXPS may include built-in SEXPs as well as some Lua code?
-
mn.evaluateSEXP springs to mind.
-
This feature has been merged into the master branch and the newest nightly already contains the new code. I added documentation for how this works to the wiki: http://wiki.hard-light.net/index.php/Dynamic_SEXPs
-
ship
This parameter is the name of a ship. The script will receive this value as a string which is the ship name. mn.Ships can be used for getting a ship handle from the name. The name may be invalid if the ship does not exist or is no longer present in the mission.
...
team
This parameter is a handle to a specific team. The script will receive this value as a team handle.
waypointpath
This parameter is a reference to a specific waypoint path. The script will receive this as a waypointlist handle.
variable
This is a reference to a SEXP variable. This is a reference to the actual variable and not its contents. The script will receive this as a sexpvariable handle.
Is there a reason Ship gets passed as a string, but everything else is passed as their handle?
-
The name may be invalid if the ship does not exist or is no longer present in the mission.
That is the reason :p
Ships are (so far) the only parameter type that can be invalid when the SEXP function is called. All other types will always be valid which means that they can be converted to their Lua handles without worrying about producing an invalid handle. Also, when the script receives the ship name as a string it can do more with an invalid name than with an invalid handle. The name can at least be used for displaying a helpful message which includes the ship name. If the script only received the invalid ship handle then all it could do would be to determine that the handle is invalid and nothing more since the invalid ship handle only has that one bit of useful information.
I hope that clears up my reason for passing this as a string.
EDIT: I changed the description of the ship type a bit to explain why it is a string.
-
Hmmm, okay. Buuuttt...
Through use of arguments/variables and unfortunate typos/cross mod files without paying attention, one could throw an invalid handle for teams or waypoint paths.
And also the standard sexp behavior for being given an invalid ship reference is just to silently fail. "self-destruct a ship that's departed? It couldn't be helped." The scripter has the power with :isValid() to check ship object validity and if it can't work they should either output a debug print saying "this function couldn't work" or just silently fail.
Now I super agree that it would be great to be able to know which object failed so it can be narrowed down a lot faster, but I also think things should be consistent. Intuitive behavior is more important I think than syntactical sugar. Either method is fine by me, just as long as its the same for every parameter. Just my two cents anyway!
-
Intuitive behavior is more important I think than syntactical sugar.
Seconded. If being able to know the name of the object is so useful in one case, it might be useful in all cases.
-
Well, ok then. I though it would be better to have more information in case of an invalid ship name but I don't really have a preference here so I will just change it to a ship handle.
EDIT: I made the changes. PR: https://github.com/scp-fs2open/fs2open.github.com/pull/1539
-
I read that more as "why not pass everything though strings, cos someone's going to mistakenly refer to Gramma wing sometime".
-
In case people look here for documentation, I made a wiki article about how to use this new system: http://wiki.hard-light.net/index.php/Dynamic_SEXPs
EDIT: Oops, I though I hadn't posted that link here yet. Well, it's better to have it twice I guess :nervous:
-
For the naming section, you might want to strongly recommend prefixing the names of SEXPs with lua- so that there is no chance of a SEXP being added later by a coder which has the same name. The example you use of cloak-disable is a good example of something which could later be added by a coder and if the lua version has different parameters to the script one it would break all the missions using the scripted version.
-
There seems to be a mismatch between the documentation and the Lua example:
ship
This parameter is the name of a ship. The script will receive this value as a ship handle. If the name provided by the mission is invalid (if the ship has not arrived yet or if it is not present anymore) then the script will receive an invalid ship handle.
[...]
variable
This is a reference to a SEXP variable. This is a reference to the actual variable and not its contents. The script will receive this as a sexpvariable handle.
But, the Lua snippet looks up the ship/variable by name:
ba.print(mn.Ships[arg2].Name)
ba.print(tostring(mn.SEXPVariables[arg3].Value))
It seems like one of these sections needs to be corrected to make them line up.
-
I forgot to fix that when the changes were merged. I don't know how the SEXP variable lookup even got in there. That was never necessary but I fixed that as well...