[Iss Mneur]

Have we considered using SWIG to do the interface between lua and code.lib? If so, why do we not?

[Nuke]

Something akin to...

Code: [Select]

//**********LIBRARY: Controls library
ade_lib l_HUD("HUD", NULL, "hu", "HUD library");
Perhaps?

that easy? who would have thought.

Have we considered using SWIG to do the interface between lua and code.lib? If so, why do we not?

if it can do what we have now without changing too much stuff, maybe. id only be concerned that it would make scripting a little bit more complicated, you would have to have some foreknowledge of the engine internals to write effective script. the current scripting system really doesnt require any info about the engine internals. on the other hand it would mean that we wouldnt need to write a 15 line proxy function in c for every new piece of data we needed (and possibly a smaller cpp file).

[Iss Mneur]

Have we considered using SWIG to do the interface between lua and code.lib? If so, why do we not?

if it can do what we have now without changing too much stuff, maybe. id only be concerned that it would make scripting a little bit more complicated, you would have to have some foreknowledge of the engine internals to write effective script. the current scripting system really doesnt require any info about the engine internals. on the other hand it would mean that we wouldnt need to write a 15 line proxy function in c for every new piece of data we needed (and possibly a smaller cpp file).

Yes, it will make you closer to the engine internals, and it does expose how horribly documented the some of the interfaces in the engine actually are, but it would give Lua instant access to all the systems in the code, with the same abilities to modify the engine behaviour that the C* code can.  The poor documentation of the interfaces is something that needs to be addressed in the C* code anyway, so, this may or may not spur some activity in fixing the documentation, which would then help both the C* and Lua coders at the same time. 

Honestly, we have the same problem with the SEXP system as well, a well defined interface that is visible to the FREDer, but you step behind the curtain you see the C* code that has no documentation at all. But I digress, as the documentation thing is a pandemic in the engine.

Also, SWIG wouldn't replace the current lua interface, at worst it would make it a depreciated interface.  We could also potentially re-implement the wrapper functions that are contained in lua.cpp in pure lua using the SWIG interface.  Using the pure lua library would be able to insulate the Lua coder from limited changes in the C interface. The library would also allow some more object oriented interfaces for the lua coders for some of the very C like interfaces in the engine.

I am attaching a discussion I had with The_E on the topic on #SCP (I have stripped out the off topic discussion that was going on at the same time).  Some of it repeats what I have already written here.

Code: [Select]

(2:14:15 PM) The_E: IssMneur: http://www.hard-light.net/forums/index.php?topic=44821.msg1408106#msg1408106
(2:14:18 PM) The_E: Errrm
(2:14:55 PM) The_E: We haven't considered it because, for all intents and purposes, lua is part of code.lib already.
(2:16:47 PM) IssMneur: well, not really, its in code.lib, but its not part of code.lib
(2:17:08 PM) IssMneur: lua is basically in the house but looking from the side of the room
(2:17:36 PM) The_E: IssMneur: Point is, why introduce another indirection layer between lua and the rest?
(2:19:01 PM) IssMneur: The_E: it wouldn't add another indirection layer, it would replace the current indirection layer
(2:19:40 PM) The_E: Which layer would that be, then?
(2:20:27 PM) IssMneur: The_E: the layer that we call lua.cpp
(2:21:14 PM) The_E: IssMneur: And what net benefit would we gain from replacing it?
(2:23:06 PM) IssMneur: "automatic"  access to all interfaces in the engine. You would have to enumerate the functions or tell swig to include the header files that are used as external interfaces.
(2:23:29 PM) IssMneur: but you would be able to use the same functions and the same code as the rest of the code does
(2:24:13 PM) The_E: Hmm
(2:24:16 PM) The_E: Interesting
(2:24:39 PM) The_E: No more dicking around with arcane hook variables etc
(2:24:44 PM) IssMneur: yep
(2:25:03 PM) The_E: Of course, that would also mean breaking all existing scripts if we introduced this, yes?
(2:25:12 PM) IssMneur: lua would become a first class interface just like the c code.
(2:25:32 PM) The_E: Unless we can find a way to preserve the current interface as well
(2:25:46 PM) IssMneur: ya, though, we could always just abandon the current interface as a "legacy" interface
(2:26:17 PM) The_E: Yes, of course. I just don't see removing that legacy interface as being an option
(2:26:36 PM) The_E: Too many big mods are already depending on it
(2:26:41 PM) IssMneur: and change the scripting hooks to cover it either at the .tbl level or at the lua level by putting all of the new stuff into namespaces
(2:31:09 PM) IssMneur: The_E: do you mind if I copy this discussion to the my post?
(2:31:24 PM) The_E: Not a problem
(2:31:29 PM) The_E: Hmm
(2:31:33 PM) The_E: One other thing
(2:31:59 PM) IssMneur: ?
(2:32:22 PM) The_E: The current lua interface is very good for us, because it includes documentation for the lua API, which can be spewed to file using -output_scripting
(2:33:38 PM) The_E: Without something like that, you'll just end up with lots and lots of wrapped, but undocumented functions
(2:33:52 PM) IssMneur: ya, basically
(2:34:05 PM) The_E: That's terribad
(2:35:44 PM) The_E: Also, nuke just raised a very good point
(2:35:51 PM) IssMneur: but because the functions are the same functions that the C code uses the documentation for them would be exactly the same, and the C functions that don't have any documentation need documentation written for them anyway.  With swig you are killing two birds (documenting the C/C++ interface, and the lua interface at the same time)


[chief1983]

* chief1983 is fan

[WMCoolmon]

I see SWIG as being very useful if you've got an engine with a well-designed API that's meant to be neat and orderly. The organization and the time and effort you spent to put it there would be carried out to all of the other languages you could write in.

A core point of the Lua interface was to make it so that any modder could work in a sandbox and not have to worry about breaking the engine. The ideal being, there would be no way for the scripter to actually crash the engine from within the scripting interface. They could make the engine do crazy things, but never have the engine just die.

Another big goal was to try and make everything well-thought-out and organized and provide a big buffer between the state of the actual code, and the way the Lua interface is. The idea being that the Lua interface would let the scripter specify what they wanted to do, whereas the actual how of it is hidden and irrelevant to the scripter.

This allows the modder to work in a safe, well-defined environment. It also allows the coder to either work in a safe, well-defined environment or a cutthroat no-holds-barred optimization-is-king environment. It also allows a coder to rip out and rewrite entire sections of the engine and not have to worry about breaking backwards compatibility. As long as the Lua/SEXP/tbl interfaces stay the same, it doesn't matter what the code is doing.

Eliminating that barrier and making the scripting interface directly implemented from the C/++ interface forces coders to not just worry about backwards compatibility in the behavior of functions, but also backwards compatibility in what the functions specify. If redesigning a system would involve changing the arguments of a function or removing a function entirely then that redesign cannot be done, or you end up with a lot of legacy code sitting around and start polluting the native Freespace interfaces with wrapper functions for old ways of doing things.

So Lua is meant to be a second-line interface...but that is/was keeping in mind that it is an open source project and you can always download the source code online if you need access to the first-generation interface.

That being said one unused feature of the way Lua is implemented now is that it's abstracted behind an interface that tries to hide the rest of the code from understanding that it's actually Lua running, and not Python or some other language. The hooks that exist should have a variable that specifies what language is being used to redirect things to the proper subset (like python.cpp instead of lua.cpp). See all the script_state and script_system stuff.

[WMCoolmon]

That being said, what SWIG is doing is a pretty interesting/impressive undertaking. It's just that, put plainly, I don't think the vast majority of the FS2_Open codebase internals are designed well enough that they can be exported to other languages without making the scripting interface a lot more unreliable/dangerous, a lot more complicated/less organized, and most importantly, 100% sensitive to changes in the FS2Open codebase.

I mean, if I understand correctly, if you extended stuff_string into the SWIG namespace, you would then have to keep the same arguments for stuff_string for all time. Every code change would have to audit the scripting interface in order to ensure that the function wasn't going to get automatically exported into scripting.

[The E]

Upon consideration, I would agree. SWIG, as impressive a piece of work as it is, is really better suited for scenarios where you can build an engine with scripting support from the ground up. The current abstraction level we have works out rather well, as we can insulate the engine from screwy data rather easily.

[General Battuta]

That being said, what SWIG is doing is a pretty interesting/impressive undertaking. It's just that, put plainly, I don't think the vast majority of the FS2_Open codebase internals are designed well enough that they can be exported to other languages without making the scripting interface a lot more unreliable/dangerous, a lot more complicated/less organized, and most importantly, 100% sensitive to changes in the FS2Open codebase.

I mean, if I understand correctly, if you extended stuff_string into the SWIG namespace, you would then have to keep the same arguments for stuff_string for all time. Every code change would have to audit the scripting interface in order to ensure that the function wasn't going to get automatically exported into scripting.

uh

bwuh

whoa

hi

welcome back

[WMCoolmon]

Thanks.

I am not sure how back I'll be. I've looked at some of the stuff that's going on and it looks like things have really changed in some (good) ways, and there are still things that people had been waiting on for years a couple years back that people are still waiting on now.

I guess this is most memorable to me because it's the largest thing I wrote (as a single system anyway) and has ended up getting a lot of attention (it looks like ~2000 lines of code were added to the file since I last committed!)

'Course the downside is that I took a long time to implement some things because I knew (or at least I believe) it is critical to carefully implement stuff in scripting or else the level of expertise involved in making changes will rival that of the codebase.

I guess I can't really say it's unexpected since I did leave without ever writing up what I had in my head as being guidelines for writing scripting objects/functions/variables but figuring out the patterns from what's there is extremely difficult. Plus not everyone will agree without explanation, and even after explanation there will probably be people who disagree.

Anyway, this is my soapbox for this week. I am glad to see that things have survived and even grown for so long.

[WMCoolmon]

Splitting this since SWIG is a big enough proposal to deserve its own topic (and not really a feature request so much as an overhaul).