Hard Light Productions Forums
Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: Wanderer on January 27, 2010, 02:23:58 am
-
Something suggested by blowfish...
Should Flashy Deaths script (http://www.hard-light.net/wiki/index.php/Script_-_Flashy_Deaths_%28new%29) be included to the code? As some people seemed to think it might be useful and easier to handle as a coded version. That is it uses separate data files and has rudimentary parser. Scripting probably adds some overhead so it presumably would run faster as a c/c++ coded version. Also direct code integration would allow for more accurate controls for certain effects.
TBH I'm not terribly fond of the idea but then again I'm really not against it either.
-
My view is that whatever can be done reasonably with scripting should be done in scripting, leaving the engine C/C++ code as an interface to the engine's exposed features.
There is a burden associated with increasing the size of the C/C++ codebase, and it would appear at present that this feature can be implemented adequately in scripting.
-
But like he said, performance of this feature is probably not the best in scripting. I'm kind of the opposite, a lot of stuff that can be done in scripting is very clunky, and support for it in the engine would make things a lot smoother sometimes.
-
I would say yes - it should be moved to code.
It's a graphics-intensive operation, and is designed to be configured by a table file. (Albeit currently a slightly different format)
- It would fit nicely into fireball.tbl, as modifiers to custom fireballs.
-
i see a lot of script used for prototyping, and then having working stuff re-written in c to improve performance. like that particle thing i did awhile back. if one looks at the numer of tables a modern mod has compaired to a retail mod, youl notice a major increase in the amount of stuff. you could call it bloat. scripting can solve this, but i think it needs to be streamlined more and made more user friendly.
frequently used systems, like parsing and metadata and input so far seem like major areas for improvement. these three systems are probably more than half of the lua used in fsrts, atmospheric flight, and advanced cockpits scripts. i usually include these systems when i start a new scripting project. if they were a part of the scripting system it would improve the system greatly and reduce the amount of lua we need to write to add a basic feature.
parsing and metadata could have their interfaces combined. meta pertaining to a ship class could be parsed directly from the ships.tbl and would be accessible via the shipclass handle. meta pertaining to a specific ship could be parsed from a mission file or created at init and modified at runtime. current meta systems have to make frequent calls to a scripted lookup function that looks for a ship with a specific object signature, if instead i could just look up this data directly in the handle to that object, it would make things much faster. most scripts with gameplay features seem to revolve around object derivitaves and their handles, and often require static class data (like how fast this kind of object can move or turn) and dynamic object instance data (how fast the particular object is moving or turning), while most of this data is available through handles, it doenst do much for you if the data is something that you need to define. essentially you should be able to stick an arbitrary piece data into the handle, and then retrieve it, modify it, put it back in, and look at it in some other part of the code, ect. by integrating the meta system in this way, it would take care of both meta and parsing.
input could use the greatest upgrade. the big issue is that input needs to be abstracted down from direct access meathods we currently have (key hooks, mouse hooks, controls library functions), which are directly associated to a physical button or mouse axis, and reduce it to a command level interface, where the control that is bound to the command is irrelevant. within the script, if you wanted to check on the state of a command, you could look that command up as a key in a lua table, and the value would indicate the control state for that command (returning a number for axis commands and a bool for button/key commands). commands should eventually be bindable through the existing control options screen. support for command aliasing/removing/renaming/addition should also be a part of the system.
-
I don't have any experience with FSO Scripting, but there are some general things to make a Lua script faster:
1. When possible use local variables, a global lookup involves at least one table (and maybe a metatable) lookup where a local variable can be accessed (nearly) immediately.
2. Make local references to often used global functions/tables.
3. When possible don't create tables in inner loops or very often called functions (say every frame or at a small timer interval). The Lua 5.1 GC is good, but many small objects may have a performance impact.
A great resource for performance tips in Lua is this PDF: http://www.lua.org/gems/sample.pdf
Looking through the 'Flashy Deaths' script, wouldn't it be faster to use Lua for the *.cfg files too? Parsing them would be mainly a matter of dofile or loadstring.
And I think making local references to e.g. math.random and some of the often used global tables (arr_*_eff) in the OnFrame section of the code could speed it up a bit.
-
Thank you for the input as well as for the advice and the link...
And yes, making the cfg files into lua would be a lot faster but speed is not of the essence in that section. That is once the date is loaded there is no more need to access the file. Also in early script versions users didn't like to input values into lua files but wanted the data input to be easier and exists outside the .lua files, though this may have changed by now.
And yeah... i havent had time to go through the flashy deaths script in detail. There are still sections in it which could use streamlining (i personally hate the one section of the parsing code, probably should made the currently parsed lined into real global in the first place instead of joggling it back and front the function calls. And i am certain several local flags could be added to the script.
-
parsing in lua really doesnt hurt ingame performance. my tbl parser can be inserted into a script (dofile), run once in an init hook, and then spit out a hierarchy of tables you can use elsewhere in the code. since tables are references this is a very fast way to do things. i used to think dofile didnt work in this implementation of lua, but apparently it requires a path relative to fsroot, and not the mod dir. this isnt such a bad thing though, since it allows you to put modular scripts in one place. , but you kinda have to not change the mod dir from what is used in your dofile paths if you wnat to run files from the local mod dir. perhaps make mod path and root path strings global variables. that way you dont have to use a hardcoded mod folder name.
the meta system im using is fairly fast (and it better be after rewriting it half a dozen times). the first system did a lot of table creation in loops to find new objects and delete removed ones from the metasystem. i had used the ship name as they key but after several attempts to streamline it, i decided i needed a unique identifier that could apply to all objects in the game, not just ships. so i added a new functions in c to get an objects unique numeric signature from within the engine, as well as a function to get an object handle from this signature. now i just use the number as the key. and now the only loop in the whole system is only used to delete dead entries every few seconds.
-
It's probably relative to the engine and not the game data root then. If we start trying to put the engine away from the game data root that could still cause issues.
-
we could add something like ba.getEngineRoot() or ba.getModRoot() to our base library which would return the proper path string. then when you needed to use a dofile, you could just use:
dofile(ba.getModRoot().."data/scripts/something.lua")
instead of
dofile("modname/data/scripts/something.lua")
for simplicity you may want to also include functions to get to your scripts root or your table root, of course these could be setup as strings at init, and that would probibly be faster. i kinda dont want to loose the ability to put scripts in a global directory, which is very useful for development purposes. you can make a modular script and test it against other scripts, which would make debugging it difficult if every script you were working on used a local copy.
-
we could add something like ba.getEngineRoot() or ba.getModRoot() to our base library which would return the proper path string. then when you needed to use a dofile, you could just use:
dofile(ba.getModRoot().."data/scripts/something.lua")
instead of
dofile("modname/data/scripts/something.lua")
I see real benefit in these two additional functions. A mod author can never be quite sure that the user installed the mod in a folder with exactly the same name as used while developing or as recommended in a manual.
It would also give benefits to those total conversion developers who initial develop as a mod, and then need a quick way to scan+replace as they move to a top level TC file structure for release.
-
My point is that I don't think the engine should care where _it_ is. I've been talking with the SCP about the possibility of making it so the exe does not have to exist in the folder with the game content at all. It would simply need to know where the game content is, which could be defined in a config file in the appdata or userdata folder. We've got a lot of work to do in this area but it really needs to be done to make the engine more cross-platform compliant, on all platforms.
-
Agreed.
-
Nuke.. Why wont just use cfile and let it do the file locating for you?
-
just because it shouldnt care where it is, doesnt mean it that it does not need to know where it is. scripting kinda also needs to know where it is, then scripters and modders dont need to make any guesswork as to where lua files are locted, just call a function for the correct path. ideally every lua function should work relative to the mod dir, but there are some built in functions, like dofile, that like to run relative to the program root. so this kinda helps meet that goal.
Nuke.. Why wont just use cfile and let it do the file locating for you?
dofile is equivalent to an include call in c, it has a completely different usage from cfile. dofile helps make script more modular and for large scripts, much more organized.
-
I'm saying, we may try to move the engine outside of the game data folder, at least on some platforms. You won't be able to rely on the program's own folder being the same as the game's main data folder or the parent of the mod folders. It probably won't affect what your talking about as long as this is taken into account.
-
One thing you should know about FS2 Open and Mods:
FS2 Open does not know which mod you are running - and it's impossible to find out.
This probably surprises you, but it's true.
The Launcher passes a list of mod folders to FS2 Open, which may be a single folder or could be several folders. The actual chosen Mod is often the first one in the list - but not necessarily.
FS2 Open also handles files (including tbls and tbms) in such a way that it would be impossible for it to pass a file location to Lua - the file could be in a VP or in the real filesystem.
It would make much more sense for FS2 Open to generically parse a set of tables and put it straight into the Lua datastructure.
- This also means that any changes to how FS2 Open finds the files wouldn't break Lua.
However, that is definitely a lot of work.
-
fortunately everything you need to write a parser defaults to the local mod dir, so thats not the issue. the biggest issue is when your script gets very complicated and you need to cut each hook up into more than one lua file. as i said dofile is like an include in c, but it runs in relation to the executable path. its part of the lua core library so it associates its path with that of the executable. if getting mod dir path data for dofile is impossible, then a substitute must be added.
the atmospheric flight script is about 900 lines of code plus another 400 for my tbl parser, it is also not my largest script. because of this i dont put the lua directly in the scripting.tbl (or xxx-sct.tbm). so my scripting.tbl looks roughly like this:
#Global Hooks
$GameInit:
[[atmospheric_flight_gh_init.lua]]
$Simulation:
[[atmospheric_flight_gh_sim.lua]]
$Hud:
[[atmospheric_flight_gh_hud.lua]]
#end
#Conditional Hooks
$State: GS_STATE_GAME_PLAY
$On Key Pressed: [[keydn.lua]]
$On Key Released: [[keyup.lua]]
$Application: FS2_Open
$On Mission Start:
[[on_mission_start.lua]]
$Application: FS2_Open
$On Mission End:
[[on_mission_end.lua]]
#End
with all the lua files being in moddir/data/scripts/. there are 2 reasons i do this. the main one being debug purposes. when an error is reported at line x, and it is in a lua file, then that line number is exactly where the problem is (because line 1 corresponds to the first line in the file). if the script was in the table file then it would consider line 1 as the first line of script in the hook, which may actually be line 900 in the text file. this makes finding the problem line really complicated. the other reason i keep my lua files seprate is to organize my code. you can also use an ide with the lua files and a text editor with your tbl files without getting your file associations confused.
now in a perfect world dofile() would default to that moddir/data/scripts/, and it might be possible to tell the lua api to use this dir at init. however if that is not the case, it would become very convenient to be able to list lua files in the table, so you could do something like:
#Global Hooks
$GameInit:
[[parser.lua]]
[[metasystem.lua]]
[[dynamics.lua]]
[[gauges.lua]]
$Simulation:
...
using a tbm for this kinda thing doesnt work either. if the script in the tbl depends on code in the tbm, theyre may be errors. tbms should mostly be used in situations where the scripts are unrelated and one doesnt depend on the other to run error free, usually for mini-scripts that add a single simple feature. for example, running the escort retical with a velocity gauge.
to avoid confusion, all the other ideas i posted for streamlining scripting (built-in parsing, meta, and input abstraction) were mainly ideas for streamlining script, and really have nothing to do with the problem of code organization and path issues.
-
Nuke, I think you missed what was meant by "using cfile". cfile in the context of FSO is not the C file api (ie. the ISO standard functions for File IO; fopen, fwrite, etc.) but but the name of a code file "cfile.cpp" that contains all (with very few exceptions, lua being one) methods of accessing the filesystem in the engine. cfile.cpp it what the argument of the -mod flag is passed to, and is what allows the engine its ability to be modded to the extent that it can. It is also because of cfile that FSO does not know or even care what mods are actually passed via the -mod flag.
FSO's cfile api allows code anywhere in the engine to load a file by just passing the files name and cfile will search the folder that corresponds to its extension in the first listed mod folder, then every .vp in the first listed mod folder that has the folder that corresponds to the extension, this is repeated for every mod folder listed, then for the root folder. For example, say -mod mod1,mod2,mod3 is passed to FSO and that you requested atmospheric.lua, cfile will search inorder:
- mod1/scripts/atmospheric.lua
- mod1/vp1.vp:data/scripts/atmospheric.lua
- ...
- mod1/vpn.vp:data/scripts/atmospheric.lua
- mod2/data/scripts/atmospheric.lua
- mod2/vp1.vp:data/scripts/atmospheric.lua
- ...
- vp1.vp:data/scripts/atmospheric.lua
- ...
- vpn.vp:data/scripts/atmospheric.lua
- data/scripts/atmospheric.lua
until it finds a file that exists. This behaviour is also what allows for the mixing of game assets, like blue planet and their layering of the ships to have new designs but still have the stock FS2 and FSU assets. Even :v: used this principle of cfile to allow for normal effects and advanced effects to be activated by adding a .vp (the name of which escapes me) that is earlier in lexical sorting, so that cfile will search and find the high end version of a graphic in the new .vp rather than using the low end one in the Root.vp.
just because it shouldnt care where it is, doesnt mean it that it does not need to know where it is. scripting kinda also needs to know where it is, then scripters and modders dont need to make any guesswork as to where lua files are locted, just call a function for the correct path. ideally every lua function should work relative to the mod dir, but there are some built in functions, like dofile, that like to run relative to the program root. so this kinda helps meet that goal.
No, this is where the problem lies, the script should not care nor should it know where it is located on the filesystem. As I described above, lua is circumventing how the entire engine finds all of its resources. The built-ins that do not use cfile should either be modified to use cfile or removed and replacements implemented which will use cfile. Using cfile does make the scripts more modifiable and flexible.
Using cfile allows anyone to do mods based on your scripting work by just putting your mod in the -mod list and having their work before yours, that way they could change the interface or the ship models, etc.
Nuke.. Why wont just use cfile and let it do the file locating for you?
dofile is equivalent to an include call in c, it has a completely different usage from cfile. dofile helps make script more modular and for large scripts, much more organized.
From your description, Nuke, it seems you are trying to use dofile as a replacement for function calls. Is there something wrong with the function calls in lua? FSO's lua?
In short, I do not mean to call you out Nuke, but why should lua be exempted from the resource location system that the rest of the engine has?
-
The other issue is that if the mod system changes the way it works, you run into a lot of trouble, and won't get any support from it.
-
the problem is there is no sane way to break up your script into more sections for organizational, debugging and modularization purposes. right now theres no way of getting around this problem in lua without using dofile. this will not work in the future. so something at the engine level needs to be done in order to make this possible. replacing dofile with something that automatically resolves the file path is a good idea. being able to explicitly run multiple lua files in a specific order for a single hook in the scripting table, and let the file system determine where the lua files are same way it would find a texture. thats also very good idea.
one of the major problems with scripting is the lack of integration with systems that the engine should manage internally. input is the best example. in the same way lua shouldnt care where its running from, lua also shouldnt care what keys are bound to what, only what the command is and what its state is. lua in its current state only seems to give low level information about whats going on with various input devices, so the script has to recreate some of the engine's functionality for itself, running in parallel to the engine's system and seriously confusing people who play mods as to why they have two sets of keybinds to deal with. so i can see where your getting at.
-
[snip]
hmm.
I didn't realize that the scripting in FSO was so far removed from the engine.
I guess that goes on the list now as well.
-
scripting these days seems like something growing on the codebase that has trouble dealing with basic game logic. it often has to re-create basic engine internals from scratch, i mean simply to use the keyboard for script input requires writing an input system from scratch, then you have to rely on either hardcoded keyboard commands or some parsed file that has to be edited in notepad to change the bindings. it is kinda reminicent of quake 1 and 2 mods (of course when quake 3 rolled around it had a scriptable interface to fix those problems). its not as bad as it was in the early days, when wmc felt the need to change names of functions (or changing get/set functions to virtual variables) in each build. i actually have an archive of scripts that used to work well and now dont work at all. most of it is obsolete crap but a few gems, like freespacelancer and my advanced turret scripts are somewhere it there. makes me think of how much of lua.cpp was written by scripters with little c expirience (im probibly the best example of that). essentially whenever i want to do something in scripting that cant be done, i go into lua.cpp copy an existing similar function, and edit it to make it look up whatever peice of information in the various structs that has whatever info i need, often with read and write access without understanding the consequences of changing that data on the fly. amazingly it usually works and wont break retail so long as it dont use scripting. lua.cpp has turned into an epic mess of ineffitient code and most of the functions are 90% identical.
dont get me wrong, lua is awesome and many popular games make use of it. its unappreciated and not used by many. freespace is so easy to mod that most modders would rather use tables and the events system to accomplish things. and there are so many new features added every month that its hard to keep up with them, let alone find things to do with scripting that cant be tabled in by next week. it kind of creates an environment where modders arent very innovative and wont implement a feature unless it can be done by adding a couple lines to the tables. and if something they want isnt available they could easily find a member of the scp to implement it for them (i remember when you had to sacrifice a small animal or two to get them to add a feature for you). when i was modding quake 1&2 you couldnt just model something and stick it in the game immediately, you always had to write code before you could use it. unless you did maps, which was incredibly difficult (as a directory filled with unfinished partially compiled quake maps on my hd would attest). my experience with quake-c was one of the reasons i supported scripting. i thought that if that system was in freespace it would be awesome. but quake was built around quake-c, all the gameplay was defined by it. unlike with freespace that had all of its gameplay defined and then had scripting thown on after the fact with a few interfaces to various aspects of the game. its one of the things that attracted me to freespace modding was how simple it was to work with. if your models were solid and your tables didnt have errors in them you could roll out models like crazy. the only thing thats hard to do is to mod gameplay, which is where scripting comes in. i just wish it exposed more of the engine and was more versitile and more intuitive to work with.