Author Topic: Food for thought: A world without sexps  (Read 5776 times)

0 Members and 1 Guest are viewing this topic.

Offline Fury

  • The Curmudgeon
  • 213
Food for thought: A world without sexps
Sorry, no tl,dr version for you.

Quote
Sep 13 19:22:33 <T-Rog|Mobile>   Would it be possible to make a primary that requires a lock?
Sep 13 19:23:32 <@The_E>   A primary what?
Sep 13 19:23:45 <@The_E>   Ah
Sep 13 19:23:48 <@The_E>   No
Sep 13 19:23:53 <@The_E>   No it wouldn't be possible
Sep 13 19:29:24 <T-Rog|Mobile>   I was thinking of making a fighter-mountable AAAf and I decided it would be too unbalanced if it didn't require a lock-on
Sep 13 19:31:51 <Fury`>   script it
Sep 13 19:32:21 <Fury`>   with lua I mean
Sep 13 19:32:45 <@The_E>   That would be possible, I suppose
Sep 13 19:33:01 <@The_E>   But why not just turn down the damage?
Sep 13 19:33:20 <MatthTheGeek>   it would be less cool
Sep 13 19:33:23 <MatthTheGeek>   comon
Sep 13 19:34:04 <MjnMixael>   you could also hack something with sexps.. but probably not the best option :)
Sep 13 19:34:34 <MatthTheGeek>   you could probably draw something that looks like the lock HUD indicator with scripts
Sep 13 19:34:39 <MatthTheGeek>   can't do that with sexps
Sep 13 19:34:45 <Fury`>   indeed
Sep 13 19:34:51 <MjnMixael>   lies
Sep 13 19:34:55 <MjnMixael>   custom HUD gauges
Sep 13 19:35:09 <MjnMixael>   would be less fancy
Sep 13 19:35:13 <MatthTheGeek>   how would you make it stick on the target ship
Sep 13 19:35:15 <MjnMixael>   but you could do something
Sep 13 19:35:22 <MatthTheGeek>   with sexps
Sep 13 19:35:44 <MjnMixael>   couldn't.. but you can get creative and show a lock some other way
Sep 13 19:35:51 <MjnMixael>   and then change target-iff-color or something
Sep 13 19:35:52 <MatthTheGeek>   I guess
Sep 13 19:35:59 <MatthTheGeek>   but it would be less cool ! c'mon
Sep 13 19:36:06 <Fury`>   hmh
Sep 13 19:36:19 <MjnMixael>   less cool, perhaps.. but I encourage creativity with sexps
Sep 13 19:36:24 <MjnMixael>   you can do a ****ton of things
Sep 13 19:36:59 <Fury`>   you can do so much more with scripting, in fact I can't help but think that one day scripting replaces sexps entirely
Sep 13 19:37:02 <MatthTheGeek>   yeah, but creativity with scripts is good too
Sep 13 19:37:21 <MatthTheGeek>   doubt scripting will replace sexps
Sep 13 19:37:24 <MjnMixael>   indeed
Sep 13 19:37:48 <MjnMixael>   and scripting can do a lot as well, most definitely.. i heart me some scripts
Sep 13 19:37:54 <MjnMixael>   it's just got a higher barrier to entry than sexps
Sep 13 19:37:57 <Fury`>   it lacks the necessary hooks into the game at the moment I believe, to replace sexps
Sep 13 19:38:09 <Fury`>   or not, I'm not sure
Sep 13 19:38:21 <MatthTheGeek>   Fury`: scripting can call sexps
Sep 13 19:38:23 <MatthTheGeek>   soooooo
Sep 13 19:38:50 <Fury`>   yes they can but can it do everything sexps can without calling them?
Sep 13 19:39:08 <MatthTheGeek>   they can't, that would be silly
Sep 13 19:39:14 <Fury`>   I mean, if one day scripting hooks go so deep mission designers could forego sexps entirely
Sep 13 19:39:15 <MatthTheGeek>   code redundancy is bad
Sep 13 19:40:08 <@The_E>   I'd love it if LUA could be used to script the entire game from start to finish
Sep 13 19:40:17 <@The_E>   That would be quite the achievement
Sep 13 19:40:55 <Fury`>   I'm not sure if that tone was sarcastic or serious
Sep 13 19:41:51 <@The_E>   I would truly, actually love it if that were possible
Sep 13 19:43:21 <Fury`>   How about tackling it the other way. Fully integrate lua into fred2_open and fs2_open to a point where lua can control all elements. Then rewrite s-expression framework into functions that perform their actions in lua instead. Then remove the old sexp framework.
Sep 13 19:43:44 <Fury`>   Your average fredder wouldn't know the difference
Sep 13 19:43:53 <Fury`>   But the power is there for anyone who knows how to use it
Sep 13 19:44:05 <Fury`>   But eh, maybe that's a bit ambitious
Sep 13 19:44:14 <Fury`>   But something that could be considered once wxfred is complete
Sep 13 19:44:18 <MatthTheGeek>   what's the point of recoding sexps into lua if it doesn't make any difference
Sep 13 19:45:27 <Fury`>   MatthTheGeek, I meant that it doesn't make any difference to your average fredder because on the surface lua functions are masked as sexps. But everything is done in lua in the background. In addition, since lua is now tightly integrated, people can write their own sexps on the fly for ****s and giggles
Sep 13 19:45:34 <Fury`>   Or do anything else they want
Sep 13 19:45:39 <Fury`>   If they know how, that is
Sep 13 19:45:47 <@The_E>   Fury`: That would indeed be awesome
Sep 13 19:45:48 <MatthTheGeek>   and the point would be ?
Sep 13 19:46:06 <MatthTheGeek>   I'm not getting it, if you can call sexps in lua, why do you want to recode sexps in lua
Sep 13 19:46:17 <Fury`>   *sigh* The_E?
Sep 13 19:46:22 <@The_E>   MatthTheGeek: It's like this
Sep 13 19:47:16 <@The_E>   Sexps are not as flexible as lua commands. So it would make sense to run the sexps on the lua system, and thus simplify how the game state can be manipulated
Sep 13 19:47:49 <MatthTheGeek>   (18:47:15) The_E: Sexps are not as flexible as lua commands. <- can you elaborate on that ? it seems like exactly the same to me
Sep 13 19:48:00 <@The_E>   Right now, the sexp system is a mini-codethulhu that has hooks everywhere, and the lua system is a mini-codethulhu that has hooks everywhere
Sep 13 19:48:20 <MatthTheGeek>   codethulu ? I thought making sexps was easy
Sep 13 19:48:26 <MatthTheGeek>   there's aven a tuto somewhere
Sep 13 19:48:55 <@The_E>   If we were to revamp the sexp system to just be a specialization of the lua interpreter, it would allow the interop between the two to be much better
Sep 13 19:49:41 <MatthTheGeek>   whouldn't that do nothing else but add overhead to the whole thing ?
Sep 13 19:49:55 <MatthTheGeek>   plus potential bugs due to recoding the whole thing to being with
Sep 13 19:49:58 <@The_E>   And yes, writing sexps is easy, but you're still manipulating the engine state with them; You have hooks into all the major subsystems of the engine (except file access)
Sep 13 19:50:23 <@The_E>   Well, that's where stuff like luaJIT comes in
Sep 13 19:51:30 <Fury`>   Imagine fred and fs without s-expression framework. It has been replace by lua scripting framework. For backwards compatibility and for fredders convenience all old sexps are now exist as easily usable pre-made functions. New "sexps" can be added when and if needed.
Sep 13 19:51:35 <Fury`>   But since lua now powers the event framework, everybody who knows at least a bit of lua can now do their own sexps easily within fred interface for any mission they want. The code is then included within the mission file. This opens the possibility for outstanding flexibility in mission designing that sexps can't hope to touch.
Sep 13 19:52:06 <@The_E>   Yeah, it would basically mean that we wouldn't have to write sexps in the engine anymore
Sep 13 19:52:21 <@The_E>   You could just supply a lua library of stuff
Sep 13 19:53:11 <Fury`>   And LuaJIT provides the performance to handle even heaviest scripted missions
Sep 13 19:53:28 <MatthTheGeek>   can you even include lua in .fs2 files ?
Sep 13 19:53:37 <@The_E>   Not yet
Sep 13 19:53:47 <MatthTheGeek>   how would you even run retail FS2 missions
Sep 13 19:53:51 <Fury`>   It's just text MatthTheGeek, just like anything else like fiction viewer
Sep 13 19:54:09 <MatthTheGeek>   Fury`: last time I checked, fiction viewer is separate file from missions
Sep 13 19:54:28 <Fury`>   MatthTheGeek, like I said, for backwards compatibility the lua framework interprets old sexps with the pre-made functions bundled into the executables
Sep 13 19:55:08 <@The_E>   Yeah, the idea would be to parse the sexps and convert them to lua on the fly
Sep 13 19:55:16 <Fury`>   MatthTheGeek, okay poor example. But lua scripts are still pure text that can be included in the mission file
Sep 13 19:55:47 <MatthTheGeek>   so instead of coding sexps in the engine, people can code sexps in lua, but old sepxs still need to be doded in the engine
Sep 13 19:56:00 <MatthTheGeek>   how is that different from making a lua script and call it with sexps to begin with
Sep 13 19:56:12 <MatthTheGeek>   -d+c
Sep 13 19:56:31 <MatthTheGeek>   aside from integrating the lua in the mission file, which is a separate feature
Sep 13 19:57:06 <Fury`>   because once you are done with the lua functions that interprets old sexps, you don't need to touch them anymore except to fix any potential bugs
Sep 13 19:57:28 <MatthTheGeek>   but why would you need them in the first place
Sep 13 19:57:28 <Fury`>   everything else from that point onwards would be in lua, no need to consider backwards compatibility
Sep 13 19:57:33 <@The_E>   MatthTheGeek: Sexp interpreter is one subsystem. Lua interpreter another. They're both completely separate, although they should, in principle, be very closely related
Sep 13 19:57:36 <MatthTheGeek>   if they're gonna stay stucked in the engine anyway
Sep 13 19:58:29 <Fury`>   I think I'm missing something that makes MatthTheGeek confused, but I don't know what
Sep 13 19:58:34 <Fury`>   it's all very clear to me at least
Sep 13 19:58:35 <@The_E>   Fury` is right in saying that, for a cleaner engine design, both subsystems should be combined into a common platform, and the only logical way to do this is to run the sexp interpreter as a subset of the lua interpreter
Sep 13 19:58:39 <Fury`>   and The_E seems to agree
Sep 13 19:59:20 <MatthTheGeek>   what confuses me is where's the point of recoding existing sexps into lua if it's gonna do the same thing anyway
Sep 13 19:59:24 <MatthTheGeek>   and add overhead on top of that
Sep 13 19:59:54 <@The_E>   The overhead would happen in a place where we don't care about it (during mission parsing)
Sep 13 20:00:07 <MatthTheGeek>   ok
Sep 13 20:00:23 <Fury`>   and how much overhead that would be?
Sep 13 20:00:29 <Fury`>   you need to parse old sexps too
Sep 13 20:01:00 <MatthTheGeek>   yes but the old sexps don't have to run their code through a whole lua interpreter that will then go through engine hooks
Sep 13 20:01:01 <Fury`>   the only overhead comes from interpreting old sexps and turning them over to lua
Sep 13 20:01:18 <MatthTheGeek>   but like E said, overhead is irrelevant in that case
Sep 13 20:01:24 <MatthTheGeek>   since it's during mission load
Sep 13 20:01:29 <@The_E>   Fury`: I can't really say. It's basically taking the event code, and compiling it into equivalent lua, which is a task of indeterminate (but manageable) complexity
Sep 13 20:01:47 <@The_E>   I don't know, call it half a second or so of CPU time?
Sep 13 20:01:59 <Fury`>   sounds about right
Sep 13 20:02:20 <Fury`>   the amount of sexps in any single mission file in terms of bits is comparably very small
Sep 13 20:02:42 <MatthTheGeek>   what do you mean in term of bits
Sep 13 20:02:43 <@The_E>   Also, it's something you'd have to do only once, there's nothing stopping you from caching the result
Sep 13 20:02:59 <Fury`>   The_E, ooh, very nice idea
Sep 13 20:03:26 <@The_E>   Basically, we're talking about introduucing a new, non backwards compatible mission file format
Sep 13 20:03:42 <@The_E>   Which we would convert old missions to on the fly
Sep 13 20:04:06 <@The_E>   brb, food
Sep 13 20:04:18 <MatthTheGeek>   wait, so you'd also have to write the lua of the old sexps in the mission files on the fly ?
Sep 13 20:04:23 <MatthTheGeek>   isn't that redundant ?
Sep 13 20:04:33 <MatthTheGeek>   shouldn't the lua code of old sexps stay in the engine ?
Sep 13 20:04:53 <Fury`>   I believe The_E was referring to caching
Sep 13 20:05:31 <MatthTheGeek>   he's talking about a new mission file format
Sep 13 20:05:38 <MatthTheGeek>   I don't understand where that comes from
Sep 13 20:06:30 <@The_E>   Well
Sep 13 20:06:40 <Fury`>   Probably because old fred and fs don't know what to do with all the new information fs2 files would now have
Sep 13 20:07:07 <MatthTheGeek>   how is that different from new sexps
Sep 13 20:07:18 <MatthTheGeek>   old fred and fs break on those just as well
Sep 13 20:07:27 <@The_E>   It's like this, if we were to switch mission scripting to be lua-based, it would make the most sense if the mission files were acceptable input to the lua parser as a whole (i.e. be proper lus files)
Sep 13 20:07:35 <@The_E>   *lua
Sep 13 20:07:48 <MatthTheGeek>   wait so we're not just talking about sexps there
Sep 13 20:07:57 <MatthTheGeek>   that's yet another feature here
Sep 13 20:08:15 <Fury`>   The_E, so you mean the .fs2 files would become themselves lua scripts in entirety?
Sep 13 20:08:47 <AndrewofDoom>   You need to write this idea down The_E
Sep 13 20:08:48 <@The_E>   Well, you're already ripping out a whole chunk of the mission parsing code, might as well go all the way
Sep 13 20:08:51 <AndrewofDoom>   I like where this is going.
Sep 13 20:09:16 <Fury`>   And there is one benefit in that. Interpretation of old sexps could be performed when fred saves the mission file, instead of when fs loads the mission.
Sep 13 20:09:54 <@The_E>   That's not gonna work if we want backwards compatibility to old missions (which we do)
Sep 13 20:10:15 <@The_E>   SCP Rule 1: FS2 retail must always be valid input for the engine
Sep 13 20:10:23 <Fury`>   Oh right, sorry, you're right
Sep 13 20:10:37 <Fury`>   But hey, what's stopping from having interpretation at both times
Sep 13 20:10:42 <MatthTheGeek>   making sure every single mission parses correctly is gonna be a *****
Sep 13 20:10:47 <@The_E>   Nothing whatsoever
Sep 13 20:11:21 <@The_E>   MatthTheGeek: See, that's the beauty of it, we can build a parser and prove it to be correct
Sep 13 20:11:21 <Fury`>   You could force fred to perform interpretation and save the results into the mission file, which fs can then parse without interpretation required
Sep 13 20:11:44 <AndrewofDoom>   You're forgetting about multiplayer though
Sep 13 20:11:58 <AndrewofDoom>   It means all missions will be invalidated when converted?
Sep 13 20:12:05 <@The_E>   We have one programming language with a strictly defined syntax and API that we're porting to a different language with strictly de3fined syntax and API
Sep 13 20:12:14 <MatthTheGeek>   since when do we have multiplayer
Sep 13 20:12:26 <AndrewofDoom>   Since QuantumDelta says we do?
Sep 13 20:12:36 <MatthTheGeek>   yeah right
Sep 13 20:12:41 <@The_E>   AndrewofDoom: That's a good point, we would have to change the messaging system there as well, but that's not that big a deal
Sep 13 20:13:04 <@The_E>   Instead of sending messages from sexps, we'd be sending messages from lua
Sep 13 20:13:21 <@The_E>   But now I really have to go for food
Sep 13 20:13:23 <@The_E>   brb
Sep 13 20:13:27 <AndrewofDoom>   Lua as I've seen makes a very huge assumption that the player is the only player.
Sep 13 20:13:55 ---»   MageKing17 (~Paladin@2601:8:af80:374:30df:a37b:c46:fcfc) has Joined #freespace
Sep 13 20:15:29 <Fury`>   AndrewofDoom, http://en.wikipedia.org/wiki/Category:Lua-scripted_video_games
Sep 13 20:15:40 <Fury`>   there's quite a few multiplayer capable games in that list
Sep 13 20:15:53 <Fury`>   which means lua itself won't be a problem as far as multiplayer is concerned
Sep 13 20:16:06 <AndrewofDoom>   Yes I know.
Sep 13 20:16:18 <MatthTheGeek>   AndrewofDoom's talking about FS' lua
Sep 13 20:16:20 <AndrewofDoom>   FSO's interpretation of lua is however making this assumption.
Sep 13 20:16:23 <MatthTheGeek>   FSO's*
Sep 13 20:16:38 <Fury`>   huh?
Sep 13 20:16:53 <MatthTheGeek>   scripts break hilariously bad in multi
Sep 13 20:17:13 <MatthTheGeek>   there's probably 90% of lua hooks that only work for the host
Sep 13 20:17:19 <AndrewofDoom>   hv.Player is always the host.
Sep 13 20:17:45 <MatthTheGeek>   I don't think we currently have the hooks to proparly lua for multi
Sep 13 20:17:55 <MatthTheGeek>   not that there's any interest for that anyway
Sep 13 20:18:24 <Fury`>   Well, okay. That is a problem with fred/fs scripting and not with lua itself
Sep 13 20:18:41 <AndrewofDoom>   Once multiplayer gets overhauled I might be. But, we'll be waiting until 2335 for that.
Sep 13 20:18:43 <MatthTheGeek>   wait what has fred to do with this
Sep 13 20:18:43 <Fury`>   Considering this would be large overhaul of scripting functions, the problem should be addressed while at it
Sep 13 20:19:03 <MatthTheGeek>   AndrewofDoom: FS mod for SC will be the multiplayer overhaul
Sep 13 20:19:39 <Fury`>   MatthTheGeek, everything. If this conversation ever materialises into something practical, both fred and fs would share same scripting framework
Sep 13 20:20:00 <Fury`>   Just like they share s-expression framework
Sep 13 20:20:04 <MatthTheGeek>   I thought this conversation was now about current SF scripting
Sep 13 20:20:12 <Fury`>   bah
Sep 13 20:20:18 <MatthTheGeek>   not potential future, where you could just as well put ponies and rainbows
Sep 13 20:20:35 <Fury`>   yeah well I was never talking about current situation
Sep 13 20:20:35 <MatthTheGeek>   hell Axem already put nyan cat
Sep 13 20:20:55 <MatthTheGeek>   so you were in your own monologue while AndrewofDoom and I were discussing multi ?
Sep 13 20:20:56 <MatthTheGeek>   that's cool
Sep 13 20:21:18 <Fury`>   there is no multi if you ask me, so yeah
Sep 13 20:21:38 <AndrewofDoom>   It's the elephant in the room.
Sep 13 20:21:40 <MatthTheGeek>   I pretty much agree with that statement
Sep 13 20:21:48 <MatthTheGeek>   Fury`'s statement
Sep 13 20:23:17 <Fury`>   okay, I'm gonne post this conversation to the forums

 
Re: Food for thought: A world without sexps
Interesting thought indeed. It would be a huge amount of work, but it could certainly yield interesting results. :)
How would new SEXPs get integrated into the engine? I mean, I'm writing a brand new SEXP in lua, I want to use it in every mission, and I'm sure others out there would like to use it as well. Does the new SEXP get hard coded into the .exe? Does it get included in some sort of "SEXP library" bundled with FSO?

Also, I might be a bit off topic, but given how deep the lua hooks would go, could that allow us to write new AIs more easily as well? (Thinking about the RTS framework) (and the mess that is the AI code)

 

Offline AndrewofDoom

  • In A.D. 2366 war was beginning
  • 29
  • Permanent yuri goggles.
    • Skype
    • Steam
    • Twitter
Re: Food for thought: A world without sexps
How would new SEXPs get integrated into the engine? I mean, I'm writing a brand new SEXP in lua, I want to use it in every mission, and I'm sure others out there would like to use it as well. Does the new SEXP get hard coded into the .exe? Does it get included in some sort of "SEXP library" bundled with FSO?

You could probably use the scripting tables for common scripting functions.

Also, I might be a bit off topic, but given how deep the lua hooks would go, could that allow us to write new AIs more easily as well? (Thinking about the RTS framework) (and the mess that is the AI code)

No. Until the scripting gets full engine access you can't change the AI. Okay...maybe you could do. Maybe. You would have to rewrite the AI from scratch entirely to make a new AI. Don't quote me on it.

One thing that is very briefly mentioned is that FSO's implementation of the lua scripting would have to be 100% multiplayer compatible first. Scripting is so terribly broken right now as pretty much everything in the system assumes that there is only one player in the game.
My Efforts:
SF Knight

20:08:19   AndrewofDoom: Though I find it mildly disturbing that a loli is giggling to mass destruction.
20:10:01   Spoon: I find it mildly arrousing
20:10:07   AndrewofDoom: Woah
20:10:15   Spoon: sound like my kind of loli
20:10:21   Spoon: and im not even a lolicon

 

Offline mjn.mixael

  • Cutscene Master
  • 212
  • Chopped liver
    • Steam
    • Twitter
Re: Food for thought: A world without sexps
How would new SEXPs get integrated into the engine? I mean, I'm writing a brand new SEXP in lua, I want to use it in every mission, and I'm sure others out there would like to use it as well. Does the new SEXP get hard coded into the .exe? Does it get included in some sort of "SEXP library" bundled with FSO?

You could probably use the scripting tables for common scripting functions.

That sounds like a nightmare compared to the convenience of SEXPs.

I mentioned it briefly in IRC and I'll say it again here. This has all kinds of wonderful and fantastic uses. It would be a great addition... for people who know LUA. The uptake on FREDing is already low... to expect a bunch of people to learn a code language to create the kind of missions that the "greats" are would be making is crazy. I've already seen people say they won't learn SEXPs because they don't think they can pull off what is done in BP, JAD or even ST:R. And SEXPs are eeeaaaaaaaasy.

Granted, I know I'm posting this on a coding specific board.. so all you coders are like, 'yeah, but so is LUA!' You are probably right. But probably not as easy as SEXPs to approach. I guess I'm saying that to dump SEXPs entirely in favor of LUA from some point on... is a quick an easy way to cut down on the number of people making missions... Missions, by the way, are what we want more of.

(This is going off the impression from the conversation that if this ever happened, people probably wouldn't make new SEXPs anymore.)
Cutscene Upgrade Project - Mainhall Remakes - Between the Ashes
Youtube Channel - P3D Model Box
Between the Ashes is looking for committed testers, PM me for details.
Freespace Upgrade Project See what's happening.

 

Offline The E

  • He's Ebeneezer Goode
  • Moderator
  • 213
  • Nothing personal, just tech support.
    • Steam
    • Twitter
Re: Food for thought: A world without sexps
For the moment, this is nothing but a bit of "What if" thinking. There are no concrete plans to make this happen at this time (since it means gutting one major subsystem and doing major additions to a second, and even bigger alterations to a third).

I do hear you on the usability part, throwing away sexps or the way events are written is silly given that it would basically mean every content creator would suddenly find one of his major skillsets invalidated. I can think of a few ways to address this (after all, if you're working in the event editor, you don't actually care about what the mission file looks like, all you care about is that the logic you set up there works as expected), but yeah. This is waaaaaay out there, and not going to happen anytime soon.
If I'm just aching this can't go on
I came from chasing dreams to feel alone
There must be changes, miss to feel strong
I really need lifе to touch me
--Evergrey, Where August Mourns

 

Offline Dragon

  • Citation needed
  • 212
  • The sky is the limit.
Re: Food for thought: A world without sexps
Granted, I know I'm posting this on a coding specific board.. so all you coders are like, 'yeah, but so is LUA!' You are probably right. But probably not as easy as SEXPs to approach. I guess I'm saying that to dump SEXPs entirely in favor of LUA from some point on... is a quick an easy way to cut down on the number of people making missions... Missions, by the way, are what we want more of.
Indeed. Part of why I'm sticking to FS instead of modding ArmA, Orbiter or some other game is how easy to mod FS is. Other games usually require you to learn a full scripting language, which I just don't feel like doing. I'd probably have to learn something like C at some point (for unrelated reasons), but SEXPs, with their drop-down menus are the simplest scripting system I know of. No thinking about syntax, brackets or even looking through wiki to find a function you want. Just a bunch of drop-down menus with functions neatly grouped into trees and fully explained in the editor. You don't even need to type more than a few characters. That's why SEXPs are so great, despite being essentially Common LISP with a bunch of FS-specific functions.

 

Offline Axem

  • 211
Re: Food for thought: A world without sexps
I've been using LUA and SEXPs together quite a bit as of late. There's somethings that only sexps can do, and somethings that only lua can do (lua's most lacking thing is control of ship flags, hud gauges (which I know m|m has a patch for), and messages, just to name a few things). Its sort of funny that you can fill in those gaps with the runSEXP and evalSEXP lua functions. :p I think LUA needs a lot of work done on it still before it could be considered to be equal to SEXPs much less replace SEXPs entirely.

I would say if the front end stayed mostly the same (that is the easy to use flyout menus and strictly enforced error checking) I would not see a downside to having missions written in LUA. LISP has a pretty weird syntax compared to most saner programming languages and there are some drawbacks to the current SEXP system (such as SEXPS can only ever return numbers and not anything else, as opposed to LUA which can return anything and everything). So in theory, LUA would probably be superior. In theory. (Also again, I stress this with the same sort of front-end we have now, so instead of a "when condition action", you'd see a "if condition then action" instead! Any new interface should be very similar to the one we use now, so die-hard FREDders should not lose sleep!)

Its a nice long term dream, emphasis on long term...

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Food for thought: A world without sexps
Ironically we had this conversation in the other direction when WMCoolmon was adding scripting. The idea was to add a LISP-style scripting language which was based on the one the SEXPs were using.

As for the idea, if FRED can still parse the mission (i.e if this isn't an on-the-lfy SEXP) to make edits, I don't have much of an objection. Right now, we have one of the easiest mission editors to use on the planet. Let's not spoil that.

As for lua not being multi-aware, it might not be that hard to code up a system similar to the multi_SEXP system I added a whole back that works for lua instead. I don't know much about scripting so I've not looked at the code much but think it would be possible. Quite frankly, I've been tempted to move all multiplayer messages to that sort of system anyway!
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Food for thought: A world without sexps
I mentioned it briefly in IRC and I'll say it again here. This has all kinds of wonderful and fantastic uses. It would be a great addition... for people who know LUA. The uptake on FREDing is already low... to expect a bunch of people to learn a code language to create the kind of missions that the "greats" are would be making is crazy. I've already seen people say they won't learn SEXPs because they don't think they can pull off what is done in BP, JAD or even ST:R. And SEXPs are eeeaaaaaaaasy.

Granted, I know I'm posting this on a coding specific board.. so all you coders are like, 'yeah, but so is LUA!' You are probably right. But probably not as easy as SEXPs to approach. I guess I'm saying that to dump SEXPs entirely in favor of LUA from some point on... is a quick an easy way to cut down on the number of people making missions... Missions, by the way, are what we want more of.

(This is going off the impression from the conversation that if this ever happened, people probably wouldn't make new SEXPs anymore.)
Okay so, it is all theoretical at this point but us tackle these concerns.

All pre-lua sexps should still available in the events editor as before, which sexp to lua parser interprets and handles over to lua subsystem to keep backwards compatibility with old missions.

There could be two ways to add new sexps. In one you can add new sexps within fred, which opens a text editor that has built-in lua syntax to help you. That is then saved into the mission file.

Second is a table (tbl or tbm), let us use sexps.tbl for convenience. Both fred and fs builds would bundle a default sexps.tbl, which can be either overridden or added to with a tbm. The sexps.tbl would most likely contain all pre-lua sexps necessary for backwards compatibility with existing missions, so that they won't break. And any new post-lua sexps would be added to a tbm instead, locking the sexps.tbl for anything but bug fixes. By having old sexps parsed through sexps.tbl into lua has the advantage that even non-coders can try and fix bugs in the parser, should there be any. Or people like Axem can wreak havoc on people by changing how even old sexps behave in JAD, without having to resort to custom builds.

As for new sexps, we could use tbm's for those like mysexp-sxp.tbm which would be easily distributable. We could also build a library of those into FSWiki and possibly even build library access right into fred itself. Just look at popular internet browsers with built-in library access for their addons. So with that, we have two possibilities to add new sexps. Either tables or directly written into the mission file. The first option can be shared and distributed easily, the second gives fredder easy platform to handle sexps unique to one mission. And a lua library would provide easy access to works of the community.

What this means is that writing new sexps is a lot easier than it has been so far. You would no longer need a coder to do it and you wouldn't need to compile new builds for it. Sharing and distribution of new sexps does not need to be difficult either, we can have a library just for that which fred could access within the application.

That said. The work involved in overhauling lua scripting to a point where it can replace sexps is daunting. But this community has tackled big projects before, so it can be done.


Edit: It just occurred to me that if lua is given access to pretty much anywhere, then it is probably necessary to move tbl/tbm parsing over to tbl to lua interpreter and handled entirely in lua. This does have the added benefit that you don't necessarily need to roll out new builds to add support for new tabling features. Not only that but giving control over tables to lua also gives astounding possibilities. For example, the question T-Rog asked in the irc chat log that began this whole discussion could be done in lua version of weapons.tbl or tbm just to give one example.
« Last Edit: September 14, 2013, 01:32:32 am by Fury »

 
Re: Food for thought: A world without sexps
Quote
You would no longer need a coder to do it and you wouldn't need to compile new builds for it.
Agree with you on the no-need-to-compile part, disagree with no longer needing a coder, as you would still need some lua coding skills, which right now is one of the barriers to getting into FSO scripting.

Then again, if the SCP decided to go this way, I'm sure the FRED tutorials would get updated as well to include some lua basics.
That being said, having the full array of control structures and data-handling mechanisms provided by an actual programming language into FRED would be frakking awesome. I am still grateful to whoever implemented the if-then-else SEXP back in 3.6.14.

Also, could you clarify the last part of your post, regarding lua parsing in other tables, such as weapons.tbl/-wep.tbm?

 

Offline m!m

  • 211
Re: Food for thought: A world without sexps
I'm just going to write down what I think could be done on the technical side of preserving the current SEXP-tree functionality as we all agree that it is easy to use and should not be changed drastically.
Instead of somehow keeping the tree around FRED should create the tree structure out of the actual "mission code"TM by parsing the lua code it into an AST (I have to check if there exists a good solution for this) to create the tree structure we are familiar with.
The opposite way would be similar, FRED creates an AST from the tree structure and serializes it as lua source code into the mission file, that would also make it easy to convert old missions as it would only require to parse the old LISP-syntax into the FRED tree and then plug that into the lua serialization. That way the old parsing code could be kept.
Another problem that may arise is that the common SEXP name is something like do-something-cool which is not a valid lua function name. For that we would probably need to do some name mapping in FRED to keep the familiar names.

A quick google search showed that there are existing solutions for parsing lua: Lua Fish and Lua Inspect. Lua Fish is a bit old, the github repository hasn't been updated for three years and Lua Inspect was last changed a year ago.

 

Offline Axem

  • 211
Re: Food for thought: A world without sexps
If (in a million years) that we do switch to a lua-based mission format, I don't think we should abandon what FRED gives us the most, a mostly readable graphical interpretation of events.

But that's not to say it couldn't be improved. Inspired by mostly Game Maker (and other easy to use "codeblock" style editors), here's a horrible mockup of my idea of a future Event Editor. Its sort of a mix between our event editor and the campaign editor. Instead of being stuck on a mostly 1 dimensional tree, we branch out to a 2 dimensional plane, which I think improves readability.



I think "if 'Alpha 1' Hitsleft < 25 then" is a ton more readable and understandable to the layman than "when < hitsleft 'Alpha 1' 25"

And of course, for power users, we could have a swap to text editor button.


 

Offline z64555

  • 210
  • Self-proclaimed controls expert
    • Steam
Re: Food for thought: A world without sexps
*takes notes*
Secure the Source, Contain the Code, Protect the Project
chief1983

------------
funtapaz: Hunchon University biologists prove mankind is evolving to new, higher form of life, known as Homopithecus Juche.
z64555: s/J/Do
BotenAlfred: <funtapaz> Hunchon University biologists prove mankind is evolving to new, higher form of life, known as Homopithecus Douche.

 

Offline Nuke

  • Ka-Boom!
  • 212
  • Mutants Worship Me
Re: Food for thought: A world without sexps
it may be totally possible to compile the pseudo-lisp that the sexps use into lua byte code. it might be possible to do this without modifying fred or the mission file syntax, and this compilation can be done at load time. this would actually be preferable for fredders because you dont have to do anything different at all. then you have to convert all the sexps into lua functions (it makes little sense to compile lisp into lua byte code, only to use the runSEXP function). if you want to do something like what axem suggested a couple posts up, you can compile the flowchart based language into lua byte code. the bad news is you have to write a compiler, which ive learned recently is not easy.

heres a vm reference

http://underpop.free.fr/l/lua/docs/a-no-frills-introduction-to-lua-5.1-vm-instructions.pdf
I can no longer sit back and allow communist infiltration, communist indoctrination, communist subversion, and the international communist conspiracy to sap and impurify all of our precious bodily fluids.

Nuke's Scripting SVN

  

Offline Nuke

  • Ka-Boom!
  • 212
  • Mutants Worship Me
Re: Food for thought: A world without sexps
One thing that is very briefly mentioned is that FSO's implementation of the lua scripting would have to be 100% multiplayer compatible first. Scripting is so terribly broken right now as pretty much everything in the system assumes that there is only one player in the game.

the only reason its not multiplayer compatible is because nobody ever implemented that functionality, or exposed any network stuff what so ever. i suggest 2 things to rectify this issue.

first add some conditions to the conditional hooks system, like $NetworkNodeType: which can be "None" or "Singleplayer" (default for compatibility), "MultiplayerHost" or "MultiplayerServer" (not sure if host and server are synonymous or not in fs), and "MultiplayerClient". so if a chunk of lua is meant to run on the host/dedicated server, you use "$NetworkNodeType: MultiplayerHost" as one of the conditions. client scripts run for every player in the game and handles stuff that needs to stay local, host script runs only on the host/server, and handles script that effects the whole game, such as stats tracking. you can also exclude single player script from multiplayer modes. you might also need conditions for multiplayer game type, or for observers.

second we need a means of client-server, server-client, server-multicast, communications so the scripts can send data to each other through the server. this could just be dedicated raw udp/tcp pipes, with the engine taking care of the addressing and maintaining connections (you can probibly hack this in now, in lua, using the luasocket module). or it can just piggyback on the regular network frames that the game already transmits.  to send you use the relevant send function, who's arguments are the data to be sent (probably vararg), with the engine handling the serialization in c. each call represents one message. to receive you also need a hook that gets called each time a message is received, you can then use said hook to handle the message in the script.
« Last Edit: September 15, 2013, 07:31:41 am by Nuke »
I can no longer sit back and allow communist infiltration, communist indoctrination, communist subversion, and the international communist conspiracy to sap and impurify all of our precious bodily fluids.

Nuke's Scripting SVN

 
Re: Food for thought: A world without sexps
it may be totally possible to compile the pseudo-lisp that the sexps use into lua byte code. it might be possible to do this without modifying fred or the mission file syntax, and this compilation can be done at load time. this would actually be preferable for fredders because you dont have to do anything different at all. then you have to convert all the sexps into lua functions (it makes little sense to compile lisp into lua byte code, only to use the runSEXP function). if you want to do something like what axem suggested a couple posts up, you can compile the flowchart based language into lua byte code. the bad news is you have to write a compiler, which ive learned recently is not easy.

heres a vm reference

http://underpop.free.fr/l/lua/docs/a-no-frills-introduction-to-lua-5.1-vm-instructions.pdf

seems to me that by implementing the sexps in lua you already have a sexp → lua bytecode compiler; you just compile the lua-ised sexps
The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell.

 

Offline z64555

  • 210
  • Self-proclaimed controls expert
    • Steam
Re: Food for thought: A world without sexps
Regarding Axem's earlier post, I did a quick search and it looks like it is totally possible to have a diagramming library based on/compatible with the wxWidgets workflow. Some examples are wxArt2D, wxShapeFramework, and wxWorkspaceView.

wxShapeFramework looks promising.
Secure the Source, Contain the Code, Protect the Project
chief1983

------------
funtapaz: Hunchon University biologists prove mankind is evolving to new, higher form of life, known as Homopithecus Juche.
z64555: s/J/Do
BotenAlfred: <funtapaz> Hunchon University biologists prove mankind is evolving to new, higher form of life, known as Homopithecus Douche.

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Food for thought: A world without sexps
first add some conditions to the conditional hooks system, like $NetworkNodeType: which can be "None" or "Singleplayer" (default for compatibility), "MultiplayerHost" or "MultiplayerServer" (not sure if host and server are synonymous or not in fs), and "MultiplayerClient".


They are not synonymous. The Server runs the game. The Host is the first player on the server. On a standalone they probably aren't even on the same PC.

Quote
second we need a means of client-server, server-client, server-multicast, communications so the scripts can send data to each other through the server. this could just be dedicated raw udp/tcp pipes, with the engine taking care of the addressing and maintaining connections (you can probibly hack this in now, in lua, using the luasocket module). or it can just piggyback on the regular network frames that the game already transmits.  to send you use the relevant send function, who's arguments are the data to be sent (probably vararg), with the engine handling the serialization in c. each call represents one message. to receive you also need a hook that gets called each time a message is received, you can then use said hook to handle the message in the script.

I suggested above simply implementing a system similar to the one I did for SEXPs. The SEXP system works by simply creating some wrapper classes you can call to send data to the other PCs. All you need to do is something like this

Code: [Select]

                        multi_start_callback();
multi_send_object(oswpt.objp);
multi_send_int(speed);
multi_send_int(axis);
multi_send_int(subjective);
multi_end_callback();


on the server. The code then adds the necessary data (which SEXP is being called on the client, etc). The client simply needs some similar code to get the data and decide what to do with it. In lua you'd simply need to have the code identify which script is listening on the other side and then code in a similar framework.


I don't think this would be a particularly huge task to do (IIRC the SEXP version took a couple of days work) but I'm not particularly familiar with lua so I'd probably need the help of a coder who was.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Nuke

  • Ka-Boom!
  • 212
  • Mutants Worship Me
Re: Food for thought: A world without sexps
They are not synonymous. The Server runs the game. The Host is the first player on the server. On a standalone they probably aren't even on the same PC.

you would probibly want conditions for both then. the host would probibly handle gameplay stuff, while the server would only need to be bothered with things like stats logging. this has possibilities such as achievements and persistent progress tracking, so you could have things like trading games that remember your currency and cargo manifest even after you leave the game. some epic possibilities there. the host would probibly need to run both the client and the host scripts to handle both local stuff for the player, and then stuff for everyone, communicating through loopback or something.

the point is you need to be able to explicitly select which code to run for which node in the network. things like hv.player would need to be sensitive to this context, or deprecate it for a more robust system like getCurrentPlayer(), getCurrentHost(), getOtherPlayer(). i seldom use hv.player, because it never seemed to work right in the first place, and i got used to alternate methods that were more reliable.

Quote
I suggested above simply implementing a system similar to the one I did for SEXPs. The SEXP system works by simply creating some wrapper classes you can call to send data to the other PCs. All you need to do is something like this

Code: [Select]

                        multi_start_callback();
multi_send_object(oswpt.objp);
multi_send_int(speed);
multi_send_int(axis);
multi_send_int(subjective);
multi_end_callback();


on the server. The code then adds the necessary data (which SEXP is being called on the client, etc). The client simply needs some similar code to get the data and decide what to do with it. In lua you'd simply need to have the code identify which script is listening on the other side and then code in a similar framework.

I don't think this would be a particularly huge task to do (IIRC the SEXP version took a couple of days work) but I'm not particularly familiar with lua so I'd probably need the help of a coder who was.

i guess that works too. start-end pairs are probibly more intuitive. you might need different send commands for lua's datatypes, or better yet handle the type conversion in c. but its pretty much basic network stuff.
I can no longer sit back and allow communist infiltration, communist indoctrination, communist subversion, and the international communist conspiracy to sap and impurify all of our precious bodily fluids.

Nuke's Scripting SVN

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Food for thought: A world without sexps
you would probibly want conditions for both then. the host would probibly handle gameplay stuff, while the server would only need to be bothered with things like stats logging. this has possibilities such as achievements and persistent progress tracking, so you could have things like trading games that remember your currency and cargo manifest even after you leave the game. some epic possibilities there. the host would probibly need to run both the client and the host scripts to handle both local stuff for the player, and then stuff for everyone, communicating through loopback or something.


Actually it's the server where you'd do that. The host is merely in charge of loadout, which mission gets played and things like who can choose to end the game. The majority of the work is done by the server. Like I said before, I don't think getting lua to support multiplayer would be that hard. It's just that it probably requires a lot more knowledge of lua than I have.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]