Author Topic: Fixing SEXPs in Multiplayer  (Read 4765 times)

0 Members and 1 Guest are viewing this topic.

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Fixing SEXPs in Multiplayer
I've been mulling over the problem of SEXPs in multiplayer for the last few days. The problem is that most of the coders don't know how multiplayer works and completely ignore it. As a result we end up with lots and lots of SEXPs that work on the server but not on the clients. Then FUBAR comes along and reports the problem and we end up with even more open bug reports on Mantis.  ;)

 Adding packets for every SEXP seems like a rather short-sighted idea as we'll run out of empty slots for new packets before we run out of SEXPs. It's also rather wasteful of bandwidth to have a new (in most cases reliable) packet sent several times in one event.

I was thinking of making a generic system in SEXP.cpp for multiplayer. That way coders only need to learn how that works and everything else can be done automatically for them. My idea worked something like this. I'll use change-iff as an example even though it has it's own packet at the moment.

Instead of calling send_change_iff_packet() to do a callback the code would call build_generic_sexp_packet(). The data that was sent to send_change_iff_packet() would now be sent to a buffer along with the op_number. Instead of sending the packet immediately the buffer would keep filling up until build_generic_sexp_packet() would overflow. At this point the packet is sent and a new one starts being built. After mission_eval_goals() finishes evaluating the events the game sends any packets left half full. That means that if only a couple of SEXP need to send a packet only one would be sent each frame. Even if a lot of SEXPs do callbacks it should still be more efficient than sending multiple packets.

On the client side the process_generic_sexp_packet() fills a similar buffer and then calls a function similar to eval_sexp(), basically a big case statement calling client side versions of the sexp functions. The coder needs to implement multi_sexp_change-iff() as well. This would be somewhat similar to the sexp_change_iff() function except that instead of working it's way through the SEXP nodes it extracts the data it needs from the buffer.

Of course I'm seeing significant problems in implementing this.

1) Endian issues are what are giving me the biggest pause. If everything was all Intel it wouldn't be a problem but if you write an int into the buffer on a PC it will be the wrong way around when it comes out again on the Mac client. I should be able to beat that but it will require having functions that are a lot lower level than I had wanted them to be.

2) Since SEXPs that can have optional arguments I'm going to have to figure out some way of terminating that can't be mistaken for data. The SP code simply checked if the next node was -1 but that obviously won't work here as that could be a legal value (since the data can be anything). The problem is that in SEXPs that work in a loop there is no way of knowing how many ships etc the SEXP will be applied to.

3) If just one function takes too much data out of the buffer somehow the buffer would become corrupt and any other SEXPs following on from it would be working with incorrect data.


Any comments before I start tearing out my hair trying to make this all work? :p
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Goober5000

  • HLP Loremaster
  • Moderator
  • 214
    • Goober5000 Productions
Re: Fixing SEXPs in Multiplayer
It's an excellent idea. :yes:

It would require a significant amount of design work up-front, though. :) You wouldn't want to get half of it done and then realize you missed an important concept.

EDIT: Also, since it's a major rework, it should go in branch instead of Trunk.  And it would probably be a good idea to change over all SCP sexps to use this method.

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Fixing SEXPs in Multiplayer
Yep, that's part of the reason I posted this rather than just plunging into doing it. :D If I've failed to spot a big gotcha I'd rather hear about it now rather than later. :)

My plan was to eventually add all the new SEXPs that needed multiplayer support to the system. If I was smart enough with how I coded the default case in the multi_eval_sexp() function (or whatever I call it) I could make it so that if the SEXP isn't supported it will just discard the data in the buffer until it reaches the terminator. Which would give a small measure of backwards compatibility. If you added a new SEXP it wouldn't work on older clients but everything else would.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Inquisitor

Re: Fixing SEXPs in Multiplayer
Random side question: Does Apple still make PPC macs?

If they are end of lifing them, in favor of the intel ones, how big of a deal is support for them re: endian-ness?
No signature.

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Fixing SEXPs in Multiplayer
BtRL had quite a few hits from people using PPC Macs and I don't particularly want to leave them in the lurch.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Fixing SEXPs in Multiplayer
If you can avoid assuming endian-ness, you should.  It keeps the code more portable in the future, among other things.  I mean, what if someday we want to run on Solaris-Ultrasparc?  :)

No, I'm not even sure what endian it uses, I'm just saying, stranger things have happened.  But, Apple is no longer making PPC products I believe, and is even pulling PPC support from 10.6 (Snow Leopard).  Still, Leopard users on G5s will be around for quite some time.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline WMCoolmon

  • Purveyor of space crack
  • 213
Re: Fixing SEXPs in Multiplayer
Quote
1) Endian issues are what are giving me the biggest pause. If everything was all Intel it wouldn't be a problem but if you write an int into the buffer on a PC it will be the wrong way around when it comes out again on the Mac client. I should be able to beat that but it will require having functions that are a lot lower level than I had wanted them to be.

There's already functions to deal with this in relation to pilot file code. Look up INTEL_INT.

Quote
2) Since SEXPs that can have optional arguments I'm going to have to figure out some way of terminating that can't be mistaken for data. The SP code simply checked if the next node was -1 but that obviously won't work here as that could be a legal value (since the data can be anything). The problem is that in SEXPs that work in a loop there is no way of knowing how many ships etc the SEXP will be applied to.

Sounds like more of a design issue. You could simply end the list of arguments with a nonvalid argument, regardless. A bigger problem I'd see is if the values are dependent on other SEXPs which may be different for the host system. Not that I can think of anything right off, but if that's not the case, then you could just send the SEXP values along as-is and rebuild the SEXP structure when they arrive at the host machine (using the postended -1 that I suggested).

You could also toss in some kind of a subpacket class that would store the size of the SEXP data and would force any related loops to stop before it got to another SEXP, and also allow accurate indexing with other SEXPs. Combined with a consistent -1 at the end of each packet, you could have pretty good protection against corruption, although you wouldn't have much method to fix it. However that's supposed to be solved by the underlying network layer, if I'm not mistaken.

Basically this is more likely to be caused with issues with the SEXP system itself than anything network-related.

Quote
3) If just one function takes too much data out of the buffer somehow the buffer would become corrupt and any other SEXPs following on from it would be working with incorrect data.

Again, this is more of a packet design issue that can easily be solved than an issue with SEXPs.
-C

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Fixing SEXPs in Multiplayer
There's already functions to deal with this in relation to pilot file code. Look up INTEL_INT.


Already have. :) For most of the network code in FS2_open you can simply use the GET_INT etc macros which call them. That's what I meant about having to do things a lot lower level than I wanted to. :)

Quote
Sounds like more of a design issue. You could simply end the list of arguments with a nonvalid argument, regardless.


That's the thing. I'm not certain I can. Any event that requires a list of strings to be sent to client machine would screw over the system. The best solution I came up with was similar to the one you suggested. After sending the ID of the SEXP the packet should state how many bytes make up the data that SEXP is sending. The client would only read out that number of bytes.

Quote
A bigger problem I'd see is if the values are dependent on other SEXPs which may be different for the host system. Not that I can think of anything right off, but if that's not the case, then you could just send the SEXP values along as-is and rebuild the SEXP structure when they arrive at the host machine (using the postended -1 that I suggested).


Well it's considered poor practice to break existing packets at the moment. I have a few changes that probably should be in 3.6.10 but I won't commit them cause they'll break compatibility with the packets current 3.6.10 builds send. The easiest solution is just to tell coders to treat new SEXPs the same way (i.e once you add multiplayer code to a SEXP, changing the data it sends counts as a backwards compatibility issue).

The alternative is to add version numbers to the packets saying which version of the code sent it out. If the version number is different you simply discard the packet (or portion thereof). Yes that means that a change breaks compatibility with older clients but at least they shouldn't crash as a result. Since there is no easy way to have the server know which versions the clients are running and therefore tailoring the packets sent to them, that's about as good as it can get unless we start getting very complicated with it.

Most games simply force clients to upgrade to the latest version if they want to play online precisely to avoid issues like this. :)

Quote
You could also toss in some kind of a subpacket class that would store the size of the SEXP data and would force any related loops to stop before it got to another SEXP, and also allow accurate indexing with other SEXPs. Combined with a consistent -1 at the end of each packet, you could have pretty good protection against corruption, although you wouldn't have much method to fix it. However that's supposed to be solved by the underlying network layer, if I'm not mistaken.

I like the idea of having -1 at the end of every packet I simply meant that it can't be the only method of ascertaining when we've reached the end of the data we want to send.  Cause -1 is a legal value for many SEXPs. However if we say that there are x bytes in the data and then check that x+1 = -1 that should give us a reasonable degree of protection against cock ups

Quote
Basically this is more likely to be caused with issues with the SEXP system itself than anything network-related.

I'm not worried about corruption in sending the data. I'm worried about corruption from the client and server expecting different amounts of data due to the coder in question screwing up. :D As you say, corruption in transit should be covered by the network/transport layer. But one mistake in one SEXP shouldn't bring the whole packet system grinding to a halt. It should simply print out a warning and recover gracefully. :D
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Fixing SEXPs in Multiplayer
And since 3.6.10 isn't finalized, if this would qualify as a bug fix, it should be done I think.  No harm making people upgrade their beta versions right now.  Anything that can make multiplayer more stable before 3.6.10 should still be done I think.  So if committing fixes things, I'd do it.  They can always grab the latest builds for testing.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Fixing SEXPs in Multiplayer
I'll commit them. But I'll commit them all at the same time. It's stupid to break compatibility now, a month from now and 3 months from now. It's far more sensible to keep them until I can say "This build is the point at which new 3.6.10 builds became incompatible with older ones"
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Fixing SEXPs in Multiplayer
Ok yeah, that makes sense :)  No need to keep breaking them over and over and over I guess.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Fixing SEXPs in Multiplayer
:bump:

I've had a big long think about this all day long about the way I plan to implement this. The goal is to end up with a system that any SCP coder can use to add multiplayer functionality to any new SEXP simply by following a short list of rules. First I'll go into what I'd want everyone else to have to write when they make a new SEXP and then I'll explain how I'm thinking about making the code do that.

What I want to do is create a series of wrapper functions that can be called from the functions that actually do the work of making the SEXP work. They'll have a generic easily understood name like multi_send_ship, multi_send_object, multi_send_int, etc. Earlier I gave the example of the change-iff SEXP coded using this system so I'll stick to that. The code that actually changes the iff and does a multiplayer callback currently looks something like this

Code: [Select]
void sexp_ingame_ship_change_iff(ship *shipp, int new_team)
{
Assert(shipp != NULL);

shipp->team = new_team;

// send a network packet if we need to
if((MULTIPLAYER_MASTER) && (shipp->objnum >= 0))
send_change_iff_packet(Objects[shipp->objnum].net_signature, new_team);
}

Now if someone were coding the change-iff SEXP today it would be up to them to code in the send_change_iff_packet() packet into multimsgs.cpp themselves. Since only a few coders understand how that all works very few have bothered in the past. So we get new SEXPs that don't work in multiplayer.
Once I have the new stuff working though you'd be able to just code this.

Code: [Select]
void sexp_ingame_ship_change_iff(ship *shipp, int new_team)
{
Assert(shipp != NULL);

shipp->team = new_team;

// send a network packet if we need to
if((MULTIPLAYER_MASTER) && (shipp->objnum >= 0))
multi_start_packet();
multi_send_ship(shipp);
multi_send_int(new_team);
multi_end_packet();
}

You'd also have to code a new function for the player side.

Code: [Select]
void multi_sexp_ingame_ship_change_iff()
{
ship * shipp;
int team;

multi_get_ship(shipp);
multi_get_int(team);

Assert(shipp != NULL);
Assert(team = whatever check to see that it's a valid team);

shipp->team = new_team;
}

Hopefully I can make it work that easily. :) Now for how I think it would be implemented.

 
First, add a global variable, set in eval_sexp() to hold the number of the SEXP operator. This prevents the coder needing to keep track of it.
Second, create a buffer to hold the data. It will need to be bigger than MAX_PACKET_SIZE so I'd suggest simply doubling that.
Third create a second buffer that holds the data type (standard ones like byte, short, etc but also a few other ones like vector, orient and string which the game already has functions for). The 2nd buffer will also hold include 3 special types OP, COUNT and TERMINATOR. OP means that the next value in the data buffer is the op_number for a SEXP (i.e it denotes that we're starting the data for a new SEXP). COUNT says how many pieces of data there will be in this next set and TERMINATOR means the end of a set of data (Basically a check that the data sent in COUNT hasn't overflowed).

With that done I can start on the real hard work. :)

Code: [Select]
void multi_start_packet() {
Write the SEXP_Operator number into the data buffer.
Write OP and then COUNT into the Type buffer.
Store the next data index as we'll need it later to write the COUNT.
}

void multi_send_int() {
Write the int into the data buffer (remembering to deal with endian issues).
Write INT into the Type buffer.
Increment the COUNT by 4 (i.e the size of an int).
Call the multi_sexp_maybe_send_packet() function.
}

void multi_end_packet() {
Write -1 into the data buffer.
Write TERMINATOR into the Type buffer.
Write the COUNT into the data buffer at the index we saved earlier.
Call the multi_sexp_maybe_send_packet() function.
}

void multi_sexp_maybe_send_packet() {
        If the index of the data buffer isn't MAX_PACKET_SIZE yet, return.
If the last entry in the type buffer is a TERMINATOR then store the current index of the data array.
If not, iterate back through the types array until we find the last one and store the corresponding data index instead.
Write the data buffer until the stored index into a SEXP_UPDATE packet and send it to to the clients.
Slide down any entries after the stored index to the start of the array.
}

That should pretty much handle the server side of things. I'd need a check in game_simulation_frame() to make sure the buffer is flushed after mission_eval_goals() returns so that you don't have any data still sitting in the buffer until the next frame of course.

On the client side when a SEXP_UPDATE arrives it would read out the data into a buffer from the start of the packet until it has read out the number specified in the COUNT. It then checks that the next one is a TERMINATOR (i.e -1). If it's not then it will have to throw out the entire packet as there is no way to recover from that error.
 If the packet is fine then it checks that the OP is valid. If it's not it throws out this SEXP and moves on to the rest of the packet (Thus giving us compatibility if the client has an older version of FSO than the server). If the OP is valid the game then calls a function similar to eval_sexp() but which figures out which function to call on the client ( multi_sexp_ingame_ship_change_iff() in the example above).

Anyone see any glaring flaws in the plan so far?
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline FUBAR-BDHR

  • Self-Propelled Trouble Magnet
  • 212
  • Master Drunk
    • 165th Beer Drinking Hell Raisers
Re: Fixing SEXPs in Multiplayer
Only thing I can think of is if there are any special needs for standalones or the client hosting on a standalone. 

Although I'm sure I'll find new and more creative ways of screwing it up once implemented. 
No-one ever listens to Zathras. Quite mad, they say. It is good that Zathras does not mind. He's even grown to like it. Oh yes. -Zathras

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Fixing SEXPs in Multiplayer
Only thing I can think of is if there are any special needs for standalones or the client hosting on a standalone. 

None that immediately spring to mind.

It's probably something I'll have to warn about when writing down the short list of rules (cause you can never assume that there will be a player on the machine executing the code) but that's something we have to worry about already anyway.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Fixing SEXPs in Multiplayer
:bump:

Okay, I've implemented most of the basic functions needed for this to work. I still have to implement all the wrapper functions like multi_send_int() etc but I've reached the point where I'd rather test this with a SEXP or two before implementing them all and finding I've made a mistake that means I have to redo them all.

So which SEXPs don't work in multi? If someone posts one that will be simple for me to test with I'll implement that one first but I want to work my way through all the ones that don't work sooner or later.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Aardwolf

  • 211
  • Posts: 16,384
Re: Fixing SEXPs in Multiplayer
Oh hey, glad to see somebody's working on this.

  

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Fixing SEXPs in Multiplayer
Like most of the stuff that is SEXP related, I'm doing it cause I want it myself. :) I've had some really nice missions ruined by the fact that you can't make a lot of SEXPs work on the client side.

EDIT : Progress

I think that's the majority of the hard bit over now. With luck it's just a case of making the SEXPs use the packet I've built.
« Last Edit: April 11, 2009, 06:20:17 pm by karajorma »
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]