Author Topic: In-Game Network Code Discussion  (Read 2893 times)

0 Members and 1 Guest are viewing this topic.

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
In-Game Network Code Discussion
So, we've tossed around ideas like RakNet replacement and such for a while, but no one has yet stepped forward to say that they think they can single-handedly spearhead the replacement.  Maybe that's the wrong approach though.  What I'd like to do is maybe get some sort of 'analysis by comittee' going, where anyone interested checks in on this thread or whatever and we try to get a group understanding and documentation of what's going on.  If we were ultimately going to use RakNet, the idea would probably be to start with replacing the in game network code module, and since that seems to be the part of the code that, until fixed, renders any other efforts towards multiplayer kind of moot, I think it's the best place to start.

But let's assume for a minute we don't need to use RakNet.  I've been theorizing lately about this.  We did not one, but two go_fasters for the graphics code, which both found some low hanging fruit and serious drains on the framerate, and fixed them in a series of individual enhancements, which led to massive gains.  Many of these were fixes for features that weren't optimized to begin with, or were optimizations on the original framework.  I think something similar might be possible for the netcode.  FS2 multiplayer was popular back in the day, so (someone correct me if I'm wrong) it had to have been playable to some extent.  But what we see today seems to me like something no one would have put up with.  If that's the case, then perhaps the changes over time to the game have actually degraded the behavior, and we need to figure out where and how, and try to speed it back up.  Even if that's not completely the case, I think we can still start with analyzing the current netcode performance, be it with enhanced logging, tracing, debugging, whatever, and attempt to identify some of the areas where the netcode might be having hiccups.

For instance, I ping my dedicated server, and see times around 35ms.  In game, FS2 reports my latency as 250-300ms.  What on earth?  Is this just because of a different transport layer, or is the code really injecting that much overhead just to ping?  If that's the case, is there similar overhead in game?  There's really no excuse for that if that's the case.

Also, from what I've seen, packets have had to grow to handle things like additional SEXPs and other expanded information over time.  Could these larger packets be slowing down client-server communication significantly at this point, resulting in significant degradation compared to retail?  If that's the case, perhaps we need to analyze how we send that kind of data and determine if there are ways to shrink the packets.

Are packets compressed?  If not, we have zlib already for png, could we use zlib to compress larger packets?  I think most modern computers can compress/decompress fast enough to make this a worthwhile consideration to get faster data transmission.

We could get a network logging/debugging branch going on Github to collaborate our efforts, if anyone has any ideas for a good place to start putting some logging facilities in.

So what say you all?  Anyone else here interested in doing a grass roots team effort towards network code refactoring?
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Re: In-Game Network Code Discussion
I don't understand just about any portion of the multiplayer code, but I'm always willing to help debug it.
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: In-Game Network Code Discussion
And that's what I'm getting at, I don't think anyone here really does, but it really does warrant the team's attention, so I was thinking that maybe instead of one person, the team could try to make a group effort to start untangling it.  Even simple things like identifying entry points and code flow and getting other coders all on the same page might help lead to better understanding of the whole by multiple team members.  And if anyone has any friends who might have an interest in net code, maybe send them our way.  I'm almost to the point of considering getting a bounty together for getting multiplayer in a state that FotG could be enjoyable, among other things.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline Cyborg17

  • 29
  • Life? Don't talk to me about life....
Re: In-Game Network Code Discussion
Well, I got a small glimpse at the netcode while fixing (the impressively still unimplemented) Campaign description update behavior.  In case this helps: most of the netcode revolves around numerous functions which each either send a packet, or interpret a received packet, or both with a check on whether or not you're the host (or sometimes the standalone server).  There's a dedicated function for writing the headers for packets before they're sent out and I believe one for sending them out.  Though I don't currently remember the function, I believe there's one which decides what kind of packet it is by the header.

I actually didn't see any functions that don't revolve around packets.  I may be able to look at it again later, but I'm in the middle of a move.
« Last Edit: September 01, 2015, 06:00:42 pm by Cyborg17 »

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: In-Game Network Code Discussion
My biggest problem with dealing with the netcode is I don't know how to debug network code. So I spend inordinate amounts of time starting up a multiplayer match on my LAN and hoping I can figure out whether the issue is client or server side so I can make sure that machine is the one I'm running MSVC on.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline z64555

  • 210
  • Self-proclaimed controls expert
    • Minecraft
    • Steam
Re: In-Game Network Code Discussion
Debug builds have some network related DCF's that can be accessed through the debug console (Shift+Enter anywhere), one notable DCF that I remember is one that simulated latency.

I'm guessing to be able to debug the network code by oneself, you'd setup a standalone server season and then boot up a client season on the same PC. I'm not sure how effective this is with the current state of FSO, however.
Secure the Source, Contain the Code, Protect the Project
chief1983

------------
funtapaz: Hunchon University biologists prove mankind is evolving to new, higher form of life, known as Homopithecus Juche.
z64555: s/J/Do
BotenAlfred: <funtapaz> Hunchon University biologists prove mankind is evolving to new, higher form of life, known as Homopithecus Douche.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: In-Game Network Code Discussion
I've found the "attach a debugger" method to have issues with clients timing out. Other methods like the debug console or liberal sprinklings of mprintf's tend to work better, but have their own limitations for certain classes of problem.  In general though I feel that on a LAN there aren't many lag issues, and trying to find someone 'across the internet' to test/debug with is extremely hard especially when you take different timezones, schedules and RL into account.
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: In-Game Network Code Discussion
Glad there's some interest here.  Standalone servers running on a cheap VPS can be helpful in that situation.  Since we can run them on *nix now, they can be on any *nix server, and have GDB attached if necessary, but the lag should at least be closer to real life for the local client.  You don't even need another player for many multiplayer missions, but the communications would still go through the standalone I believe.  I have a retail standalone server running on occasion in fact, under GDB with AddressSanitizer running to catch any issues that crop up.  Anyone can feel free to use it for testing.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline niffiwan

  • 211
  • Eluder Class
Re: In-Game Network Code Discussion
Is your server with ASAN+gdb this one?  "SCP WebUI Test Server"

Also, based on what z64555 said I had a super quick look at these look like the relevant debug console lag controls:

Code: [Select]
$ git grep DCF | egrep -w 'lag|netd'
code/network/multi.cpp:DCF(netd, "change netgame debug flags (Mulitplayer)")
code/network/multilag.cpp:DCF(lag, "Sets the lag base value (Muliplayer)")
code/network/multilag.cpp:DCF(lag_min, "Sets the lag min value (Multiplayer)")
code/network/multilag.cpp:DCF(lag_max, "Sets the lag max value (Multiplayer)")
code/network/multilag.cpp:DCF(lagloss, "Help provider for the lag system (Multiplayer)")
code/network/multilag.cpp:DCF(lag_streak, "Sets the duration of lag streaks (Multiplayer)")
code/network/multilag.cpp:DCF(lag_bad, "Lag system shortcut - Sets for 'bad' lag simulation (Multiplayer)")
code/network/multilag.cpp:DCF(lag_avg, "Lag system shortcut - Sets for 'average' lag simulation (Multiplayer)")
code/network/multilag.cpp:DCF(lag_good, "Lag system shortcut - Sets for 'good' lag simulation (Multiplayer)")
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: In-Game Network Code Discussion
Yup that's the one.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: In-Game Network Code Discussion
We may want to fix those function descriptions too, dunno what 'Mulitplayer' and 'Muliplayer' are, although Mulitplayer does remind me of the Vandals' Ape Drape song.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline jr2

  • The Mail Man
  • 212
  • It's prounounced jayartoo 0x6A7232
    • Steam
Re: In-Game Network Code Discussion
Would it be possible to make regular-ish builds that collect more logging data, and ask people to use those for multi matches?

You could even have them auto-send the logs to your email maybe (assuming you made that clear to the testers, and maybe put a status upon exiting that it's sending the logs).

Just an idea, not sure if that would help.  But if so, put it in the announcements, etc.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: In-Game Network Code Discussion
Possibly but I don't think we're to that point yet, I think just getting a handle on the tools we have, and wrapping our heads around what we do know would be more useful currently.  I already learned some things here.

Another thing to keep in mind is that multi has its own log, in data/multi.log.  One trick for getting a handle on what code runs when might be to watch that log during the game and identify what messages get written, and then we could find those messages in the code.  Might help with getting the per-frame flow understood.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline JGZinv

  • 211
  • The Last Dual! Guardian
    • The FringeSpace Conversion Mod
Re: In-Game Network Code Discussion
Not that I'm a coder, but would it be possible to host a shared folder, or even something akin to a dropbox, so someone could pull the multi.log in near realtime while several people play?
Would that be of use?
True power comes not from strength, but from the soul and imagination.
Max to PCS2 to FS2 SCP Guide
The FringeSpace Conversion Mod

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: In-Game Network Code Discussion
Not sure how much help that would be, compared to simpler ideas.  But I am willing to allow SSH access to the standalone server on the SCP account I set up, which runs the standalone and can access the multi.log on it.  So anyone using it would have access to their client and the standalone at least.  Other players' logs could be uploaded after the fact.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline niffiwan

  • 211
  • Eluder Class
Re: In-Game Network Code Discussion
I had a thought - it should be fairly simple to test the theory that SCP additions have slowed down the netcode, someone could run up retail multi and see how the lag differs from the current FSO version?
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline jg18

  • A very happy zod
  • 210
  • can do more than spellcheck
Re: In-Game Network Code Discussion
Good idea, niffiwan.

Honestly, I don't think we'll know whether to try to optimize the existing netcode or just replace it with RakNet until we know  how optimizable the existing netcode is, which requires a deep understanding of it. The prerequisites for understanding it include a basic, if not moderately deep, understanding of both socket programming in general and the two networking libraries that FSO uses, Winsock and Unix sockets. That's quite a bit to learn.

If we decide the existing netcode is worth optimizing then a good first step for readability/maintainability is to port the messy blend of Winsock/Unix sockets to a cross-platform socket library like Boost Asio.

After I finish the SquadWar support I can dig into the netcode some more and hopefully get some sort of documentation going.