Author Topic: Welcome to the Rendering Engine Overhaul Project  (Read 118209 times)

0 Members and 1 Guest are viewing this topic.

Offline Bobboau

  • Just a MODern kinda guy
    Just MODerately cool
    And MODest too
  • 213
Re: Welcome to the Rendering Engine Overhaul Project
and I thought I was on the edge with my 8 cores.
Bobboau, bringing you products that work... in theory
learn to use PCS
creator of the ProXimus Procedural Texture and Effect Generator
My latest build of PCS2, get it while it's hot!
PCS 2.0.3


DEUTERONOMY 22:11
Thou shalt not wear a garment of diverse sorts, [as] of woollen and linen together

 
Re: Welcome to the Rendering Engine Overhaul Project
The problem is still that the engine was never designed for that kind of abuse (and the lack of documentation certainly doesn't help).
FSO relies heavily on global structures, and synchronising access to those is likely to be an extremely difficult task.
Just think about what would happen if you tried to call the dereference operator on a vector at the same time as another thread was doing an insert or push_back operation (yes, particles, looking at you). You'll crash straight away and not know why, and worse still, it'll be an intermittent crash that won't show up easily.
Not to mention that MT is just difficult to start with. SCP does not have anyone with sufficient experience to do MT reliably in FSO.
STRONGTEA. Why can't the x86 be sane?

 

Offline DaBrain

  • Screensniper
  • Moderator
  • 212
    • Shadows of Lylat board
Re: Welcome to the Rendering Engine Overhaul Project
and I thought I was on the edge with my 8 cores.

Well... if you have the money, you can buy a system with 48 cores...
http://cgi.ebay.com/NEW-TYAN-AMD-OPTERON-6174-G34-48-CORE-MAGNY-COURS_W0QQitemZ270529337772QQcmdZViewItemQQptZCOMP_EN_Servers?hash=item3efcce19ac

I'd really like to see the render times on that monster. Though that's the only thing I can think of that might use all cores.



The problem is still that the engine was never designed for that kind of abuse (and the lack of documentation certainly doesn't help).
FSO relies heavily on global structures, and synchronising access to those is likely to be an extremely difficult task.
Just think about what would happen if you tried to call the dereference operator on a vector at the same time as another thread was doing an insert or push_back operation (yes, particles, looking at you). You'll crash straight away and not know why, and worse still, it'll be an intermittent crash that won't show up easily.
Not to mention that MT is just difficult to start with. SCP does not have anyone with sufficient experience to do MT reliably in FSO.

I know it won't happen, just because it needs to happen. It's not like I know what the solution is either...

All I'm saying is, that when I have slow-downs in FSO, most of the time is caused by the CPU limitation, which is a shame, because FSO only uses one of my four cores.

The real question is: Does FSO have a future without MT?
If it doesn't, what can we do? Is nobody here interested in learning MT coding?


From here on I doubt we'll see many non-MT games anymore, except for small freeware titles.
In fact I think more and more games will not only use MT (more and more cores), but also make use of GPGPUs (via OpenCL, CUDA, Stream, ect.).
--------------------------------------------------
SoL is looking for a sound effect artist
Please PM me in case you want to apply
---------------------------------
Shadows of Lylat - A Freespace 2 total conversion
(hosted by Game-Warden)
----------------------------------

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Welcome to the Rendering Engine Overhaul Project
From my layman point of view, there are only three possible coding approaches to improve FSO performance significantly.
1) Multi-threading
2) GPGPU
3) Find code blocks that uses most CPU time and rewrite them for better single-thread performance

None of the three solutions are exactly easy, but FSO is not getting any lighter as new features are added, on the contrary. It's a fact that something needs to be done about the CPU bottleneck as soon as possible. If this does not get any developer focus, FSO will eventually hit a large wall where adding wanted features simply can't be done because most people don't own a CPU capable of running FSO anymore.

 

Offline Nuke

  • Ka-Boom!
  • 212
  • Mutants Worship Me
Re: Welcome to the Rendering Engine Overhaul Project
if we havent already, soon the engine will reach a point where no new cpu-intensive features will possible to add, and we will just have to dig in and optimize what we have, then just focus on gameplay features for mods, and that will be it.

other option would be to rewrite core components of the engine with multithreading in mind (although not quite possible at the time), one at a time. when enough of those core components are modernized then you look again at the possiblility of multithreading, at which point you convert over all the data structures to multicore friendly equivalents. sounds like a lot of work to me. but in the longrun it would be worth it. otherwise wel have an engine that will kinda fizzle out and die as people start modding newer games.
I can no longer sit back and allow communist infiltration, communist indoctrination, communist subversion, and the international communist conspiracy to sap and impurify all of our precious bodily fluids.

Nuke's Scripting SVN

 

Offline Zacam

  • Magnificent Bastard
  • Administrator
  • 211
  • I go Sledge-O-Matic on Spammers
    • Minecraft
    • Steam
    • Twitter
    • ModDB Feature
Re: Welcome to the Rendering Engine Overhaul Project
For example: While we are using SSE/SSE2 compiler optimizations, there are no references in the code for those defines for areas that might benefit (3D Drawing and OpenGL, I'm looking at you) where actual SSE/SSE2 written code (based on #ifdef likely) could be used for further performance gains.
Report MediaVP issues, now on the MediaVP Mantis! Read all about it Here!
Talk with the community on Discord
"If you can keep a level head in all this confusion, you just don't understand the situation"

¤[D+¬>

[08/01 16:53:11] <sigtau> EveningTea: I have decided that I am a 32-bit registerkin.  Pronouns are eax, ebx, ecx, edx.
[08/01 16:53:31] <EveningTea> dhauidahh
[08/01 16:53:32] <EveningTea> sak
[08/01 16:53:40] * EveningTea froths at the mouth
[08/01 16:53:40] <sigtau> i broke him, boys

 

Offline Iss Mneur

  • 210
  • TODO:
Re: Welcome to the Rendering Engine Overhaul Project
The problem is still that the engine was never designed for that kind of abuse (and the lack of documentation certainly doesn't help).
FSO relies heavily on global structures, and synchronising access to those is likely to be an extremely difficult task.
Just think about what would happen if you tried to call the dereference operator on a vector at the same time as another thread was doing an insert or push_back operation (yes, particles, looking at you). You'll crash straight away and not know why, and worse still, it'll be an intermittent crash that won't show up easily.
Not to mention that MT is just difficult to start with. SCP does not have anyone with sufficient experience to do MT reliably in FSO.
I was under the impression that documenting the engine so that we could do exactly this was the point of the Overhaul project?

I agree that multi-threading is currently impossible to do in FSO (and I have only had a detailed look into cfile, talk about race conditions), but I do not think we are unable to do it.  Personally, I think the best way to start multi-threading FSO is to work toward removing the usage of global variables in the engine and finish FSO transformation into a strictly object orientated engine, if only to formalize the engine internals and complete the documentation.

On the topic of GPGPU, one advantage of reaching for this target, especially using OpenCL, is that it would allow the use of multiple CPU cores if the user does not have or want to use the graphics card to do the offloaded features.  One of the many reasons for not doing the GPGPU work on the GPU could be from only having one GPU (and not enough spare time and/or bandwidth to actually speed up the game).  As I think was stated before, the most obvious (in my opion, of course) would be the physics engine which I assume includes the collisions for weapons and ships.
"I love deadlines. I like the whooshing sound they make as they fly by." -Douglas Adams
wxLauncher 0.9.4 public beta (now with no config file editing for FRED) | wxLauncher 2.0 Request for Comments

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Welcome to the Rendering Engine Overhaul Project
I don't believe it's something beyond the coders here either. But it is a huge task and we can't expect the mods and TCs to wait for it.

My suggestion would be to do the same thing I suggested for the graphics overhaul. If you did want to add multi-threading to the engine, you'd split a branch off from trunk, get multi-threading working and then patch any features added to trunk in the intervening months (or years!) back into the new, improved code. That branch would now become the default FSO code and the old, single-threaded branch could be forgotten about.

The question is whether there is anyone willing to do all that? If there is I wish them luck. :yes:
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Welcome to the Rendering Engine Overhaul Project
My suggestion would be to do the same thing I suggested for the graphics overhaul.
Wouldn't that be in practice impossible to do? I mean, you have two or more branches in which people are overhauling major parts of the engine. Each branch would probably end up overhauling same code as well, for different purposes. Then you'd have to merge changes from not only main branch, but those off-branches too.

Sounds extremely hard to pull off.

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Welcome to the Rendering Engine Overhaul Project
You'd only do one at a time. Only an idiot would attempt both. :D
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Welcome to the Rendering Engine Overhaul Project
Indeed? I see several problems with that.

- Overhaul projects will probably end up taking years to complete.
- During these years, people and priorities change.
- We have no multi-threading experts on SCP right now, but the need for multi-threading is pretty much immediate due to performance issues we're having right now.
- With Hery gone to other projects we don't really have graphics experts right now either, but graphics overhaul seems to have larger momentum even though there is no quite as big need for it. Unless you can drastically reduce CPU usage with just overhauled graphics engine?

Anyway, I have my doubts sequential overhaul projects are going to really work because of the first two points.
« Last Edit: March 08, 2010, 05:24:47 am by Fury »

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Welcome to the Rendering Engine Overhaul Project
Well you could try doing both using the same branch. What I mean is that only an idiot would try having a graphics branch and a multi-threading branch, as you said, too many things need to change in both branches.

Of course that means that the amount of planning you need to do goes way up.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Spicious

  • Master Chief John-158
  • 210
Re: Welcome to the Rendering Engine Overhaul Project
It would be an interesting test of version control merging though. When is this going to turn into the "refactoring project"?

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Welcome to the Rendering Engine Overhaul Project
I already mentioned it earlier in this topic, but the fastest way of getting some results in breaking this CPU bottleneck would be concentrated effort in code optimization rather than new features. Might be worth giving it a short for 3.7.0? Only finish the features that couldn't be completed in time for 3.6.12, but focus only to bug fixes and code optimization? If this concentrated code optimization period is met with success, I believe there would be a lot of happy people.

I mean, higher and smoother framerates, what's not to like? :)

Edit: Hell, Darius gave his permission to give BP SVN access to any SCP coder who needs access to a mod that is stable, most of the time error-free and wastes plenty of system resources if that's what is needed to optimize code. :) *cough* carrot *cough* :D
« Last Edit: March 08, 2010, 06:30:04 am by Fury »

 

Offline Spicious

  • Master Chief John-158
  • 210
Re: Welcome to the Rendering Engine Overhaul Project
Wouldn't that reinforce the obstacles to making major changes?

 
Re: Welcome to the Rendering Engine Overhaul Project
Edit: Hell, Darius gave his permission to give BP SVN access to any SCP coder who needs access to a mod that is stable, most of the time error-free and wastes plenty of system resources if that's what is needed to optimize code. :) *cough* carrot *cough* :D

Fury`: Carrot is all well and good, but there are other issues :) I will add that BPSVN is one of the best ways to performance test the engine.
karajorma: Your idea cannot work. What happens when an interface changes that mean that the 'features' can't be easily ported to new structures without a _LOT_ of work?  (and that's the whole point of the branch you suggested)
STRONGTEA. Why can't the x86 be sane?

 

Offline The E

  • He's Ebeneezer Goode
  • 213
  • Nothing personal, just tech support.
    • Steam
    • Twitter
Re: Welcome to the Rendering Engine Overhaul Project
Then that feature has to be reevaluated. If it cannot be made to work with the new architecture, it should be dropped, if only to ease the process of designing and switching over to the new architecture. If that feature is deemed critical (for whatever mod), then it should be put on the fast track toward reimplementation, but getting the engine on a surer footing has to have priority.
If I'm just aching this can't go on
I came from chasing dreams to feel alone
There must be changes, miss to feel strong
I really need lifе to touch me
--Evergrey, Where August Mourns

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Welcome to the Rendering Engine Overhaul Project
karajorma: Your idea cannot work. What happens when an interface changes that mean that the 'features' can't be easily ported to new structures without a _LOT_ of work?  (and that's the whole point of the branch you suggested)

Well there you have the catch 22 which made me suggest that in the first place. Either you branch or you basically tell the mods that they won't get any new features for however long it takes to add multithreading or better graphics. Given that you are talking about years rather than months you're almost certainly going to lose the support of all the TCs and most of the mods if you try it.

The_E is spot on with the solution. You reimplement the feature as fast as you possibly can. And if for some reason you really, really, really can't reimplement it then it means someone ****ed up. Either the overhaul team cocked up the redesign of the engine cause any features that major should have been mentioned as part of the Hosted Projects Collaboration and they should have known it was coming or the mod/TC ****ed up by not having it on their list, in which case it's their problem.
« Last Edit: March 09, 2010, 03:07:20 am by karajorma »
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline FUBAR-BDHR

  • Self-Propelled Trouble Magnet
  • 212
  • Master Drunk
    • 165th Beer Drinking Hell Raisers
Re: Welcome to the Rendering Engine Overhaul Project
Also seems to me that Portej05 suggested a similar approach on IRC yesterday but now is against it because it would require work. 
No-one ever listens to Zathras. Quite mad, they say. It is good that Zathras does not mind. He's even grown to like it. Oh yes. -Zathras

 
Re: Welcome to the Rendering Engine Overhaul Project
At some point you're going to have to break something.
Look at the mess you're in with the cutscenes - one mod has done things one way, another mod wants things another way.
That's the kind of mess you get into when all you worry about is what mods think. You have to be more proactive - the mods need to use what you provide.
If the mods know about the changes that are coming, they can adapt.
STRONGTEA. Why can't the x86 be sane?