Hard Light Productions Forums
Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: Aardwolf on August 06, 2009, 10:38:35 pm
-
So, it seems that the FSO rendering engine is kind of outdated... even with the fairly recent addition of GLSL support, it fails at many simple things like proper draw order / depth sorting.
I, for one, believe that it would be better to attempt a ground-up rewrite of much of the rendering code, rather than be limited by the capabilities currently available, and having to deal with increasingly messy code as small patches and improvements begin to mess with each other. The idea is to do a big project now rather than suffering indefinitely from the obsolescence of the rendering engine.
I believe this is doable, mainly because it was done to give FSO OpenGL support.
Discuss.
I personally have a decent amount of experience with OpenGL, so I could in theory assist with such an endeavor, but since I'm in a year-round college, even during the summer I can only offer freelance contribution...
-
Have fun.
-
I am The E, and I like this.
First thing to do then would be to identify all the data that goes into the engine from the AI, player handling etc., as well as finding out what data it sends back out. Don't know a thing about rendering engine code, though, so I have no idea how much work would be involved...
-
Seems you don't know exactly what I'm talking about. :(
-
Lately inefficiency of texture rendering in fs2_open has been revealed to me. If anything, that part of the code needs serious overhaul. But like almost everyone here, I can't help at all.
-
i think its something that should be done. but nothing im qualified to work on.
-
I think one should fix existing bugs first before taking on a greater challenge.
-
what if the ancient rendering engine is the source of those bugs? :D i mean you can spend time debugging old, inefficient, poorly organized code, or you can use the time to re-code things which should have been deprecated years ago.
-
I totally agree :D
"stupid" things like not rendering large frame effects properly, not being able to have differant blending options for effects, and using to many resources to render a them...
-
What you say would apply if at least gameplay affecting bugs are fixed. Which isn't the case.
-
Since we're at this subject - if multile maps share the same normal map (or glowmap or shinemap), you still have to make a "different" one for each map, thus wasting much more memory and slowing things down.
Would it maybe be possible to somehow enable map sharing? (I dunno, like maybe a text file with the name of the "original" map, followed by the names of all other maps that share it's normals, glow or specular maps)
-
What you say would apply if at least gameplay affecting bugs are fixed. Which isn't the case.
That said, sometimes we're having trouble tracking down those monsters - there's apparantly a laundry list of multi issues to look at soon as well (including behaviour changes from retail->FSO).
I'd support this endeavour - I had a look through some of the rendering code this afternoon and have barely recovered.
-
If you do, it's branch time! And it would have to stay as a "branch" for a long, long time...
IMHO it's probably not really feasible, since so much of the game and modding functionality is built around the current rendering engine, quirks and all. But I'd be happy to be proven wrong. :D
-
Such efforts would almost certainly have to be undertaken in the context of a 3.7.x (or similar branch), with the note that backwards compatibility would be deprecated where it conflicted with the interests of re-design and re-coding of a modern OpenGL engine.
-
IMHO this is a *must* for the long term developability of the engine. However as of right now, there are no qualified coders to carry out. Moreover this will be a long term project with results at a far off time...
...so yeah, branch it is, and time to learn OpenGL in a C++ environment. Another thing that could be done would be the objectification/modularization of the existing code to better suit further development.
-
/me dons armour and flak jackets, buys tank and thick concrete bunker
-
Is it even possible to "rip out" the current engine and replace it with something better (the level of modularity that exists there)?
If it is, would it be possible to replace it with something that already exists?
If it isn't, won't the new engine be a mess too? Unless you rewrite a lot more than just the engine?
-
I had a look through some of the rendering code this afternoon and have barely recovered.
;7
You just revlead that you have knowledge about rendering code... (run!)
But honestly, how should this be realized? When should the rewrite begin?
There are many more questions concerning this.
FS2 is multiplatform now, should we stick to OpenGL?
Which version? 3.0, 3.1, or the new 3.2?
I suppose we cannot just take another rendering engine with a suitable license and put it into the game without more effort than writing a new one.
-
I have this itchy feeling Kazan will burst in any moment now yelling "Told ya so!" while holding a neon sign reading "Ferrium". ;)
-
You do have to wonder when conversation along these lines goes from "upgrading FS2_Open" to "coding a brand-new space sim engine." :p
-
It's known that the engine isn't perfectly compartmentalized yet, but short of rewriting it completely, could work be done to fix those areas where it's not, as it pertains to the graphics code?
-
I am going to show total ignorance of the technology involved here but is a stage by stage upgrade available for example
3.7.1
Upgrade render order
3.7.2
upgrade texture handling
e.t.c.
-
You do have to wonder when conversation along these lines goes from "upgrading FS2_Open" to "coding a brand-new space sim engine." :p
Well, that's exactly what I'm afraid it would take. But like I said, I'd be happy to be proven wrong.
-
Have a look through modelcollide for some truly horrible coding. Allow me to elaborate:
// Textured Poly
// +0 int id
// +4 int size
// +8 vec3d normal
// +20 vec3d normal_point
// +32 int tmp = 0
// +36 int nverts
// +40 int tmap_num
// +44 nverts*(model_tmap_vert) vertlist (n,u,v)
void model_collide_tmappoly(ubyte * p)
They didn't use a struct for 'p' - they've used offsets within 'p'....
And they've done it enough to probably require a fair amount of work to fix it - I think this might be worth looking at in Antipodes (it shouldn't be behaviour changing, but if a mistake is made, it might require some sharp eyes to spot errors).
-
i was contemplating the idea of support for the physx api to handle collision detection. i ha planned to implement it in my own game engine first. but it hasnt gone anywhere.
-
Changing the physics behaviour would probably screw up the AI in some way or other :(
I'd really like to see a lot of the code that handles the models be cleaned up, made more robust and modular. (think supporting new formats or something like that)
BTW, where are Goober or karajorma or taylor? I'd have expected them to weigh in at this point (although, kara said he's going to be away for a while)
-
doing the actual physics with the physx api might screw up the feel of the game, but using it for collision detection might speed things up a lot.
-
Yeah, collision detection is the biggest framerate-sucker right now (next to rendering). It's definitely a good target for improving game speed.
-
Yeah, some tests revealed that the collision detection costs a lot(!) of CPU time. Considering that the engine doesn't take advantage of multiple cores, this is really annoying.
I think the collision detection is the main reason for all slow-downs in SoL.
-
To be honest, I believe it is time to give up on retail compatibility and create a branch of fs2_open which focuses on optimal code instead of maintaining backwards compatibility. That said, it is still possible to port FS1/FS2 data and material to be compatible with the new engine when it has matured to a stable point. It's just that the original as-is wouldn't work. The port wouldn't be 100% accurate original FS experience, but you'd still get the play the original campaigns in brand new engine.
And remember, the current branch wouldn't disappear anywhere. So it is not like you lose what you have now. Only that new development would be focused on the branch, instead of this nightmare of a code and inefficiency we have now.
SCP and FSU have both outdone themselves by maintaining backwards compatibility and yet providing such amazing engine that is fs2_open. But it has its limitations which cannot be removed without removing backwards compatibility. Instead of being stuck with what was coded back in the 90's, SCP should be given the chance to bring us a modern engine that plays like a dream on modern PC's.
I'm all for maintaining backwards compatibility, but when it becomes more trouble than it is worth, it's time for a change. Port the data to be compatible with new engine.
-
About time someone admitted to wanting to take FreeSpace one step further.
-
It was never an issue about the comunity not wanting to make a new engine from scrach, the issue is that it's too much work for just one or a few... so who's gonna take the heavy burden of doing all that work?
I might help but I'm not an expert coder and honestly every day that passes by I get less time to fiddle around with FS :S
-
To be honest, I believe it is time to give up on retail compatibility and create a branch of fs2_open which focuses on optimal code instead of maintaining backwards compatibility.
UGH!
This one comes up once in a while and it's a very bad idea.
- We're basically talking about a fork in the code here. We're not talking about a branch cause at no point will this new code every be merged back into trunk. And any coder on the SCP who isn't getting chills at the idea of maintaining two increasingly disparate codebases with the skeleton crew of coders we currently have really needs to question their sanity. There is a very good chance one codebase or the other would quickly end up being discarded and forgotten about. This is exactly what happened with the HEAD vs Stable branch and the result is that all kinds of code went into HEAD and after the switch to SVN no one has any idea what most of it was or how to get it back out without a line by line review of the code. And to be honest given the reasons I'm going to go into below, I suspect it will actually be the new one that goes. That's exactly what happened with HEAD and it will happen here for much the same reason + Several new ones.
- The issue of how major the changes would is a major one. If the changes are minor then some (but not all) of my objections vanish. However, Portej05 is already talking about changing the way pofs work. That basically invalidates every single pof in the world. Not to mention making PCS II worthless for working on this branch until it is also changed. And since the changes are also going to accumulate there we might even be talking about a fork there too. At which point Kazan is going to turn up and throw rocks at you since he hates forks even more than I do. :p
- Okay, you've invalidated all the game data. How the hell are you going to test the engine works now? We've already seen that major bugs can creep into the codebase in between releases because people don't play the entire campaign and point them out all the time. So now we're talking about not being able to test the whole campaign works for somewhere between several months and a year or two while this process is going on. I'm getting chills thinking how badly this is going to affect stability by the time the new fork is ready for serious testing.
- Furthermore you've basically stuck the existing mods and TCs in a position where they will be using the original fork and not this one. No sane project is going to put up with the kind of disruption this would cause to their work. They'll just wait until you're done and use it then. Except when they try they're going to find a lot of their data doesn't work either. Meaning you've added several months of dev work to the mods and TCs that do use this fork.
- They're going to want to add code for new features too. Which is definitely going to break on the new fork quite often.
- Once you break compatibility the first time you've lost the main argument against doing it again. So the cycle starts again when some coder finds a major problem that needs a compatibility break. And we will undoubtedly find them. So either we make the same arguments against a second compatibly break I've just made or we say no and leave the broken code in. Which means real pressure to catch every bug in the first break and then means that all we've done is simply reset which compatibility issues we're stuck with, sure some of the :v: ones will be gone but I'm not arrogant enough to assume we're not going to make some howlers of our own.
- So if this goes ahead we're not going to see any progress in FSUP or any other project which uses this engine for a very long time. Yeah you still have the old version, but either FSUP keep improving that (and making themselves more and more loathe to swap) or they keep using the new engine and don't make much progress for a very long time
So my point of view is that if we could magically break compatibility and automatically switch all the mods over to the new system, I'd say go for it. But my main objection to the idea is that we'd probably end up with an engine that was in development for a very long time with no data to use on it.
-
To put this in context:
// Flat Poly
// +0 int id
// +4 int size
// +8 vec3d normal
// +20 vec3d center
// +32 float radius
// +36 int nverts
// +40 byte red
// +41 byte green
// +42 byte blue
// +43 byte pad
// +44 nverts*int vertlist
void model_collide_flatpoly(ubyte * p) {
...
mc_check_sphereline_face(nv, points, vp(p+20), fl(p+32), vp(p+8), NULL, -1, p);
...
modelcollide.cpp is a mess, and one that is not going to be fixable with the way things are currently set up (think compiler structure alignments for starters - if everything is 4 byte aligned, not so bad, but this doesn't appear to be the case everywhere - not to mention the single letter macros all over the place).
Additionally, in modelread.cpp in read_model_file, we find:
cfread(pm->submodel[n].bsp_data,1,pm->submodel[n].bsp_data_size,fp);
i.e. bsp_data is stored within the POF file itself (if this is wrong, someone correct me). Which is additional badness because the structure of bsp_data cannot under any circumstances be changed without breaking everything.
I'd love to see an updated roadmap posted (even if it's only Kazans internal roadmap), based on what we've currently got, 3.6.11 doesn't look remotely like the roadmap currently.
-
Yeah the current roadmap, well, sucks. It's not even really guidelines at this point, at least not the way the numbers line up. Coders that proposed many of those ideas are no longer working on them and they will likely never see the light of day. However, an engine rewrite would not have to mean elimination of backwards compatibility. If the code were better structured, support for all legacy formats could be maintained while opening up support for new ways of doing things. D2X-XL was rewritten from scratch and it fully supports D2 _and_ D1 assets. There's no reason FS2 couldn't achieve the same with FS2. However, for the time being, incremental rewrites are still possible and could wring a lot out of the engine. In the end it comes down to what the coders are motivated to do. If people want to keep tweaking, that's what will happen. We can't just order everyone to stop work on this engine and start rewriting a new one, but if there's motivation to that end, then it's always a possibility.
-
Basically, if we are going to change the rendering engine, we first need to compartmentalize so that we have:
Artwork Files --> Rendering Engine --> Rest of Game
all cleanly separated and working, WITHOUT changing the rendering engine.
Then, you'd have to figure out how to replace the rendering engine without changing any of the forward or backward dependencies. :)
In other words, let's make the rendering engine truly modular before we even talk about replacing it. The only way it's ever going to work is if we can truly treat the rendering engine as separate from the rest of the game (which we currently can't). Otherwise, we have the nasty scenario Kara outlined...
-
Yep. I really can't see why we need to lose retail compatibility in order to upgrade the engine anyway. The issues with retail can simply be dealt with by flagging the newer data in some way. Yes, that increases the size of the code base and makes it more work to add features, but it's also the only sensible way I can see of doing things.
-
And if it's properly modularized, it should be easier to work with than it is now.
-
Or, don't bother flagging the newer data, but flag internally the old data.
For missions, we already have the in-engine list so that they give the :v: Icon. For POF's, this would be a bit tricky, and I'm sure "new" pof's can be flagged someway, maybe by changing the extension so that it can load compatibility data (and ignore additional data) if used in a situation where compatability is needed.
-
We can't internally flag all old data. That would include all third party assets, campaigns, models, etc. We don't know what exists for certain, so I don't think that's very feasible. Simply marking stuff as "v.2" in its header for all the new stuff shouldn't be that big of a deal.
-
Ah. Well, in that case, I heartily agree.
-
ya know, I was just thinking about this.
I think this all points at an endeavour which we keep putting off, but that has to be done eventually.
Thorough engine documentation.
I don't just mean the randomly asserted FS Wiki stuff we've currently got (e.g. table structures for the modders), but more at the structure and use level.
Going through the engine to detail this stuff would probably be worth it in the long run. Even if we only gain a better understanding of limitations.
On that note, is the POF format, the VP format and other formats and their identifying characteristics actually formally documented anywhere?
-
The original POF format is documented somewhat, but it's been expanded since then.
-
I'd find it humorous if after another five or even another entire decade of the FSU Project it still played the same as it did since it first came out, it would be like playing Asteroids with X3: Terran Conflict's graphics.
-
Hah, there isn't much in the way of polishing or expansion you could do to the actual gameplay.
FS2 got it right in one go.
-
I'd find it humorous if after another five or even another entire decade of the FSU Project it still played the same as it did since it first came out, it would be like playing Asteroids with X3: Terran Conflict's graphics.
Something like this?
(http://image.com.com/gamespot/images/2007/331/936563_20071128_screen001.jpg)
(http://image.com.com/gamespot/images/2007/331/936563_20071128_screen010.jpg)
(http://image.com.com/gamespot/images/2007/331/936563_20071128_screen015.jpg)
-
The original POF format is documented somewhat, but it's been expanded since then.
Has the whole thing been documented, and where can I find it?
(and the VP format, and the table formats, and the etc...)
-
The best documentation you're going to find for POF structure and table files is the code itself. As for VPs, the wiki may hold some answers (http://www.hard-light.net/wiki/index.php/*.VP) ;)
-
If that's the case, I'm not going anywhere near code that deals with POFs.
-
It's pretty easy to understand the basic structure of the files, as well as the layout of many of the simpler chunks. And this is coming from someone who isn't really a hard core coder ;) I'd imagine the structure of the BSP data is a little more difficult to understand, but that's another matter :shaking:
-
The BSP data is read as a binary chunk from the POF file, and then hard coded offsets are used within those data blocks to navigate around. There's no formal documentation anywhere on the BSP tree (other than a brief explanation at the start of code blocks as to what it expects at each offset - but this changes function to function...).
Problem with just reading the code is that if you miss something during reverse engineering, you've wasted a lot of time, and you'll spend a long time defending your code (and ultimately, it'll be yanked out and circular filed).
Also, the POF format has changed (apparantly) since retail, which doesn't help things.
-
The biggest change I think was Kazans additions (via PCS2) for SLDC chunks.
-
Which I'm guessing wasn't fully documented, it was just coded?
-
We don't need no stinkin docs.
-
Weren't glowpoints added at some point too?
You could always try traipsing through the pcs2 code as well.
-
If that's the case, I'm not going anywhere near code that deals with POFs.
theres also the pcs2 source, which is a little bit better organized
-
Any chance of fs2_open receiving working SSE2/3/4 and x86_64 support to give our poor CPU's some breathing room while rendering engine is being overhauled?
-
64 bit would be nice
-
64 bit would be nice
Can't happen without a fair little overhaul - there's a number f places in the engine that make assumptions about 32bit prcessors. Also, you can't use inline assembly in a 64bit build (so int 3 needs to be changed - DebugBreak IIRC)
-
It needs inline assembly?
-
There're a few places inline assembly is used through the engine.
It's mostly debug code, and I'd have to go chasing for anything that wasn't.
Any that's found needs to be replaced with equivalent calls - VS2008 doesn't like inline assembly for 64bit targets.