Hard Light Productions Forums

Modding, Mission Design, and Coding => FS2 Open Coding - The Source Code Project (SCP) => Topic started by: Nuke on January 10, 2007, 06:24:02 pm

Title: so, how about them shaders?
Post by: Nuke on January 10, 2007, 06:24:02 pm
before you get bogged down for requests for stuff thats already implemented i thought id ask the obvious question. whats the plan for sharers. are they being worked on now or is there alot of other stuff to do first? dont want to rush yea but us modders are getting a tad tweaky.
Title: Re: so, how about them shaders?
Post by: taylor on January 10, 2007, 06:39:51 pm
There is a lot of stuff to do, but not necessarily to do first. :)

I'm going to work on 3 main things in the next couple of months: the new FS2NetD code, OpenGL code upgrades and reorganization (will at the very least lower memory requirements and provide much better support for IG cards, like Intel stuff), normal map support.  During that time I'll also give some coding time to: compressed VP support, new audio code, new ini based system/mod settings (ie, profile.ini support, ground work for the new Launcher), shaders, font code upgrade with TTF support.

None of those extra things actually have any sort of priority though, they will just be worked on as I get the time, and what I feel like working on with that time.  All of that code is already at various stages of completion, so none of it is from scratch and I will basically just be finishing code that I started on last year.  The compressed VP supoprt, the TTF support, and the shader support will likely be completed first since they all have a significant amount of work done on them already.

The "3 main things" is stuff that I want to get into 3.6.10.  There is a very slim possibility that I will also have time to fold in some very basic shader support for 3.6.10 as well.  But since 3.6.10 is largely going to be more bug-fixage, it's not going to get any big new features (like full-blown shader support).
Title: Re: so, how about them shaders?
Post by: Turey on January 10, 2007, 06:56:38 pm
Quote from: taylor on January 10, 2007, 06:39:51 pm
compressed VP support

Sounds very useful. If you can get .zip-level compression, I'll be impressed. if you can get .7z-level compression, I'll be amazed.  :yes:
Title: Re: so, how about them shaders?
Post by: taylor on January 10, 2007, 07:25:35 pm
Quote from: Turey on January 10, 2007, 06:56:38 pm
Sounds very useful. If you can get .zip-level compression, I'll be impressed. if you can get .7z-level compression, I'll be amazed.  :yes:
It's much closer to .7z, but without the higher memory requirements for decompression.  At the moment it's using bzip2, but wolf recommened lzma to me (what 7z uses) and I'm still considering going with that instead.  The test results for decompression speed and memory usage didn't impress me in my tests though, so I'm still leaning towards bzip2 instead.  It will take longer to actually compress the VPs in the first place, but using them in-game should be a little faster than lzma according to my tests.  Memory usage on decompression for lzma is what actually killed it for me, but I'm working towards removing that as an obstacle so that I can re-evaluate lzma based purely on speed and the API (has be easily usable in-game obviously).

Here is copy of a post I made to SCP internal talking about my work on this a few months ago...

Quote from: taylor
Here is what the current 3.6.8 Zeta MediaVPs would look like as CVPs.  They are the VP as it would be installed, the CVP as it would be installed and the VP as you download it (ie, zipped up).

mv_core:
 - as VP: 9.3 Meg
 - as CVP: 2.6 Meg
 - as zipped VP: 2.9 Meg

mv_effects:
 - as VP: 90 Meg
 - as CVP: 21 Meg
 - as zipped VP: 25 Meg

mv_models:
 - as VP: 183 Meg
 - as CVP: 64 Meg
 - as zipped VP: 67 Meg

mv_textures:
 - as VP: 180 Meg
 - as CVP: 77 Meg
 - as zipped VP: 85 Meg

 mv_adveffects:
 - as VP: 245 Meg
 - as CVP: 60 Meg
 - as zipped VP: 75 Meg

Total installed size as VP: 707.3 Meg
Total installed size as CVP: 224.6 Meg

Since mv_music.vp couldn't work compressed, since we have to stream data from it, I'm not listing it in this comparison as the size wouldn't change between VP and CVP. All CVPs are ready as-is for use by the game (when the game side of the code ever gets done).

I switched to using level 6 compression for the files since it's about 3 times less memory needed and a noticeably faster.  Even so, the average file sizes increased no more than 5% on the various tests I made.

That skips the bulk of the conversation, with specifics on speed and memory usage as well as compression method, but this gives you the basics of filesize comparison at least.  That post is from back in June though and I've made several improvements since then.  I'm also adding CRC checking to the VP itself, so that through the game it can automatically detect if the VP is corrupted. :)
Title: Re: so, how about them shaders?
Post by: ni1s on January 10, 2007, 07:51:29 pm
Forgive my failed memory, but wasn't there talk(long ago) about having fs2_open rely more on SDL? Or is that what we are doing(sdl-ttf comes to mind)?
Title: Re: so, how about them shaders?
Post by: taylor on January 10, 2007, 08:46:28 pm
Quote from: ni1s on January 10, 2007, 07:51:29 pm
Forgive my failed memory, but wasn't there talk(long ago) about having fs2_open rely more on SDL? Or is that what we are doing(sdl-ttf comes to mind)?
It will happen, but it's low priority.  Although I haven't comitted the code yet, I do have a tree which has SDL support on all platforms.  It is still off by default for Windows, but can be turned on with a simple compile option.  The only reason for the delay is the amount of code which needs to be removed first in order to keep the code clean and easy to manage.  It's not a hard job, just a long and boring one.  :)

We aren't going to use SDL-ttf though.  Instead we are going to use the FreeType2 lib directly.  Outside of the core SDL lib, we aren't planning to use any of the SDL support libs.
Title: Re: so, how about them shaders?
Post by: Trivial Psychic on January 10, 2007, 10:12:15 pm
Regarding the compressed VPs:  I assume that all existing VP editor and viewer utilities will not be able to read the new format, right?  That means that either the compressor program (like Winzip, if you were to use a zip-based compression) would become the new editor/view software), or that some kind of new, customized VP editor/viewer program would be needed, that can accomplish the task and contain the source code of the compression format.  What's it gonna be?
Title: Re: so, how about them shaders?
Post by: Turey on January 10, 2007, 10:13:40 pm
I would assume VPMage would be able to (eventually) create these compressed vps?
Title: Re: so, how about them shaders?
Post by: Bobboau on January 10, 2007, 10:41:03 pm
I would love to be able to sit down and discuss options for reorganiseing the abstraction layer, I have been asking for literaly years now, and I had a bunch of realy nice code done on my side for organiseing geometry. I'd like to keep D3D support if we can, and I don't think we'll have to sacrafice much to do it.

the graphics code desperately needs reoganisation, look at either OGL or D3D *_render_buffer functions, they are terifying, when in fact they should be small and compact, doing nothing but rendering buffers, instead they are handleing multable passes and all sorts of material settings, and all the compatability code therein.

and look at set_bitmap, this one function is the only way to change alpha blend modes, and texture filtering, WHY!?!?!? there should be three seperate functions, and each one should also be able to specify wich of the 8 texture stages it effects, rather than just the fisrt stage, leaveing the API level code to figure out how to handle stuff by _global ints_ storing the handles of texture files to be used in effects, wich double in posable complexity every time an new one is added. I know I started the whole global int thing, but realy that was a design flaw made by a first year coder who could barely remember the syntax of a for loop, and even then I knew it was a bad idea, I realy expected that to be just a temporary bandaid was of bypassing the crappy bitmap setting code, why we are still useing my horrably flawed anti-pattern after all these years I can't fathom.

I konw we can make a nice simple abstraction layer that everyone would be happy with at least as far as the render level stuff goes (I have no idea how OGL handles things like how it handles blending between it's equivelent to texture stages, but if nothing else we could make one big struct that represented all the settings)

I realy wish I could talk to someone with OGL knowlege about this.
Title: Re: so, how about them shaders?
Post by: taylor on January 10, 2007, 10:52:08 pm
Quote from: Trivial Psychic on January 10, 2007, 10:12:15 pm
Regarding the compressed VPs:  I assume that all existing VP editor and viewer utilities will not be able to read the new format, right?  That means that either the compressor program (like Winzip, if you were to use a zip-based compression) would become the new editor/view software), or that some kind of new, customized VP editor/viewer program would be needed, that can accomplish the task and contain the source code of the compression format.  What's it gonna be?
The existing VP utils will just have to be modified to work with the format.  It is a custom format though, so you couldn't open the files in WinZip or anything else.  Unlike something like the .pak type format that Doom/Quake uses, the CVP spec dictates that each individual file be compressed, not the entire archive.  To work seamlessly with the game, that is how it needs to work.  And fortunately, for the content that we need to compress, we get better overall compression ratios that way too.  As an added bonus we get to filter what files are actually compressed.  This way something like an OGG file, which doesn't compress worth a crap, can get no compression at all and we save the overhead of using it as a compressed file, and all of the other files in the same CVP can still be compressed.  A CVP doesn't have to have any compressed files in it even.  It can still end up being smaller than a similar VP though, since the file index would be compressed regardless and that can save anywhere from a couple of k, to a meg or more.
Title: Re: so, how about them shaders?
Post by: Trivial Psychic on January 10, 2007, 11:18:56 pm
So, basically VPview will become obsolete, and programs such as VPMage, VP Constructor Suite, and perhaps QuickVP would need to be upgraded to work properly?  I assume that the engine will retain support for the standard, uncompressed VPs in order to retain Retail compatibility?  *slaps self in face*  Of course it would... otherwise we'd have to completely re-export the entire game (retail VPs) as compressed VPs for such-equipped versions of FSO to run.  Not a bad idea for things like Shivan's Freespace Packs though.
Title: Re: so, how about them shaders?
Post by: miskat on January 10, 2007, 11:19:51 pm
First, I have to say that thoe figures on the CVP compression results are phenomenal!  Are we only going to be compressing the mediavps or are we going to be compressing the main FS2 vps as well?  Anything to save hard disk space wins in my book.  lol

Also, I have this nifty Hyperspace OGL screensaver that demnstrates to me the effectiveness of pixelshaders in it's own way, so I'm pretty excited, but how can we expect pixel shaders to effect FS2_open?  I mean, in terms of both looks and performance?  I'll assume that decreased performace for enabled pixel shaders is a given, but byhow much are we talking?  Any ideas as to how big of a graphical improvement they will make?

I like what I'm hearing about the coming releases.  Good work, team!
Title: Re: so, how about them shaders?
Post by: Bobboau on January 10, 2007, 11:34:25 pm
moved elsewere
 :shaking:
Title: Re: so, how about them shaders?
Post by: Bobboau on January 10, 2007, 11:36:45 pm
Quote from: miskat on January 10, 2007, 11:19:51 pm
how can we expect pixel shaders to effect FS2_open?  I mean, in terms of both looks and performance?  I'll assume that decreased performace for enabled pixel shaders is a given, but byhow much are we talking?  Any ideas as to how big of a graphical improvement they will make?

actualy the performance we get would probly be better than currently, due to us needing to reorganise things first. though it depends largely on what sort of shaders are used. the graphical improvement will be huge.
Title: Re: so, how about them shaders?
Post by: miskat on January 10, 2007, 11:48:18 pm
Huge graphical improvement with a POSITIVE hit on performance?  Wow, that's a pretty big claim to make.  But it also sounds like a reasonable one.  Definitely exciting!

I can hardly wait!

Damned that I don't know anything past basic C!  I wanna help. >.<
Title: Re: so, how about them shaders?
Post by: Nuke on January 11, 2007, 12:00:12 am
shaders should improve performance for lots of stuff _if_ your card supports them. i have 96 unified shader cores so i dont think il have to wory. shaders improve performance cause they do stuff that would otherwise neeed to be done in software or on other video card resources.

Quote from: Bobboau on January 10, 2007, 10:41:03 pm
I would love to be able to sit down and discuss options for reorganiseing the abstraction layer, I have been asking for literaly years now, and I had a bunch of realy nice code done on my side for organiseing geometry. I'd like to keep D3D support if we can, and I don't think we'll have to sacrafice much to do it.

the graphics code desperately needs reoganisation, look at either OGL or D3D *_render_buffer functions, they are terifying, when in fact they should be small and compact, doing nothing but rendering buffers, instead they are handleing multable passes and all sorts of material settings, and all the compatability code therein.

and look at set_bitmap, this one function is the only way to change alpha blend modes, and texture filtering, WHY!?!?!? there should be three seperate functions, and each one should also be able to specify wich of the 8 texture stages it effects, rather than just the fisrt stage, leaveing the API level code to figure out how to handle stuff by _global ints_ storing the handles of texture files to be used in effects, wich double in posable complexity every time an new one is added. I know I started the whole global int thing, but realy that was a design flaw made by a first year coder who could barely remember the syntax of a for loop, and even then I knew it was a bad idea, I realy expected that to be just a temporary bandaid was of bypassing the crappy bitmap setting code, why we are still useing my horrably flawed anti-pattern after all these years I can't fathom.

I konw we can make a nice simple abstraction layer that everyone would be happy with at least as far as the render level stuff goes (I have no idea how OGL handles things like how it handles blending between it's equivelent to texture stages, but if nothing else we could make one big struct that represented all the settings)

I realy wish I could talk to someone with OGL knowlege about this.

im all for the reworking of stuff like this. id like to have things like less restrictive alpha textures and be able to use more subobjects without slowing the game down. it would really make it possible to use animation features more freely without worrying about bringing the framerate down.
Title: Re: so, how about them shaders?
Post by: taylor on January 11, 2007, 01:42:36 am
Quote from: miskat on January 10, 2007, 11:19:51 pm
First, I have to say that thoe figures on the CVP compression results are phenomenal!  Are we only going to be compressing the mediavps or are we going to be compressing the main FS2 vps as well?  Anything to save hard disk space wins in my book.  lol
Well that's really up to you, and/or who ever you get the retail VPs from.  If they are distributed as CVPs then that's all you need, unless you need to retain compatibility with retail or older FSO builds that is.  There will be a command line utility to convert between VP and CVP though.  It's the same utility set that will initially be ready for messing around with the files anyway (cfilearchiver/cfileextractor, which are both command line VP utilities).

Before any CVPs are made available though, the docs for the file format will be posted so that coders can update the other VP utilities to handle CVPs.  The interface for it is pretty simple (on purpose), so it shouldn't take much more than a few hours of coding time for each app to add full support for CVPs.

Quote from: miskat on January 10, 2007, 11:19:51 pm
Also, I have this nifty Hyperspace OGL screensaver that demnstrates to me the effectiveness of pixelshaders in it's own way, so I'm pretty excited, but how can we expect pixel shaders to effect FS2_open?  I mean, in terms of both looks and performance?  I'll assume that decreased performace for enabled pixel shaders is a given, but byhow much are we talking?  Any ideas as to how big of a graphical improvement they will make?
As Bobboau already said, it will most likely increase performance.  That will depend on the shader types in use though.  We'll be able to do more texturing, lighting and fogging stuff at one time that we do now, with much higher quality, and much faster.  This will not only improve graphics quality, but also open up the use of even more features, like proper shadows.  The shader code will even be available to help render Theora movies much more efficiently. 
Title: Re: so, how about them shaders?
Post by: PzyCrow on January 11, 2007, 02:57:11 am
I did some measurements on different compression tools (bzip2, lzop and ucl)
results:
Both lzo and ucl decompress in-place so no memory requirements

.vp time is just the time to read the file from disc
mv_core.vp9.3Mdecompression time: 0:00.04
mv_core.vp.bz22.5Mdecompression time: 0:01.55
mv_core.vp.lzop3.4Mdecompression time: 0:00.05
mv_core.vp.ucl3.2Mdecompression time: 0:00.09
mv_effects.vp90Mdecompression time: 0:00.09
mv_effects.vp.bz221Mdecompression time: 0:10.25
mv_effects.vp.lzop29Mdecompression time: 0:01.63
mv_effects.vp.ucl28Mdecompression time: 0:02.43
mv_models.vp183Mdecompression time: 0:04.21
mv_models.vp.bz264Mdecompression time: 0:31.40
mv_models.vp.lzop78Mdecompression time: 0:03.20
mv_models.vp.ucl73Mdecompression time: 0:04.62
mv_textures.vp180Mdecompression time: 0:04.47
mv_textures.vp.bz280Mdecompression time: 0:37.31
mv_textures.vp.lzop97Mdecompression time: 0:03.44
mv_textures.vp.ucl93Mdecompression time: 0:14.26
mv_adveffects.vp245Mdecompression time: 0:15.53
mv_adveffects.vp.bz260Mdecompression time: 0:33.75
mv_adveffects.vp.lzop91Mdecompression time: 0:06.97
mv_adveffects.vp.ucl87Mdecompression time: 0:07.83
Title: Re: so, how about them shaders?
Post by: taylor on January 11, 2007, 03:57:16 am
Both LZOP and UCL are GPL'd, so we couldn't use them anyway.

The choice is basically just between LZMA and BZIP2.  They both have about the same compressed size, so it's only decompression speed and API to consider.  My tests didn't give LZMA very good decompression speeds compared to BZIP2 though, contrary to what I've seen in other benchmarks.  The main thing for me right now is that BZIP2 has a better API.  It will basically be transparent to plug into the existing CFILE code and it will automatically handle both normal and compressed files without having to code in tests for them.
Title: Re: so, how about them shaders?
Post by: PzyCrow on January 11, 2007, 05:04:49 am
Quote from: taylor on January 11, 2007, 03:57:16 am
Both LZOP and UCL are GPL'd, so we couldn't use them anyway.

Oh, I hadn't realized. To bad.

Edit:
Quote
http://www.oberhumer.com/opensource/lzo/ (http://www.oberhumer.com/opensource/lzo/)
LZO and the LZO algorithms and implementations are distributed under the terms of the GNU General Public License (GPL)  . Special licenses for commercial and other applications are available by contacting the author.

So I guess if you're intereste it is possible to get a custom license. I mean, it isn't exactly a MAJOR licens-incompatibility.
Title: Re: so, how about them shaders?
Post by: Mehrpack on January 11, 2007, 11:50:21 am
Quote from: Bobboau on January 10, 2007, 10:41:03 pm
[...]

I realy wish I could talk to someone with OGL knowlege about this.

hi,
i read in a great german 3D forum and there is are guys with very good, in my minds, knowlege about OpenGL and/or Direct 3D.

on of that guys is zeckensack, he know much about OpenGL and writing some time ago and OpenGL 3Dfx wrapper.
- profile: http://www.forum-3dcenter.org/vbulletin/member.php?u=2038
- private message: http://www.forum-3dcenter.org/vbulletin/private.php?do=newpm&u=2038

a other guy is demiru, he work as programmer for a software-developer (dont know ifs game only or other software too) and he has really much cognition about Direct 3D and OpenGL.
he has programm some tools for D3D and the direct tweaker.
- profile: http://www.forum-3dcenter.org/vbulletin/member.php?u=2714
- email in the forum: http://www.forum-3dcenter.org/vbulletin/sendmessage.php?do=mailmember&u=2714
- privvat message: http://www.forum-3dcenter.org/vbulletin/private.php?do=newpm&u=2714

i think some of that guys can you help or will you help maybe, but i dont know it exactly.
but i thinks its a chance and why not try it, more as no anwser you can get.

Mehrpack
Title: Re: so, how about them shaders?
Post by: Scooby_Doo on January 14, 2007, 11:44:01 pm
Too bad you can't use zip or rar or 7z for the archives.  Then MediaVP won't be needed at all.