Author Topic: Shader versioning test  (Read 6919 times)

0 Members and 1 Guest are viewing this topic.

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Shader versioning test
There's a pull request to hopefully fix shader compilation on certain drivers, but it may break others. In the interests of collecting data, we'd like people to run both a recent Nightly Build and this test build with shaders enabled and report if there are any differences (e.g. if you can run a normal build with shaders, but can't run this build with shaders, or vice versa, let us know and share your debug log).

SSE2: http://deviance.duckish.net/downloads/fs2_fred2_open_PR316_SSE2.7z
AVX: http://deviance.duckish.net/downloads/fs2_fred2_open_PR316_AVX.7z

Please note that since we want to test shader compilation, you will need to remove the -no_glsl ("Disable GLSL (shader) Support") command-line option (and probably -disable_glsl_model / "Don't use shaders for model rendering" as well) for this test.
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline m!m

  • 210
Re: Shader versioning test
If you disabled the commandline options AdmiralRalwood mentioned you should be able to test it by simply going to the Tech Room and looking at a ship model.

 

Online chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️À➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Shader versioning test
It would help to use the MediaVPs so that we actually are using most of the shaders too?
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline Echelon9

  • Moderator
  • 210
Re: Shader versioning test
The background context here is standards compliance with the OpenGL GLSL shader language.

Per the specifications, it is clear that the start of a shader (whether a fragment, vertex, or geometry shader etc) *must* begin with a version declaration which informs the driver which feature set is expected by the program / shader. e.g.

Code: [Select]
#version 120
// Shader code goes here

Now driver conformance with the OpenGL specification is a mixed bag. As engine developers, we have to work with these quirks in the same way web developers have worked around browser-specific interpretations.

In this case, the specification is clear we *must* include a #version directive. We want to understand from this test which combinations (hardware, driver and OS) correctly interpret this. i.e.
  • Mesa (Intel, nVidia, and AMD; open source driver on Linux): correctly requires a #version directive.
  • nVidia GeForce (nVidia, closed source driver on Windows/Linux): correctly requires a #version directive?
  • AMD Catalyst (AMD, closed source driver on Windows/Linux): barfs with a #version directive and certain OpenGL features.
  • OS X ([all?], closed source driver on OS X): correctly requires a #version directive.
Now it may also be that our fs2open shaders are making errors too, which we can confirm by putting them through a reference GLSL compiler.

As GPUs become fully programmable devices, the former fixed function pipeline approaches are less relevant and shader-driven approaches come to the fore. We'd like to better understand the environment we are working with at the moment.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: Shader versioning test
OK - recent master builds work, but a build from the PR does not. It exits as soon as I view a ship in the ship lab or tech room.

OS: Linux Mint 17 64bit
Card: GeForce GTS 450
NVIDIA Driver Version: 340.76 (closed source drivers supplied by distro)

Log here, as a wild guess I think my setup wants #version 140 in the shaders rather than #version 120 :)
« Last Edit: August 24, 2015, 02:45:46 am by niffiwan »
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Linux OBS Packages: FSO 3.7.0 | FSO BP Build | wxLauncher (?) | PCS2 (?) | wxVPView (?)
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Re: Shader versioning test
* AdmiralRalwood wonders if this means the AMD drivers would work with #version 140
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline m!m

  • 210
Re: Shader versioning test
OK - recent master builds work, but a build from the PR does not. It exits as soon as I view a ship in the ship lab or tech room.

OS: Linux Mint 17 64bit
Card: GeForce GTS 450
NVIDIA Driver Version: 340.76 (closed source drivers supplied by distro)

Log here, as a wild guess I think my setup wants #version 140 in the shaders rather than #version 120 :)
Looks like another error on our side. I think Nvidia wants to have #extension GL_EXT_gpu_shader4 : enable or #version 140 in the shader. We use OpenGL 2.1 so enabling the extension seems like the way to go. Can you try inserting "#extension GL_EXT_gpu_shader4 : enable\n" in code/globalincs/def_files.cpp:1340 and see if that fixes the error? We might have to do additional extension checking if we want to do that though.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: Shader versioning test
OK, that removed one error, just leaving this:

Code: [Select]
0(28) : warning C7508: extension ARB_texture_buffer_object not supported
0(70) : error C7532: global function texelFetch requires "#version 130" or later

And for the heck of it, I tried setting #version 140, and even more errors spat out  :sigh:

Code: [Select]
0(28) : warning C7508: extension ARB_texture_buffer_object not supported
0(28) : warning C7555: 'attribute' is deprecated, use 'in/out' instead
0(35) : warning C7555: 'varying' is deprecated, use 'in/out' instead
0(61) : warning C7555: 'varying' is deprecated, use 'in/out' instead
0(64) : warning C7555: 'varying' is deprecated, use 'in/out' instead
0(65) : warning C7555: 'varying' is deprecated, use 'in/out' instead
0(102) : error C7616: global variable gl_TexCoord is removed after version 140
0(102) : error C7616: global variable gl_TextureMatrix is removed after version 140
0(102) : error C7616: global variable gl_MultiTexCoord0 is removed after version 140
0(103) : error C7616: global variable gl_Vertex is removed after version 140
0(109) : error C7616: global variable gl_ModelViewMatrix is removed after version 140
0(118) : error C7616: global variable gl_ProjectionMatrix is removed after version 140
0(120) : error C7616: global variable gl_Color is removed after version 140
0(123) : error C7616: global variable gl_NormalMatrix is removed after version 140
0(123) : error C7616: global variable gl_Normal is removed after version 140
0(156) : error C7616: global variable gl_ClipVertex is removed after version 140
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Linux OBS Packages: FSO 3.7.0 | FSO BP Build | wxLauncher (?) | PCS2 (?) | wxVPView (?)
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 
Re: Shader versioning test
Seems both builds work on win 7 and ancient amd drivers 13.12.
Log from PR316 shows:
Code: [Select]
  Using OpenGL driver workarounds:
    No #version: Don't add #version to OpenGL shaders
    <none>

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Re: Shader versioning test
Yes, the build currently does not add the #version line to any AMD drivers due to it already being known that they don't play nice with it.
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline m!m

  • 210
Re: Shader versioning test
Did you try going to the tech room or playing a mission? The shader problems usually only happen when one of the main shaders is compiled.
Now that I think about it, it would be nice to add a way to override which workarounds are used for testing purposes. I'll try to add that to the PR.

 
Re: Shader versioning test
I checked Sathanas, Arcadia in Tech Room and Ship Lab, tested in mission and it works (no visible difference).

Tested on Xubuntu 15.04 64bit with Catalyst 15.3 and it works.
« Last Edit: August 27, 2015, 06:43:45 pm by tomimaki »

 

Offline m!m

  • 210
Thanks for all your help but I can't figure out how to properly fix this for Nvidia drivers. Our OpenGL rendering engine is pretty much FUBAR in my opinion and without in depth knowledge of the rendering code there is no way I can fix this.

 

Offline The Dagger

  • 29
  • I like zod ships
Have some strange issues with this build (SSE2 version) but none with nightlies. In the F3 lab, it looks like only subobjects detached from hull (like the cockpit glass or moving parts) are rendered correctly while the first subobject is fixed in a certain direction (original position IIRC, since I'm looking straight into the thrusters). In-mission, those objects are invisible.

OS: Windows 7
Card: Nvidia GeForce GTX 460M
Driver version: 355.60

Here's a log running the latest MediaVPSs: http://pastebin.com/iJMcvSz9

There was a HUGE amount of repeated
Code: [Select]
ERROR! Unable to create vertex shader!
Compiling new shader:
        Model Rendering
   Loading built-in default shader for: main-v.sdr
   Loading built-in default shader for: main-f.sdr
Vertex shader failed to compile:
0(34) : warning C7508: extension ARB_texture_buffer_object not supported
0(35) : error C7532: global type samplerBuffer requires "#version 140" or later
0(35) : error C0000: ... or #extension GL_EXT_gpu_shader4 : enable
0(75) : error C7532: global function texelFetch requires "#version 130" or later
that I deleted in order to upload it.

 
I don't have account on Github so I write here.
Vendors for Mesa depends on GPU
and for Intel is:
Code: [Select]
OpenGL vendor string: Intel Open Source Technology Centerfor AMD opensource driver:
Code: [Select]
OpenGL vendor string: X.Orgfor NVIDIA opensource driver:
Code: [Select]
OpenGL vendor string: nouveau
So maybe easier add the same workaround for NVIDIA Corporation like for ATI Technologies Inc.? :)

 

Offline m!m

  • 210
@tomimaki: Thanks for providing the vendor strings. Those are really hard to identify so I just enabled the version workaround for NVIDIA which should fix the issues without the need for more shader fixing.

@The Dagger: Thanks for the log, our shaders don't really like the #version at the top. The latest commit to the PR should fix those issues but I can't compile new builds right now.

 

Offline AdmiralRalwood

  • 211
  • The Cthulhu programmer himself!
    • Skype
    • Steam
    • Twitter
Downloads have been updated with fresh builds.
Ph'nglui mglw'nafh Codethulhu GitHub wgah'nagl fhtagn.

schrödinbug (noun) - a bug that manifests itself in running software after a programmer notices that the code should never have worked in the first place.

When you gaze long into BMPMAN, BMPMAN also gazes into you.

"I am one of the best FREDders on Earth" -General Battuta

<Aesaar> literary criticism is vladimir putin

<MageKing17> "There's probably a reason the code is the way it is" is a very dangerous line of thought. :P
<MageKing17> Because the "reason" often turns out to be "nobody noticed it was wrong".
(the very next day)
<MageKing17> this ****ing code did it to me again
<MageKing17> "That doesn't really make sense to me, but I'll assume it was being done for a reason."
<MageKing17> **** ME
<MageKing17> THE REASON IS PEOPLE ARE STUPID
<MageKing17> ESPECIALLY ME

<MageKing17> God damn, I do not understand how this is breaking.
<MageKing17> Everything points to "this should work fine", and yet it's clearly not working.
<MjnMixael> 2 hours later... "God damn, how did this ever work at all?!"
(...)
<MageKing17> so
<MageKing17> more than two hours
<MageKing17> but once again we have reached the inevitable conclusion
<MageKing17> How did this code ever work in the first place!?

<@The_E> Welcome to OpenGL, where standards compliance is optional, and error reporting inconsistent

<MageKing17> It was all working perfectly until I actually tried it on an actual mission.

<IronWorks> I am useful for FSO stuff again. This is a red-letter day!
* z64555 erases "Thursday" and rewrites it in red ink

<MageKing17> TIL the entire homing code is held up by shoestrings and duct tape, basically.

 

Offline The Dagger

  • 29
  • I like zod ships
Re: Shader versioning test
Tested the new builds, it's all good ingame and no errors are reported on the debug log (NVidia GPU).

 

Offline Echelon9

  • Moderator
  • 210
Because we have a good group of motivated OpenGL testers here, thought I'd hijack this thread to provide an update on recent fs2open renderer refactoring that I've been looking at.

Branch for code here (no binaries as yet): https://github.com/Echelon9/fs2open.github.com/tree/fix/SHADER_MODEL-concept-removal

Background

On reviewing the OpenGL GLSL setup code in fs2open, it was apparent we use the 'Use_GLSL' variable to encapsulate three different concepts. This is a mess.

At present, depending on the value, this variable is used for:

  • Zero: Implies all GLSL shaders are disabled.
  • One: Implies a deliberately partial GLSL shader support, as set by -disable_glsl_model command line option.
  • Two+: Implies a "Shader Model" level, which is a DirectX concept and not relevant to OpenGL.
The last bullet point is important, as the code readability is improved by removing legacy references to DirectX-only concepts. A broad mapping between DirectX Shader Model versions and GLSL versions can be found here: https://www.opengl.org/wiki/Detecting_the_Shader_Model.

Implementation: Refactoring 'Use_GLSL' variable

So, in order to do something about this, I've refactored the 'Use_GLSL' global variable to properly encapsulate only one concept. The branch does this by splitting the concept of (a) the GLSL version reported by the driver, and (b) whether a command line option has been used to disable GLSL shaders all together.

The third option, -disable_glsl_model appears less relevant today than when it was added. Today, most driver and GPU combinations will happily support > OpenGL GLSL 3.30 (cf. Steam Hardware Survey). There are a small number of users on older hardware that won't support GLSL shaders at all. There are significantly less in between, with partially working driver implementations. So I've taken this opportunity to remove a little used command line option.

The #version directive, which is required for standards conformance, is added in a dynamic manner. Am hoping this works across implementations, but I don't have all the hardware to test.

I'd be keen to see how users with a variety of GPUs and drivers handle this branch. Please test away.

Particularly keen to see if errors along the lines of "varying no longer support", or "global variable gl_TexCoord is removed after version 140".

Mesa open source drivers

Importantly, this branch was developed on an Intel GPU using the Linux Mesa open source drivers. The tooling and infrastructure is frankly fantastic. I want to be clear that the support and ability to see how the driver is internally handling the shaders is very helpful, and hopefully dispel some long-held, but now unsupported negative views on recent Intel GPUs.

I'll have a follow-up post that includes performance figures benchmarking the fs2open engine shaders through various Mesa versions and their shader-db tool, to see how they have improved of late.

 

Offline Swifty

  • 210
  • I reject your fantasy & substitute my own
An idea I had floating in my mind was to have something called gr_is_capable() which would be some sort of function other parts of the code outside of the gr_screen specific code that would refer to when trying to figure out if a particular piece of code needed to be run or not.

gr_is_capable would accept a bunch of identifier constants like "CAPABILITY_DEFERED_LIGHTING", "CAPABILITY_ENV_MAPPING", "CAPABILITY_SOFT_PARTICLES", etc and would do all the work in determining if an end system is capable of a certain graphics feature.

Internally inside the gr_opengl code (and hypothetically the gr_vulkan code), we would have a table in the backend that lists all the pre-requisite extensions and shader versions needed for a particular graphics feature. So that way, the only code that needs to really know about extensions or a shader versions would be in the code that actually deals with OpenGL.

Echelon9's stuff seems to be a good step in the right direction.