[The E]

Just curious: the result looks a heck of a lot like Crysis 2's AA filter. Is it the same thing?

(Also, I'd appreciate anything that could be done for performance. The current AA already drops my framerates by 10-20%, which is quite a bit when I can get as low as 22 fps without the FXAA)

ED: For curiosity sake, I did a bench to compare the hit of the ver1 FXAA shaders.

With FXAA: 30 fps
Same scene without FXAA: 77 fps

Ver2 FXAA shaders:

With FXAA: 58 fps
Same scene without FXAA: 76 fps

Core2Duo 2.4 GHz, Mobility 3650 HD @ 593/791. The hardware is ****tier than that found in a console, so I think I'll stick with my ver2 shaders, if at all

Have you done these tests in the ship lab? The problem with the lab is that framerate achieved there are not at all representative for ingame performance, and very sensitive to changes in the rendering infrastructure (like FXAA). I will still maintain that the performance loss under real-life situations is negligible.

Anyway, a few tests of my own show that FXAA adds somewhere ~2 Milliseconds to the render time per frame (On a Radeon Mobile 5470). This will have some effect on the global framerate, but an effect that actually decreases as FPS get lower.

[Zacam]

While these (and the attendant code) are still being moderately tweaked with, seems nVidia cards (despite this being nV generated) has a few issues, so here's a micro-update:

Code: (fxaa-v.sdr) [Select]

#extension GL_EXT_gpu_shader4 : enable
noperspective varying vec2 pos;
uniform float rt_w;
uniform float rt_h;
varying vec2 rcpFrame;

void main() {
gl_Position = gl_Vertex;

rcpFrame = vec2(1.0/rt_w, 1.0/rt_h);

pos = gl_Vertex.xy*0.5 + 0.5;
}

Not much changed there, but note the "#extension" bit.
This was on in the Fragment Shader, but was not in the original Vertex shader, leading to errors and un-linking shaders re 'noperspective'.
This change resolves that.

Code: (fxaa-f.sdr) [Select]

#extension GL_EXT_gpu_shader4 : enable

// Copyright for FXAA Source
//
// Copyright (c) 2010 NVIDIA Corporation. All rights reserved.
//
// TO  THE MAXIMUM  EXTENT PERMITTED  BY APPLICABLE  LAW, THIS SOFTWARE  IS PROVIDED
// *AS IS*  AND NVIDIA AND  ITS SUPPLIERS DISCLAIM  ALL WARRANTIES,  EITHER  EXPRESS
// OR IMPLIED, INCLUDING, BUT NOT LIMITED  TO, IMPLIED WARRANTIES OF MERCHANTABILITY
// AND FITNESS FOR A PARTICULAR PURPOSE.  IN NO EVENT SHALL  NVIDIA OR ITS SUPPLIERS
// BE  LIABLE  FOR  ANY  SPECIAL,  INCIDENTAL,  INDIRECT,  OR  CONSEQUENTIAL DAMAGES
// WHATSOEVER (INCLUDING, WITHOUT LIMITATION,  DAMAGES FOR LOSS OF BUSINESS PROFITS,
// BUSINESS INTERRUPTION, LOSS OF BUSINESS INFORMATION, OR ANY OTHER PECUNIARY LOSS)
// ARISING OUT OF THE  USE OF OR INABILITY  TO USE THIS SOFTWARE, EVEN IF NVIDIA HAS
// BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

#define FXAA_GLSL_120 1

/*============================================================================
                                    FXAA                                 
============================================================================*/
 
/*============================================================================
                                 API PORTING
============================================================================*/
#ifndef     FXAA_GLSL_120
    #define FXAA_GLSL_120 0
#endif
#ifndef     FXAA_GLSL_130
    #define FXAA_GLSL_130 0
#endif

#define int2 ivec2
#define float2 vec2
#define float3 vec3
#define float4 vec4
#define FxaaBool3 bvec3
#define FxaaInt2 ivec2
#define FxaaFloat2 vec2
#define FxaaFloat3 vec3
#define FxaaFloat4 vec4
#define FxaaBool2Float(a) mix(0.0, 1.0, (a))
#define FxaaPow3(x, y) pow(x, y)
#define FxaaSel3(f, t, b) mix((f), (t), (b))
#define FxaaTex sampler2D
/*--------------------------------------------------------------------------*/
#if FXAA_GLSL_120
    // Requires "#version 120" or better
    #define FxaaTexLod0(t, p) texture2DLod(t, p, 0.0)
    #define FxaaTexOff(t, p, o, r) texture2DLodOffset(t, p, 0.0, o)
#endif
/*--------------------------------------------------------------------------*/
#if FXAA_GLSL_130
    // Requires "#version 130" or better
    #define FxaaTexLod0(t, p) textureLod(t, p, 0.0)
    #define FxaaTexOff(t, p, o, r) textureLodOffset(t, p, 0.0, o)
#endif
/*--------------------------------------------------------------------------*/

#define FxaaToFloat3(a) FxaaFloat3((a), (a), (a))
float4 FxaaTexGrad(FxaaTex tex, float2 pos, float2 grad) {
    #if FXAA_GLSL_120
        return texture2DGrad(tex, pos.xy, grad, grad);
    #endif
    #if FXAA_GLSL_130
        return textureGrad(tex, pos.xy, grad, grad);
    #endif
}

/*============================================================================
                                 SRGB KNOBS
------------------------------------------------------------------------------
FXAA_SRGB_ROP - Set to 1 when applying FXAA to an sRGB back buffer (DX10/11).
                This will do the sRGB to linear transform,
                as ROP will expect linear color from this shader,
                and this shader works in non-linear color.
============================================================================*/
#define FXAA_SRGB_ROP 0

/*============================================================================
                                DEBUG KNOBS
------------------------------------------------------------------------------
All debug knobs draw FXAA-untouched pixels in FXAA computed luma (monochrome).
 
FXAA_DEBUG_PASSTHROUGH - Red for pixels which are filtered by FXAA with a
                         yellow tint on sub-pixel aliasing filtered by FXAA.
FXAA_DEBUG_HORZVERT    - Blue for horizontal edges, gold for vertical edges.
FXAA_DEBUG_PAIR        - Blue/green for the 2 pixel pair choice.
FXAA_DEBUG_NEGPOS      - Red/blue for which side of center of span.
FXAA_DEBUG_OFFSET      - Red/blue for -/+ x, gold/skyblue for -/+ y.
============================================================================*/
#ifndef     FXAA_DEBUG_PASSTHROUGH
    #define FXAA_DEBUG_PASSTHROUGH 0
#endif   
#ifndef     FXAA_DEBUG_HORZVERT
    #define FXAA_DEBUG_HORZVERT    0
#endif   
#ifndef     FXAA_DEBUG_PAIR   
    #define FXAA_DEBUG_PAIR        0
#endif   
#ifndef     FXAA_DEBUG_NEGPOS
    #define FXAA_DEBUG_NEGPOS      0
#endif
#ifndef     FXAA_DEBUG_OFFSET
    #define FXAA_DEBUG_OFFSET      0
#endif   
/*--------------------------------------------------------------------------*/
#if FXAA_DEBUG_PASSTHROUGH || FXAA_DEBUG_HORZVERT || FXAA_DEBUG_PAIR
    #define FXAA_DEBUG 1
#endif   
#if FXAA_DEBUG_NEGPOS || FXAA_DEBUG_OFFSET
    #define FXAA_DEBUG 1
#endif
#ifndef FXAA_DEBUG
    #define FXAA_DEBUG 0
#endif
 
/*============================================================================
                              COMPILE-IN KNOBS
------------------------------------------------------------------------------
FXAA_PRESET - Choose compile-in knob preset 0-5.
------------------------------------------------------------------------------
FXAA_EDGE_THRESHOLD - The minimum amount of local contrast required
                      to apply algorithm.
                      1.0/3.0  - too little
                      1.0/4.0  - good start
                      1.0/8.0  - applies to more edges
                      1.0/16.0 - overkill
------------------------------------------------------------------------------
FXAA_EDGE_THRESHOLD_MIN - Trims the algorithm from processing darks.
                          Perf optimization.
                          1.0/32.0 - visible limit (smaller isn't visible)
                          1.0/16.0 - good compromise
                          1.0/12.0 - upper limit (seeing artifacts)
------------------------------------------------------------------------------
FXAA_SEARCH_STEPS - Maximum number of search steps for end of span.
------------------------------------------------------------------------------
FXAA_SEARCH_ACCELERATION - How much to accelerate search,
                           1 - no acceleration
                           2 - skip by 2 pixels
                           3 - skip by 3 pixels
                           4 - skip by 4 pixels
------------------------------------------------------------------------------
FXAA_SEARCH_THRESHOLD - Controls when to stop searching.
                        1.0/4.0 - seems to be the best quality wise
------------------------------------------------------------------------------
FXAA_SUBPIX_FASTER - Turn on lower quality but faster subpix path.
                     Not recomended, but used in preset 0.
------------------------------------------------------------------------------
FXAA_SUBPIX - Toggle subpix filtering.
              0 - turn off
              1 - turn on
              2 - turn on full (ignores FXAA_SUBPIX_TRIM and CAP)
------------------------------------------------------------------------------
FXAA_SUBPIX_TRIM - Controls sub-pixel aliasing removal.
                   1.0/2.0 - low removal
                   1.0/3.0 - medium removal
                   1.0/4.0 - default removal
                   1.0/8.0 - high removal
                   0.0 - complete removal
------------------------------------------------------------------------------
FXAA_SUBPIX_CAP - Insures fine detail is not completely removed.
                  This is important for the transition of sub-pixel detail,
                  like fences and wires.
                  3.0/4.0 - default (medium amount of filtering)
                  7.0/8.0 - high amount of filtering
                  1.0 - no capping of sub-pixel aliasing removal
============================================================================*/
#ifndef FXAA_PRESET
    #define FXAA_PRESET 3
#endif
/*--------------------------------------------------------------------------*/
#if (FXAA_PRESET == 0)
    #define FXAA_EDGE_THRESHOLD      (1.0/4.0)
    #define FXAA_EDGE_THRESHOLD_MIN  (1.0/12.0)
    #define FXAA_SEARCH_STEPS        2
    #define FXAA_SEARCH_ACCELERATION 4
    #define FXAA_SEARCH_THRESHOLD    (1.0/4.0)
    #define FXAA_SUBPIX              1
    #define FXAA_SUBPIX_FASTER       1
    #define FXAA_SUBPIX_CAP          (2.0/3.0)
    #define FXAA_SUBPIX_TRIM         (1.0/4.0)
#endif
/*--------------------------------------------------------------------------*/
#if (FXAA_PRESET == 1)
    #define FXAA_EDGE_THRESHOLD      (1.0/8.0)
    #define FXAA_EDGE_THRESHOLD_MIN  (1.0/16.0)
    #define FXAA_SEARCH_STEPS        4
    #define FXAA_SEARCH_ACCELERATION 3
    #define FXAA_SEARCH_THRESHOLD    (1.0/4.0)
    #define FXAA_SUBPIX              1
    #define FXAA_SUBPIX_FASTER       0
    #define FXAA_SUBPIX_CAP          (3.0/4.0)
    #define FXAA_SUBPIX_TRIM         (1.0/4.0)
#endif
/*--------------------------------------------------------------------------*/
#if (FXAA_PRESET == 2)
    #define FXAA_EDGE_THRESHOLD      (1.0/8.0)
    #define FXAA_EDGE_THRESHOLD_MIN  (1.0/24.0)
    #define FXAA_SEARCH_STEPS        8
    #define FXAA_SEARCH_ACCELERATION 2
    #define FXAA_SEARCH_THRESHOLD    (1.0/4.0)
    #define FXAA_SUBPIX              1
    #define FXAA_SUBPIX_FASTER       0
    #define FXAA_SUBPIX_CAP          (3.0/4.0)
    #define FXAA_SUBPIX_TRIM         (1.0/4.0)
#endif
/*--------------------------------------------------------------------------*/
#if (FXAA_PRESET == 3)
    #define FXAA_EDGE_THRESHOLD      (1.0/8.0)
    #define FXAA_EDGE_THRESHOLD_MIN  (1.0/24.0)
    #define FXAA_SEARCH_STEPS        16
    #define FXAA_SEARCH_ACCELERATION 1
    #define FXAA_SEARCH_THRESHOLD    (1.0/4.0)
    #define FXAA_SUBPIX              1
    #define FXAA_SUBPIX_FASTER       0
    #define FXAA_SUBPIX_CAP          (3.0/4.0)
    #define FXAA_SUBPIX_TRIM         (1.0/4.0)
#endif
/*--------------------------------------------------------------------------*/
#if (FXAA_PRESET == 4)
    #define FXAA_EDGE_THRESHOLD      (1.0/8.0)
    #define FXAA_EDGE_THRESHOLD_MIN  (1.0/24.0)
    #define FXAA_SEARCH_STEPS        24
    #define FXAA_SEARCH_ACCELERATION 1
    #define FXAA_SEARCH_THRESHOLD    (1.0/4.0)
    #define FXAA_SUBPIX              1
    #define FXAA_SUBPIX_FASTER       0
    #define FXAA_SUBPIX_CAP          (3.0/4.0)
    #define FXAA_SUBPIX_TRIM         (1.0/4.0)
#endif
/*--------------------------------------------------------------------------*/
#if (FXAA_PRESET == 5)
    #define FXAA_EDGE_THRESHOLD      (1.0/8.0)
    #define FXAA_EDGE_THRESHOLD_MIN  (1.0/24.0)
    #define FXAA_SEARCH_STEPS        32
    #define FXAA_SEARCH_ACCELERATION 1
    #define FXAA_SEARCH_THRESHOLD    (1.0/4.0)
    #define FXAA_SUBPIX              1
    #define FXAA_SUBPIX_FASTER       0
    #define FXAA_SUBPIX_CAP          (7.0/8.0)
    #define FXAA_SUBPIX_TRIM         (1.0/8.0)
#endif
/*--------------------------------------------------------------------------*/
#if (FXAA_PRESET == 6)
    #define FXAA_EDGE_THRESHOLD      (1.0/12.0)
    #define FXAA_EDGE_THRESHOLD_MIN  (1.0/24.0)
    #define FXAA_SEARCH_STEPS        32
    #define FXAA_SEARCH_ACCELERATION 1
    #define FXAA_SEARCH_THRESHOLD    (1.0/4.0)
    #define FXAA_SUBPIX              1
    #define FXAA_SUBPIX_FASTER       0
    #define FXAA_SUBPIX_CAP          (1.0)
    #define FXAA_SUBPIX_TRIM         (0.0)
#endif
/*--------------------------------------------------------------------------*/
#define FXAA_SUBPIX_TRIM_SCALE (1.0/(1.0 - FXAA_SUBPIX_TRIM))

/*============================================================================
                                   HELPERS
============================================================================*/
// Return the luma, the estimation of luminance from rgb inputs.
// This approximates luma using one FMA instruction,
// skipping normalization and tossing out blue.
// FxaaLuma() will range 0.0 to 2.963210702.
float FxaaLuma(float3 rgb) {
    return rgb.y * (0.587/0.299) + rgb.x; }
/*--------------------------------------------------------------------------*/
float3 FxaaLerp3(float3 a, float3 b, float amountOfA) {
    return (FxaaToFloat3(-amountOfA) * b) +
        ((a * FxaaToFloat3(amountOfA)) + b); }
/*--------------------------------------------------------------------------*/
// Support any extra filtering before returning color.
float3 FxaaFilterReturn(float3 rgb) {
    #if FXAA_SRGB_ROP
        // Do sRGB encoded value to linear conversion.
        return FxaaSel3(
            rgb * FxaaToFloat3(1.0/12.92),
            FxaaPow3(
                rgb * FxaaToFloat3(1.0/1.055) + FxaaToFloat3(0.055/1.055),
                FxaaToFloat3(2.4)),
            rgb > FxaaToFloat3(0.04045));
    #else
        return rgb;
    #endif
}
 
/*============================================================================
                                VERTEX SHADER
============================================================================*/
float2 FxaaVertexShader(
// Both x and y range {-1.0 to 1.0 across screen}.
float2 inPos) {
    float2 pos;
    pos.xy = (inPos.xy * FxaaFloat2(0.5, 0.5)) + FxaaFloat2(0.5, 0.5);
    return pos; } 
 
/*============================================================================
                                PIXEL SHADER
============================================================================*/
float3 FxaaPixelShader(
// Output of FxaaVertexShader interpolated across screen.
//  xy -> actual texture position {0.0 to 1.0}
float2 pos,
// Input texture.
FxaaTex tex,
// RCPFRAME SHOULD PIXEL SHADER CONSTANTS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// {1.0/frameWidth, 1.0/frameHeight}
float2 rcpFrame) {
   
/*----------------------------------------------------------------------------
            EARLY EXIT IF LOCAL CONTRAST BELOW EDGE DETECT LIMIT
------------------------------------------------------------------------------
Majority of pixels of a typical image do not require filtering,
often pixels are grouped into blocks which could benefit from early exit
right at the beginning of the algorithm.
Given the following neighborhood,
 
      N   
    W M E
      S   
   
If the difference in local maximum and minimum luma (contrast "range")
is lower than a threshold proportional to the maximum local luma ("rangeMax"),
then the shader early exits (no visible aliasing).
This threshold is clamped at a minimum value ("FXAA_EDGE_THRESHOLD_MIN")
to avoid processing in really dark areas.   
----------------------------------------------------------------------------*/
    float3 rgbN = FxaaTexOff(tex, pos.xy, FxaaInt2( 0,-1), rcpFrame).xyz;
    float3 rgbW = FxaaTexOff(tex, pos.xy, FxaaInt2(-1, 0), rcpFrame).xyz;
    float3 rgbM = FxaaTexOff(tex, pos.xy, FxaaInt2( 0, 0), rcpFrame).xyz;
    float3 rgbE = FxaaTexOff(tex, pos.xy, FxaaInt2( 1, 0), rcpFrame).xyz;
    float3 rgbS = FxaaTexOff(tex, pos.xy, FxaaInt2( 0, 1), rcpFrame).xyz;
    float lumaN = FxaaLuma(rgbN);
    float lumaW = FxaaLuma(rgbW);
    float lumaM = FxaaLuma(rgbM);
    float lumaE = FxaaLuma(rgbE);
    float lumaS = FxaaLuma(rgbS);
    float rangeMin = min(lumaM, min(min(lumaN, lumaW), min(lumaS, lumaE)));
    float rangeMax = max(lumaM, max(max(lumaN, lumaW), max(lumaS, lumaE)));
    float range = rangeMax - rangeMin;
    #if FXAA_DEBUG
        float lumaO = lumaM / (1.0 + (0.587/0.299));
    #endif       
    if(range < max(FXAA_EDGE_THRESHOLD_MIN, rangeMax * FXAA_EDGE_THRESHOLD)) {
        #if FXAA_DEBUG
            return FxaaFilterReturn(FxaaToFloat3(lumaO));
        #endif
        return FxaaFilterReturn(rgbM); }
    #if FXAA_SUBPIX > 0
        #if FXAA_SUBPIX_FASTER
            float3 rgbL = (rgbN + rgbW + rgbE + rgbS + rgbM) *
                FxaaToFloat3(1.0/5.0);
        #else
            float3 rgbL = rgbN + rgbW + rgbM + rgbE + rgbS;
        #endif
    #endif       
   
/*----------------------------------------------------------------------------
                               COMPUTE LOWPASS
------------------------------------------------------------------------------
FXAA computes a local neighborhood lowpass value as follows,
 
  (N + W + E + S)/4
 
Then uses the ratio of the contrast range of the lowpass
and the range found in the early exit check,
as a sub-pixel aliasing detection filter.
When FXAA detects sub-pixel aliasing (such as single pixel dots),
it later blends in "blendL" amount
of a lowpass value (computed in the next section) to the final result.
----------------------------------------------------------------------------*/
    #if FXAA_SUBPIX != 0
        float lumaL = (lumaN + lumaW + lumaE + lumaS) * 0.25;
        float rangeL = abs(lumaL - lumaM);
    #endif       
    #if FXAA_SUBPIX == 1
        float blendL = max(0.0,
            (rangeL / range) - FXAA_SUBPIX_TRIM) * FXAA_SUBPIX_TRIM_SCALE;
        blendL = min(FXAA_SUBPIX_CAP, blendL);
    #endif
    #if FXAA_SUBPIX == 2
        float blendL = rangeL / range;
    #endif
    #if FXAA_DEBUG_PASSTHROUGH
        #if FXAA_SUBPIX == 0
            float blendL = 0.0;
        #endif
        return FxaaFilterReturn(
            FxaaFloat3(1.0, blendL/FXAA_SUBPIX_CAP, 0.0));
    #endif   
   
/*----------------------------------------------------------------------------
                    CHOOSE VERTICAL OR HORIZONTAL SEARCH
------------------------------------------------------------------------------
FXAA uses the following local neighborhood,
 
    NW N NE
    W  M  E
    SW S SE
   
To compute an edge amount for both vertical and horizontal directions.
Note edge detect filters like Sobel fail on single pixel lines through M.
FXAA takes the weighted average magnitude of the high-pass values
for rows and columns as an indication of local edge amount.
 
A lowpass value for anti-sub-pixel-aliasing is computed as
    (N+W+E+S+M+NW+NE+SW+SE)/9.
This full box pattern has higher quality than other options.
 
Note following this block, both vertical and horizontal cases
flow in parallel (reusing the horizontal variables).
----------------------------------------------------------------------------*/
    float3 rgbNW = FxaaTexOff(tex, pos.xy, FxaaInt2(-1,-1), rcpFrame).xyz;
    float3 rgbNE = FxaaTexOff(tex, pos.xy, FxaaInt2( 1,-1), rcpFrame).xyz;
    float3 rgbSW = FxaaTexOff(tex, pos.xy, FxaaInt2(-1, 1), rcpFrame).xyz;
    float3 rgbSE = FxaaTexOff(tex, pos.xy, FxaaInt2( 1, 1), rcpFrame).xyz;
    #if (FXAA_SUBPIX_FASTER == 0) && (FXAA_SUBPIX > 0)
        rgbL += (rgbNW + rgbNE + rgbSW + rgbSE);
        rgbL *= FxaaToFloat3(1.0/9.0);
    #endif
    float lumaNW = FxaaLuma(rgbNW);
    float lumaNE = FxaaLuma(rgbNE);
    float lumaSW = FxaaLuma(rgbSW);
    float lumaSE = FxaaLuma(rgbSE);
    float edgeVert =
        abs((0.25 * lumaNW) + (-0.5 * lumaN) + (0.25 * lumaNE)) +
        abs((0.50 * lumaW ) + (-1.0 * lumaM) + (0.50 * lumaE )) +
        abs((0.25 * lumaSW) + (-0.5 * lumaS) + (0.25 * lumaSE));
    float edgeHorz =
        abs((0.25 * lumaNW) + (-0.5 * lumaW) + (0.25 * lumaSW)) +
        abs((0.50 * lumaN ) + (-1.0 * lumaM) + (0.50 * lumaS )) +
        abs((0.25 * lumaNE) + (-0.5 * lumaE) + (0.25 * lumaSE));
    bool horzSpan = edgeHorz >= edgeVert;
    #if FXAA_DEBUG_HORZVERT
        if(horzSpan) return FxaaFilterReturn(FxaaFloat3(1.0, 0.75, 0.0));
        else         return FxaaFilterReturn(FxaaFloat3(0.0, 0.50, 1.0));
    #endif
    float lengthSign = horzSpan ? -rcpFrame.y : -rcpFrame.x;
    if(!horzSpan) lumaN = lumaW;
    if(!horzSpan) lumaS = lumaE;
    float gradientN = abs(lumaN - lumaM);
    float gradientS = abs(lumaS - lumaM);
    lumaN = (lumaN + lumaM) * 0.5;
    lumaS = (lumaS + lumaM) * 0.5;
   
/*----------------------------------------------------------------------------
                CHOOSE SIDE OF PIXEL WHERE GRADIENT IS HIGHEST
------------------------------------------------------------------------------
This chooses a pixel pair.
For "horzSpan == true" this will be a vertical pair,
 
    [N]     N
    [M] or [M]
     S     [S]
 
Note following this block, both {N,M} and {S,M} cases
flow in parallel (reusing the {N,M} variables).
 
This pair of image rows or columns is searched below
in the positive and negative direction
until edge status changes
(or the maximum number of search steps is reached).
----------------------------------------------------------------------------*/   
    bool pairN = gradientN >= gradientS;
    #if FXAA_DEBUG_PAIR
        if(pairN) return FxaaFilterReturn(FxaaFloat3(0.0, 0.0, 1.0));
        else      return FxaaFilterReturn(FxaaFloat3(0.0, 1.0, 0.0));
    #endif
    if(!pairN) lumaN = lumaS;
    if(!pairN) gradientN = gradientS;
    if(!pairN) lengthSign *= -1.0;
    float2 posN;
    posN.x = pos.x + (horzSpan ? 0.0 : lengthSign * 0.5);
    posN.y = pos.y + (horzSpan ? lengthSign * 0.5 : 0.0);
   
/*----------------------------------------------------------------------------
                         CHOOSE SEARCH LIMITING VALUES
------------------------------------------------------------------------------
Search limit (+/- gradientN) is a function of local gradient.
----------------------------------------------------------------------------*/
    gradientN *= FXAA_SEARCH_THRESHOLD;
   
/*----------------------------------------------------------------------------
    SEARCH IN BOTH DIRECTIONS UNTIL FIND LUMA PAIR AVERAGE IS OUT OF RANGE
------------------------------------------------------------------------------
This loop searches either in vertical or horizontal directions,
and in both the negative and positive direction in parallel.
This loop fusion is faster than searching separately.
 
The search is accelerated using FXAA_SEARCH_ACCELERATION length box filter
via anisotropic filtering with specified texture gradients.
----------------------------------------------------------------------------*/
    float2 posP = posN;
    float2 offNP = horzSpan ?
        FxaaFloat2(rcpFrame.x, 0.0) :
        FxaaFloat2(0.0, rcpFrame.y);
    float lumaEndN = lumaN;
    float lumaEndP = lumaN;
    bool doneN = false;
    bool doneP = false;
    #if FXAA_SEARCH_ACCELERATION == 1
        posN += offNP * FxaaFloat2(-1.0, -1.0);
        posP += offNP * FxaaFloat2( 1.0,  1.0);
    #endif
    #if FXAA_SEARCH_ACCELERATION == 2
        posN += offNP * FxaaFloat2(-1.5, -1.5);
        posP += offNP * FxaaFloat2( 1.5,  1.5);
        offNP *= FxaaFloat2(2.0, 2.0);
    #endif
    #if FXAA_SEARCH_ACCELERATION == 3
        posN += offNP * FxaaFloat2(-2.0, -2.0);
        posP += offNP * FxaaFloat2( 2.0,  2.0);
        offNP *= FxaaFloat2(3.0, 3.0);
    #endif
    #if FXAA_SEARCH_ACCELERATION == 4
        posN += offNP * FxaaFloat2(-2.5, -2.5);
        posP += offNP * FxaaFloat2( 2.5,  2.5);
        offNP *= FxaaFloat2(4.0, 4.0);
    #endif
    for(int i = 0; i < FXAA_SEARCH_STEPS; i++) {
        #if FXAA_SEARCH_ACCELERATION == 1
            if(!doneN) lumaEndN =
                FxaaLuma(FxaaTexLod0(tex, posN.xy).xyz);
            if(!doneP) lumaEndP =
                FxaaLuma(FxaaTexLod0(tex, posP.xy).xyz);
        #else
            if(!doneN) lumaEndN =
                FxaaLuma(FxaaTexGrad(tex, posN.xy, offNP).xyz);
            if(!doneP) lumaEndP =
                FxaaLuma(FxaaTexGrad(tex, posP.xy, offNP).xyz);
        #endif
        doneN = doneN || (abs(lumaEndN - lumaN) >= gradientN);
        doneP = doneP || (abs(lumaEndP - lumaN) >= gradientN);
        if(doneN && doneP) break;
        if(!doneN) posN -= offNP;
        if(!doneP) posP += offNP; }
   
/*----------------------------------------------------------------------------
               HANDLE IF CENTER IS ON POSITIVE OR NEGATIVE SIDE
------------------------------------------------------------------------------
FXAA uses the pixel's position in the span
in combination with the values (lumaEnd*) at the ends of the span,
to determine filtering.
 
This step computes which side of the span the pixel is on.
On negative side if dstN < dstP,
 
     posN        pos                      posP
      |-----------|------|------------------|
      |           |      |                  |
      |<--dstN--->|<---------dstP---------->|
                         |
                    span center
                   
----------------------------------------------------------------------------*/
    float dstN = horzSpan ? pos.x - posN.x : pos.y - posN.y;
    float dstP = horzSpan ? posP.x - pos.x : posP.y - pos.y;
    bool directionN = dstN < dstP;
    #if FXAA_DEBUG_NEGPOS
        if(directionN) return FxaaFilterReturn(FxaaFloat3(1.0, 0.0, 0.0));
        else           return FxaaFilterReturn(FxaaFloat3(0.0, 0.0, 1.0));
    #endif
    lumaEndN = directionN ? lumaEndN : lumaEndP;
   
/*----------------------------------------------------------------------------
         CHECK IF PIXEL IS IN SECTION OF SPAN WHICH GETS NO FILTERING
------------------------------------------------------------------------------
If both the pair luma at the end of the span (lumaEndN)
and middle pixel luma (lumaM)
are on the same side of the middle pair average luma (lumaN),
then don't filter.
 
Cases,
 
(1.) "L",
 
               lumaM
                 |
                 V    XXXXXXXX <- other line averaged
         XXXXXXX[X]XXXXXXXXXXX <- source pixel line
        |      .      |
    --------------------------                   
       [ ]xxxxxx[x]xx[X]XXXXXX <- pair average
    --------------------------           
        ^      ^ ^    ^
        |      | |    |
        .      |<---->|<---------- no filter region
        .      | |    |
        . center |    |
        .        |  lumaEndN
        .        |    .
        .      lumaN  .
        .             .
        |<--- span -->|
       
                       
(2.) "^" and "-",
 
                               <- other line averaged
          XXXXX[X]XXX          <- source pixel line
         |     |     |
    --------------------------                   
        [ ]xxxx[x]xx[ ]        <- pair average
    --------------------------           
         |     |     |
         |<--->|<--->|<---------- filter both sides
 
 
(3.) "v" and inverse of "-",
 
    XXXXXX           XXXXXXXXX <- other line averaged
    XXXXXXXXXXX[X]XXXXXXXXXXXX <- source pixel line
         |     |     |
    --------------------------                   
    XXXX[X]xxxx[x]xx[X]XXXXXXX <- pair average
    --------------------------           
         |     |     |
         |<--->|<--->|<---------- don't filter both!
 
         
Note the "v" case for FXAA requires no filtering.
This is because the inverse of the "-" case is the "v".
Filtering "v" case turns open spans like this,
 
    XXXXXXXXX
   
Into this (which is not desired),
 
    x+.   .+x
    XXXXXXXXX
 
----------------------------------------------------------------------------*/
    if(((lumaM - lumaN) < 0.0) == ((lumaEndN - lumaN) < 0.0))
        lengthSign = 0.0;
 
/*----------------------------------------------------------------------------
                COMPUTE SUB-PIXEL OFFSET AND FILTER SPAN
------------------------------------------------------------------------------
FXAA filters using a bilinear texture fetch offset
from the middle pixel M towards the center of the pair (NM below).
Maximum filtering will be half way between pair.
Reminder, at this point in the code,
the {N,M} pair is also reused for all cases: {S,M}, {W,M}, and {E,M}.
 
    +-------+
    |       |    0.5 offset
    |   N   |     |
    |       |     V
    +-------+....---
    |       |
    |   M...|....---
    |       |     ^
    +-------+     |
    .       .    0.0 offset
    .   S   .
    .       .
    .........
 
Position on span is used to compute sub-pixel filter offset using simple ramp,
 
             posN           posP
              |\             |<------- 0.5 pixel offset into pair pixel
              | \            |
              |  \           |
    ---.......|...\..........|<------- 0.25 pixel offset into pair pixel
     ^        |   ^\         |
     |        |   | \        |
     V        |   |  \       |
    ---.......|===|==========|<------- 0.0 pixel offset (ie M pixel)
     ^        .   |   ^      .
     |        .  pos  |      .
     |        .   .   |      .
     |        .   . center   .
     |        .   .          .
     |        |<->|<---------.-------- dstN
     |        .   .          .   
     |        .   |<-------->|<------- dstP   
     |        .             .
     |        |<------------>|<------- spanLength   
     |
    subPixelOffset
   
----------------------------------------------------------------------------*/
    float spanLength = (dstP + dstN);
    dstN = directionN ? dstN : dstP;
    float subPixelOffset = (0.5 + (dstN * (-1.0/spanLength))) * lengthSign;
    #if FXAA_DEBUG_OFFSET
        float ox = horzSpan ? 0.0 : subPixelOffset*2.0/rcpFrame.x;
        float oy = horzSpan ? subPixelOffset*2.0/rcpFrame.y : 0.0;
        if(ox < 0.0) return FxaaFilterReturn(
            FxaaLerp3(FxaaToFloat3(lumaO),
                      FxaaFloat3(1.0, 0.0, 0.0), -ox));
        if(ox > 0.0) return FxaaFilterReturn(
            FxaaLerp3(FxaaToFloat3(lumaO),
                      FxaaFloat3(0.0, 0.0, 1.0),  ox));
        if(oy < 0.0) return FxaaFilterReturn(
            FxaaLerp3(FxaaToFloat3(lumaO),
                      FxaaFloat3(1.0, 0.6, 0.2), -oy));
        if(oy > 0.0) return FxaaFilterReturn(
            FxaaLerp3(FxaaToFloat3(lumaO),
                      FxaaFloat3(0.2, 0.6, 1.0),  oy));
        return FxaaFilterReturn(FxaaFloat3(lumaO, lumaO, lumaO));
    #endif
    float3 rgbF = FxaaTexLod0(tex, FxaaFloat2(
        pos.x + (horzSpan ? 0.0 : subPixelOffset),
        pos.y + (horzSpan ? subPixelOffset : 0.0))).xyz;
    #if FXAA_SUBPIX == 0
        return FxaaFilterReturn(rgbF);
    #else       
        return FxaaFilterReturn(FxaaLerp3(rgbL, rgbF, blendL));
    #endif
}

uniform sampler2D tex0;
varying vec2 rcpFrame;
noperspective varying vec2 pos;

void main() {
gl_FragColor.xyz = FxaaPixelShader(pos, tex0, rcpFrame);
}


A refined version from earlier. Changes include:
     Removing all the HLSL stuff (as we don't use it)
     Adding a "FXAA_PRESET == 6" for those of you daring enough to use it
     A slight tweak to FXAA_PRESET 5.

[The E]

First post has been updated. You may wish to redownload the builds and shaders.

[Zacam]

Do note, the shaders I last linked work for the first builds E posted.

They may even work in the new builds. But the first post has updated shaders for use with the newer builds which allows you to "preview" the FXAA Presets in the LAB and then set a preset option in the commandline. It's highly recommended that you use those. Mine will remain posted for no other reason than I'm lazy.

[Kolgena]

btw, thought I should mention that FXAA is especially awesome for all the effects in the game. Beams look sooo much nicer when not aliased.

[DaBrain]

It does reduce the image sharpness a bit, but I can live with that. I kills aliasing pretty well and even helps with shader aliasing and texture shimmering.
I hardly noticed any performance drop and I'm only using an older Geforce 260.


The only thing I mind is the option in the launcher for AA.
Either take it out, or update it. It never worked and now that might even lead people to the assumption AA is not possible in FSO at all.


Anyway, great work!

[Luis Dias]

clap clap clap!

[Sobbsy]

This appears to work really well. I tested it out in a few missions and it has pretty much no effect on the framerates, to my surprise.

Out of curiosity, is there a way to stop the 'Hide post-processing' flag from being set in the F3 lab? Every time I fire up the game I have to go into that and untick that option to get FXAA and Post-Processing to work together. I have PP enabled in the launcher along with FXAA, so I don't see why it's doing that.

Looks fantastic though... Finally some working AA!

[Zacam]

Unless you are missing "-bloom_intensity ###" and "-fxaa_preset #" in your command line, you don't need to go to the Lab to set values for gameplay use.

And if you don't have the command line settings, then default values will get used. Which if you are changing those in the Lab, will carry through to game-play.

But you do not need to go into the Lab to turn on Post-Processing that is already turned on in the launcher in order to have it work.

The "Hide Post Processing" tick box is not an on/off switch for the game, it's just a visual aid element. Most people still use the Lab when modding/modeling for the game, so it defaults to not showing the post processing effects. But again, it is not an on/off switch for the game itself.

[Sobbsy]

Interesting, I'll have to double check when I get home again later. I do have both of those commands entered properly into my command line. Initially, I enabled FXAA and Post-Processing, made sure the command line was correct, then just fired up FS2 and proceeded to test a mission (The Great Hunt - it offers some good contrast against the nebula background to test with) and noticed that there was absolutely no sign of any post-processing or AA. So I went back to the main menu, opened the lab and unticked that 'Hide post-processing' option (which made it work just fine in the lab there) and fired up the same mission... Seemed to fix the problem and it was clearly working.

I'll check it again later but on at least one other occasion after that I had to do the same thing to get it working in-game - fire up the lab, uncheck 'Hide post-processing'. Is there absolutely in no way, shape or form any kind of link between the lab and the actual game? Maybe I've just got some obscure little problem and the controls in the lab are fiddling with the actual application ingame somehow... I will have to take another look tonight just to be absolutely certain that I'm not having a brainfart!

Thanks again.

[Ace]

I've been having some issues with FXAA when combined with render to texture, resulting in crashes. I think this probably has more to do with the RTT code than post processing, but here's a log:
http://www.hard-light.net/forums/index.php?action=dlattach;topic=76339.0;attach=16468

[The E]

Ummm... I can't open that link. Could you either attach the log to your post here, in this thread, or paste it into your post (surrounded by [code] tags, of course)?

[The E]

I'll check it again later but on at least one other occasion after that I had to do the same thing to get it working in-game - fire up the lab, uncheck 'Hide post-processing'. Is there absolutely in no way, shape or form any kind of link between the lab and the actual game? Maybe I've just got some obscure little problem and the controls in the lab are fiddling with the actual application ingame somehow... I will have to take another look tonight just to be absolutely certain that I'm not having a brainfart!

Thanks again.

The "Hide post-processing" checkbox is a lab-only override. Technically, the lab uses the same render pipeline as the regular game, including post-processing. If the "Hide post-processing" box is marked, the lab (and only the lab!) will skip the PP render stage. It doesn't have any effect beyond the lab.

That being said, it would be helplful if I could get some more info. What builds are you using? Which version of the FXAA shaders? And, above all else, can you post an fs2_open.log?

[Macfie]

Do the FXAA shaders replace the the main-f and main-v shaders or are they used in addition to them?

[The E]

They are used in addition to the rest of the shaders (Note: Recent nightlies have the shaders integrated into the exe, so you don't need the files posted here to use FXAA).

FXAA is a separate rendering step, inserted into the pipeline before the rest of the post-processing effects.

[Macfie]

They are used in addition to the rest of the shaders (Note: Recent nightlies have the shaders integrated into the exe, so you don't need the files posted here to use FXAA).

FXAA is a separate rendering step, inserted into the pipeline before the rest of the post-processing effects.

Does that mean I don't need the FXAA-f and FXAA-V shader files in the effects Folder?
Is the same true for the main-f and main-v shader files?

[Angelus]

Yeah, shader files in the folder will override the built-in ones.

[The E]

Does that mean I don't need the FXAA-f and FXAA-V shader files in the effects Folder?
Is the same true for the main-f and main-v shader files?

Yes and yes.

[Kolgena]

Quick suggestion: Make the preset 0 instead use the fast and dirty console version. Ver2 shader is like twice as fast, and at least to me, looks better than preset 0.

[Zacam]

Wait, what? That entirely made no sense what so ever.

If the ver2 is twice as fas and looks better, then why would we make preset 0 of the "console" version the default?

The presets are arranged in terms of staging the available options so that a user can scale the option based on performance feedback. Not everybody is going to be able to rock out with preset 6, for example.