Maybe you’ve seen my post Lua as an Effect Format. To recap, you’d like to avoid creating a separate file for every friggin’ shader in your OpenGL program, especially with the advent of purely shader-based OpenGL and the new programmable stages in OpenGL 4.0. You’d also like to avoid big frameworks that attempt to accommodate other API’s like DirectX.

So, you need a lightweight solution that can organize your shader strings, load them up in a convenient manner, and automatically prepend #line directives. Not all developers like to use Lua. To that end, I wrote an “OpenGL Shader Wrangler” (GLSW), a tiny library that does the following:

• Chops up simple “effect” files into lists of strings
• Automatically prepends shaders with #line to enable proper error reporting
• Optionally prepends shader directives like #version and #extension
• Associates each shader with a quasi-hierarchical path called a shader key

The OpenGL Shader Wrangler is covered by the MIT license. It’s written in ANSI C and consists of four source files:

• glsw.h
• glsw.c
• bstrlib.h
• bstrlib.c

Those last two files are from Paul Hsieh’s Better String Library, which is a nice substitute for the classic (and unsafe) string functions like strcat and sprintf.

GLSW has no dependencies on OpenGL; it’s just a string manager. The API consists of only six functions:

int glswInit();
int glswShutdown();
int glswSetPath(const char* pathPrefix, const char* pathSuffix);
const char* glswGetShader(const char* effectKey);
const char* glswGetError();
int glswAddDirectiveToken(const char* token, const char* directive);

In every function, a non-zero return value indicates success. I’ll show you some examples shortly, but first let’s establish a vocabulary, some of which is borrowed from my Lua post:

effect

Simple text file that contains a bundle of shaders in any combination. For example, an effect might have 3 vertex shaders and 1 fragment shader. It might have 0 vertex shaders, or it might contain only tessellation shaders. Effects should not be confused with OpenGL program objects. A program object is a set of shaders that are linked together; an effect is just a grouping of various shader strings.

shader key

Identifier for a shader consisting of alphanumeric characters and periods. In a way, the periods serve as path separators, and the shader key is a path into a simple hierarchy of your own design. However, GLSW does not maintain a hierarchy; it stores shaders in a flat list.

effect key

Same as a shader key, but prefaced with a token for the effect. Your C/C++ code uses an effect key when requesting a certain shader string.

token

Contiguous sequence of alphanumeric characters in a shader key (or effect key) delimited by periods.

section divider

Any line in an effect file that starts with two dash characters (--). If the dashes are followed by a shader key, then all the text until the next section divider officially belongs to the corresponding shader.

directive

Any line of GLSL code that starts with a pound (#) symbol.

Here’s an example effect key with four tokens:

In the above example, I placed the shader stage in the second token and the API version in the third token. You aren’t required to arrange your shader keys in this way. The only requirement is that the first token corresponds to the effect file. You don’t need to use the GL3 notation for your version tokens either; in fact, you don’t have to include a version token at all. The onus is on you to create a consistent hierarchy and naming convention.

Simple Example

Here’s a text file called Blit.glsl:

-- Vertex

in vec4 Position;
void main() {
    gl_Position = Position;
}

-- Fragment

uniform sampler2D Sampler;
out vec4 FragColor;
void main() {
    ivec2 coord = ivec2(gl_FragCoord.xy);
    FragColor = texelFetch(Sampler, coord, 0);
}

Here’s how you’d use GLSW to pluck out strings from the preceding file and automatically prepend them with #line:

glswInit();
glswSetPath("./", ".glsl");
const char* vs = glswGetShader("Blit.Vertex");
const char* fs = glswGetShader("Blit.Fragment");
// ... use vs and gs here ...
glswShutdown();

GLSW does all the file I/O for you; it takes the first token from the effect key that you pass to glswGetShader, then checks if it has the effect cached; if not, it finds a file by decorating the effect name with a path prefix (in this case, “./”) and a path suffix (in this case, “.glsl”).

Ignored Text

If a section divider in your effect file does not contain any alphanumeric characters, then all text is ignored until the next section divider (or the end of the file). Also, any text that precedes the first section divider is ignored (this could be useful for a copyright notice).

If a section divider does declare a legal shader key (i.e., the first contiguous sequence of alphanumeric characters and periods), then any characters appearing before and after the shader key are ignored.

For example, the following is a silly but valid effect file:

______  _  _  _   
| ___ \| |(_)| |  
| |_/ /| | _ | |_ 
| ___ \| || || __|
| |_/ /| || || |_ 
\____/ |_||_| \__|
                  
-------- Vertex --------

in vec4 Position;
void main() {
    gl_Position = Position;
}

--- /\/\/\/\/\/\/\/\ ---

Contrariwise, if it was so, it might be; and if it were so,
it would be; but as it isn't, it ain't. That's logic. 

--[[[[[ Fragment <== Brigadier Lethbridge-Stewart

uniform sampler2D Sampler;
out vec4 FragColor;
void main() {
    ivec2 coord = ivec2(gl_FragCoord.xy);
    FragColor = texelFetch(Sampler, coord, 0);
}

Error Handling and Shader Key Matching

GLSW never aborts or throws exceptions. It returns 0 on failure and lets you fetch the most recent error string using glswGetError. This can happen if it can’t find the file for your specified effect key, or if you forget to call glswInit.

When GLSW tries to find a shader key that matches with the effect key requested from your C/C++ code, it doesn’t necessarily need to find an exact match. Instead, it finds the longest shader key that matches with the beginning of your requested effect key. For example, consider an effect file called TimeMachine.glsl that looks like this:

-- Vertex
FOO
-- Geometry
BAR
-- Fragment.Erosion
BAZ
-- Fragment.Grassfire
QUX
-- TessControl
QUUX
-- TessEvaluation
QUUUX

If your C++ code does this:

void test(const char* effectKey)
{
    const char* shader = glswGetShader(effectKey);
    if (shader)
        cout << shader << endl;
    else
        cout << glswGetError() << endl;
}

void main()
{
    glswInit();
    glswSetPath("", ".glsl");
    test("TimeMachine.Vertex.Grassfire");
    test("TimeMachine.Fragment.Grassfire");
    test("TimeMachine.Fragment.Cilantro");
    test("Madrid");
    glswShutdown();
}

Then your output would be this:

FOO
QUX
Could not find shader with key 'TimeMachine.Fragment.Cilantro'.
Unable to open effect file 'Madrid.glsl'.

Note that the first requested effect key did not report an error even though that exact shader key does not exist in the effect file. Here’s an example where this is useful: you’d like to use the same vertex shader with two different fragment shaders, but your rendering engine forms the string for effect keys in a generic way. For example, maybe your rendering engine defines a function that looks like this:

GLuint BuildProgram(const char* effectVariant)
{
    // ...snip...

    sprintf(effectKey, "TimeMachine.Vertex.%s", effectVariant);
    const char* vs = glswGetShader(effectKey);

    sprintf(effectKey, "TimeMachine.Fragment.%s", effectVariant);
    const char* fs = glswGetShader(effectKey);

    // ...snip...
}

If you want to use the grassfire variant of the TimeMachine effect, you’d call BuildProgram(“Grassfire”). Even though the vertex shader happens to be the same for all variants of the TimeMachine effect, the rendering engine doesn’t need to know. GLSW will simply return the longest vertex shader that matches with the beginning of “TimeMachine.Vertex.Grassfire”. This avoids complicating the file format with support for cross-referencing.

Directive Mapping

This is the only function we haven’t explained yet:

int glswAddDirectiveToken(const char* token, const char* directive);

This tells GLSW to add a token-to-directive mapping. Whenever it sees a shader key that contains the specified token, it prepends the specified directive to the top of the shader. If the specified token is an empty string, then the specified directive is prepended to all shaders. For example, given this C/C++ code:

glswInit();
glswSetPath("./", ".glsl");
glswAddDirectiveToken("", "#define FOO 1");
glswAddDirectiveToken("GL32", "#version 140");
glswAddDirectiveToken("iPhone", "#extension GL_OES_standard_derivatives : enable");

string effectKey;

effectKey = "Blit.ES2.iPhone.Fragment";
cout << effectKey << ':' << endl
     << glswGetShader(effectKey.c_str()) << endl << endl;

effectKey = "Blit.GL32.Fragment";
cout << effectKey ':' << endl
     << glswGetShader(effectKey.c_str()) << endl << endl;

glswShutdown();

You’d get output like this:

Blit.ES2.iPhone.Fragment:
#define FOO 1
#extension GL_OES_standard_derivatives : enable
#line 7
void main() {}

Blit.GL32.Fragment:
#define FOO 1
#version 140
#line 51
void main() {}

You can also use an effect name for the token parameter in glswAddDirectiveToken, which tells GLSW to prepend the specified directive to all shaders in that effect.

C’est tout!

Download

You can download GLSW as a zip file:

• glsw.zip
• glsw.tar.gz

Or, you can download the files separately:

• glsw.h
• glsw.c
• bstrlib.h
• bstrlib.c

If you’re an OpenGL developer, you might find yourself pining for an effect file format. You’d like a standard way of specifying shader strings, but without creating a kazillion little files for all your shaders.

There aren’t many alternatives. You’re faced with such tantalizing possibilities as:

• Learn COLLADA and get mired in a Turing tarpit of entangled cross-referencing and XML namespacing.
• Travel back in time to stop Khronos from abandoning their glFX effort.
• Find someone’s made-from-scratch solution, then discover that it’s hopelessly outdated.

In this post, we’ll explore the idea of using Lua as an effect file format. We can live without some of the features that we’d expect in a more full-blown FX format (for example, having the ability to associate an effect with a specific blend or cull state). In a way, we’re simply using Lua as an organization tool for shaders.

Let’s try designating each Lua file as a single “effect”, which we’ll loosely define as “a bundle of shaders”. Usually an effect declares at least one string for each programmable stage in the OpenGL pipeline. Later in the article, we’ll group together shaders for various generations of OpenGL; this is convenient for applications that detect the platform’s capabilities at run-time.

Here’s an obvious approach to specifying an effect with Lua (note how nicely Lua handles multi-line strings):

-- Vertex Shader
MyVertexShader = [[
in vec4 Position;
uniform mat4 Projection;
void main()
{
    gl_Position = Projection * Position;
}]]

-- Fragment Shader
MyFragmentShader = [[
out vec4 FragColor;
uniform vec4 FillColor;
void main()
{
    FragColor = FillColor;
}]]

Then, in your C/C++ code, you can do something like this:

lua_State* L = lua_open();
luaL_dofile(L, "MyEffect.lua");
lua_getglobal(L, "MyVertexShader");
lua_getglobal(L, "MyFragmentShader");
const char* vs_text = lua_tostring(L, -2);
const char* fs_text = lua_tostring(L, -1);
lua_pop(L, 2);
glShaderSource(vs_handle, 1, &vs_text, 0);
glShaderSource(fs_handle, 1, &fs_text, 0);

Obviously you’d want to hide code like this behind a utility function or class method. Also, you should always check the return value from luaL_dofile. The code in this article is for illustrative purposes only.

Fixing up the Line Numbers

The problem with the above approach is that any error reporting from the shader compiler will have incorrect line numbers. In the preceding example, if the shader compiler reports an issue with line 1 of the fragment shader, then the relevant line is actually line 12 of the Lua file.

We can fix this using the #line directive in GLSL and the debug.getinfo function in Lua. Instead of declaring strings directly, we’ll need the Lua script to call a function. We can define this function in a separate file called ShaderFactory.lua:

-- Shader Factory
function DeclareShader(name, source)
    local lineNumber = debug.getinfo(2).currentline
    preamble = "#line " .. lineNumber .. "\n"
    _G[name] = preamble .. source
end

Cool usage of the _G table eh? It’s Lua’s built-in table for globals. Now our effect file becomes:

-- Vertex Shader
DeclareShader('MyVertexShader', [[
in vec4 Position;
uniform mat4 Projection;
void main()
{
    gl_Position = Projection * Position;
}]])

-- Fragment Shader
DeclareShader('MyFragmentShader', [[
out vec4 FragColor;
uniform vec4 FillColor;
void main()
{
    FragColor = FillColor;
}]])

On the C/C++ side of things, we can load in the strings just as we did before, except that our script’s usage of debug.getinfo means that we should load in Lua’s debugging library before anything else. I usually load all the utility libraries in one fell swoop using luaL_openlibs. So, our C/C++ code now looks like this:

// Create the Lua context:
lua_State* L = lua_open();

// Open Lua's utility libraries, including the debug library:
luaL_openlibs(L);

// Run the script that defines the DeclareShader function:
luaL_dofile(L, "ShaderFactory.lua");

// Load in the effect files:
luaL_dofile(L, "BrilligEffect.lua");
luaL_dofile(L, "SlithyEffect.lua");
luaL_dofile(L, "ToveEffect.lua");

// As before, extract strings and use them:
lua_getglobal(L, "MyVertexShader");
lua_getglobal(L, "MyFragmentShader");
...

Accommodating Multiple Versions of GLSL

#line isn’t the only directive we’re interested in. One of the great annoyances of OpenGL 3.0+ is the #version directive, which is required if you’d like to use the latest and greatest GLSL syntax. Ideally our DeclareShader function would somehow know if a given shader is from the OpenGL 2.0 era or the OpenGL 3.0+ era, and prepend the string accordingly. One idea is passing in the version number as an argument to the DeclareShader function, like this:

-- Shader Factory
function DeclareShader(name, version, source)
    local lineNumber = debug.getinfo(2).currentline
    preamble = "#line " .. lineNumber .. "\n" ..
               "#version " .. version .. "\n"
    _G[name] = preamble .. source
end

Although the above example is a simple solution, it’s not always good enough. Consider a situation where you need to declare multiple versions of the same shader:

-- Vertex Shader for OpenGL 3.0
DeclareShader('MyVertexShader', 130, [[
in vec4 Position;
uniform mat4 Projection;
void main()
{
    gl_Position = Projection * Position;
}]])

-- Vertex Shader for OpenGL 2.0
DeclareShader('MyVertexShader', 120, [[
void main()
{
    gl_Position = ftransform();
}]])

Unfortunately, the second call to DeclareShader will overwrite the first call. Also note that the version of the shading language itself isn’t the same as the OpenGL API. The shading language for OpenGL 2.0+ is version 120, and the shading language for 3.0+ is 130.

Ok, so we need to scope the shader names to prevent naming collisions, plus it might be nice to have DeclareShader automatically infer the language version from the API version number. Lua’s tables are great for organizing strings. The ShaderFactory.lua file now becomes:

VertexShaders = { GL2 = {}, GL3 = {}, GL4 = {}, ES2 = {} }
GeometryShaders = { GL3 = {}, GL4 = {} }
FragmentShaders = { GL2 = {}, GL3 = {}, GL4 = {}, ES2 = {} }
TessControlShaders = { GL4 = {} }
TessEvaluationShaders = { GL4 = {} }
ApiVersionToLanguageVersion = { GL2 = 120, GL3 = 130, GL4 = 150 }

function DeclareShader(stage, apiVersion, techniqueName, source)
    local tableName = stage .. "Shaders"
    local languageVersion = ApiVersionToLanguageVersion[apiVersion]
    local lineNumber = debug.getinfo(2).currentline
    _G[tableName][apiVersion][techniqueName] = 
        '#version ' .. languageVersion .. '\n' ..
        '#line ' .. lineNumber .. '\n' .. source
end

Shaders for multiple versions of OpenGL can now be bundled into a single file:

-- Vertex Shader for OpenGL 2.0
DeclareShader('Vertex', 'GL2', 'SnazzyEffect', [[
void main() { /* FOO */ }
]])

-- Vertex Shader for OpenGL 3.0
DeclareShader('Vertex', 'GL3', 'SnazzyEffect', [[
void main() { /* BAR */ }
]])

Now that our shaders are hidden inside nested Lua tables, it’s a bit more footwork to access them from the C/C++ side, but we can hide the footwork behind a nice utility function like this:

const char* GetShaderSource(lua_State* L, const char* techniqueName,
                            const char* apiVersion, const char* shaderStage)
{
    // Append "Shaders" to the shader stage to obtain the table name:
    char tableName[32];
    strncpy(tableName, shaderStage, 24);
    strncat(tableName, "Shaders", 7);

    // Fetch the table from the Lua context and make sure it exists:
    lua_getglobal(L, tableName);
    if (!lua_istable(L, -1))
        return 0;

    // Make sure a table exists for the given API version:
    lua_pushstring(L, apiVersion);
    lua_gettable(L, -2);
    if (!lua_istable(L, -1))
        return 0;

    // Fetch the shader string:
    lua_pushstring(L, techniqueName);
    lua_gettable(L, -2);
    const char* shaderSource = lua_tostring(L, -1);

    // Clean up the Lua stack and return the string:
    lua_pop(L, 3);
    return shaderSource;
}

A Generalized and Terse Solution

Revisiting the Lua file, note that the shader declaration is still a bit more verbose than a dedicated effect language would be:

DeclareShader('Vertex', 'GL2', 'SnazzyEffect', 'void main() {}')

It might be nice to have Lua do some string parsing for us. Dot separators make for a nice, terse syntax. This is preferable:

DeclareShader('Vertex.GL2.SnazzyEffect', 'void main() {}')

Let’s call these dot-seperated strings shader keys. Couple more observations:

• Since we’ve deemed that each effect corresponds to a single Lua file, we can infer the effect name from the filename of the script itself.
• Instead of pre-declaring a bunch of Lua tables in ShaderFactory.lua for each programmable stage, we can create the tables dynamically.

Okay, so here’s our final version of ShaderFactory.lua:

ApiVersionToLanguageVersion = { GL2 = 120, GL3 = 140, GL4 = 150 }

function DeclareShader(shaderKey, shaderSource)
	
    -- Prepend the line number directive for proper error messages.
	local lineNumber = debug.getinfo(2).currentline
    shaderSource = "#line " .. lineNumber .. "\n" .. shaderSource

    -- Extract the technique name from the fullpath of the Lua script.
    local fullpath = debug.getinfo(2).source
    local f, l, technique = string.find(fullpath, "([A-Za-z]+)%.lua")

    -- If a table for this technique does not exist, create it.
    if _G[technique] == nil then
        _G[technique] = {}
    end

    -- Make sure this shader hasn't already been declared.
    if _G[technique][shaderKey] then
        error("Shader '" .. shaderKey .. "' has been declared twice.")
    end

    -- Check if an API version is in the shader key and prepend #version.
    local pos = 0
    repeat
        dummy, pos, token = string.find(shaderKey, "([A-Za-z0-9]+)", pos + 1)
        if token and ApiVersionToLanguageVersion[token] then
        	local langVersion = ApiVersionToLanguageVersion[token]
            shaderSource = "#version " .. langVersion .. "\n" .. shaderSource
        end
    until token == nil

    -- Add the shader to Lua's globals.
    _G[technique][shaderKey] = shaderSource

end

Now your effect file can look something like this:

------------ Vertex Shader for OpenGL 3.0 ------------

DeclareShader('Vertex.GL3.Erosion', [[
in vec4 Position;
void main()
{
    gl_Position = Position;
}
]])

------------ Fragment Shaders for OpenGL 3.0 ------------

DeclareShader('Fragment.GL3.Erosion.Kirk', [[
out vec4 FragColor;
void main()
{
    // ...snip...
}
]])

DeclareShader('Fragment.GL3.Erosion.Spock', [[
out vec4 FragColor;
void main()
{
    // ...snip...
}
]])

The above effect contains two fragment shaders but only one vertex shader. You’ll often find that the same vertex shader can be used for multiple fragment shaders, which is why COLLADA and the D3D FX format have so much cross-referencing. Our solution is a bit more simple: we’ll have our C/C++ utility function simply find the shader that has the longest matching shader key. You’ll see what I mean after an example.

Let’s define another term before listing out the new C/C++ utility function: an effect key is like a shader key, except that it has the effect name prepended.

const char* GetShaderSource(lua_State* L, const char* effectKey)
{
    // Extract the effect name:
    const char* targetKey = strchr(effectKey, '.');
    if (!targetKey++)
        return 0;

    char effectName[32] = {0};
    strncpy(effectName, effectKey, targetKey - effectKey - 1);

    // Fetch the table from the Lua context and make sure it exists:
    lua_getglobal(L, effectName);
    if (!lua_istable(L, -1))
    {
        lua_pop(L, 1);

        // Delay-load the Lua file:
        char effectPath[64];
        sprintf(effectPath, "%s.lua", effectName);
        if (luaL_dofile(L, effectPath))
            return 0;
        
        // If it's still not there, give up!
        lua_getglobal(L, effectName);
        if (!lua_istable(L, -1))
            exit(1);
    }

    const char* closestMatch = 0;
    int closestMatchLength = 0;

    int i = lua_gettop(L);
    lua_pushnil(L);
    while (lua_next(L, i) != 0)
    {
        const char* shaderKey = lua_tostring(L, -2);
        int shaderKeyLength = strlen(shaderKey);

        // Find the longest key that matches the beginning of the target key:
        if (strstr(targetKey, shaderKey) != 0 && shaderKeyLength > closestMatchLength)
        {
            closestMatchLength = shaderKeyLength;
            closestMatch = lua_tostring(L, -1);
        }

        lua_pop(L, 1);
    }

    lua_pop(L, 1);

    return closestMatch;
}

We can use the above utility function like this:

void main()
{
    lua_State* L = lua_open();
    luaL_openlibs(L);
    luaL_dofile(L, "ShaderFactory.lua");

    cout << GetShaderSource(L, "MyEffect.Vertex.GL3.Kirk") << endl << endl;
    cout << GetShaderSource(L, "MyEffect.Fragment.GL3.Kirk") << endl << endl;

    lua_close(L);
}

On line 7, the caller requests a shader from the MyEffect.lua file with a shader key of Vertex.GL3.Kirk. Since the longest matching shader key is Vertex.GL3, that’s what gets returned. Also note that the Lua script for the effect gets delay loaded.

By the way, before I let you go, let me show you a trick for commenting out big swaths of Lua code that have multi-line strings. Normally you’d use the - -[[ and - -]] delimiters for multi-line comments, but they don’t work if you’re commenting out sections that have strings delimited with [[ and ]]. Lua allows you to use alternative delimiters by inserting an arbitrary number of equal signs between the square brackets, like this:

--[==[
DeclareShader('Fragment.GL3.DisabledEffect', [[
out vec4 FragColor;
void main()
{
    // ...snip...
}
]])
--]==]

DeclareShader('Fragment.GL3.EnabledEffect', [[
out vec4 FragColor;
void main()
{
    // ...snip...
}
]])

Cool eh?