Recently I’ve had to work more and more on OpenGL code as part of my job, which is hardly surprising given that my new job is to work on graphics. One thing that’s annoyed me since I started, however, is the relative difficulty of debugging OpenGL code compared to normal C/C++ code that just runs on the CPU.
The main reason for this, I’ve found, is that keeping track of which OpenGL commands have been issued, with what parameters, and what state has been set on your GL context is actually a rather hard task. In fact, it’s such a common problem that some bright hackers came up with the holy grail of OpenGL debugging tools – a tool called apitrace.
Put simply, apitrace is just a command tracer that logs all the OpenGL calls you make from your application. Why is that so wonderful? Well, for a start it decouples your development and testing environments. It allows you to record a series of OpenGL commands on a target device or piece of software that’s exhibiting a bug, which you can then replay and debug at your leisure in your native development environment.
Anyone who has had the pleasure of trying to debug OpenGL ES 2.0 code running on an embedded device running something like Android will understand the value here. You can just trace your buggy application, put the trace file on your desktop or laptop, analyse the GL commands you issued and modify them, then fix the bug. Problem solved! No messing around with a huge number of printf() statements or GL debugging states.
Well that’s all well and good, but how do you use this thing? Turns out on Android, that’s not so easy. First off you’ll need to grab my Android branch of apitrace (I’m working on getting these patches upstreamed, so don’t worry), and build it for your device:
export ANDROID_NDK=/path/to/your/android/ndk cmake \ -DCMAKE_TOOLCHAIN_FILE=android/android.toolchain.cmake \ -DANDROID_API_LEVEL=9 -Bbuild -H. make -C build
When this is done, you’ll find a file called egltrace.so in build/wrappers which you can then put somewhere on your device, such as /data/local.
However, on Android I have yet to find a way to preload a library, using the LD_PRELOAD environment variable or otherwise, so you’ll have to put the following lines of code before you make any gl calls in your application:
setenv("TRACE_FILE", "/path/to/trace/file", false); dlopen("/data/local/egltrace.so", RTLD_LAZY);
This will ensure that the symbols can be found, but you also need to actually look up the value of each gl function you’re hoping to use before you can start to get anywhere. In the case of glGetError(), this can be:
typedef glGetErrorFn (GLenum (*)(void)); glGetErrorFn fGetError = (glGetErrorFn)dlsym(RTLD_DEFAULT, "glGetError");
Unfortunately this will need to be done for all the symbols you’re planning to use, but on the up side you get total control over when your dynamic libraries are loaded and used, which means you can optimise your startup time accordingly.
Once that’s all set up you can go ahead and run your application and grab the tracing output. This is where the fun part starts. apitrace has a GUI written in Qt by Zack Rusin that can be used to do all sorts of crazy stuff, such as:
- View all GL commands issued, frame by frame
- Modify parameters passed into GL commands on the fly
- View current texture data*
- View bound shader programs
- Inspect GL context state at any point
- Replay traces
- View current uniforms
You get the idea. Whilst not all of the features seem to be working at the moment with EGL/GLESv2 traces, I hope to devote some spare cycles to fixing those. The most important one to me right now is that qapitrace is unable to view current texture data from traces we obtain from Firefox/Android. It seems unlikely that it’s an issue with our tracing support as replaying the traces using eglretrace works fine, but without investigating further I can’t say whether this is a limitation of qapitrace with EGL/GLESv2 or an issue with our tracing in Firefox. I do get the impression that upstream are targeting desktop GL rather than embedded GL, but that just gives me an opportunity to learn a bit more GL and help out!
Getting EGL playbacks to work on Linux can be a bit trying however. First off you will need to get the Mesa EGL and GLESv2 development libraries, as well as the usual requirements for building apitrace – Qt version 4.7 and so on – and you can build as per the installation instructions. Before running qapitrace or eglretrace though, you will need to set the following environment variable or I found (on my system at least) DRI fails to authenticate with the kernel:
Of course everything that’s been said here also works great for debugging desktop GL applications, but there’s significantly less pain involved as you shouldn’t need to resort to dlopen/dlsym magic.