eglCreateImageKHR returning EGL_BAD_ATTRIBUTE error - linux

Question

I have implemented hardware decoding on Linux using VAAPI via FFmpeg. Since I have an OpenGL application, I am converting the decoded VAAPI surfaces to OpenGL textures using vaCopySurfaceGLX. This is working fine except that there is a copy (on the GPU) that is made. I was told that I could directly use the VAAPI surface as OpenGL textures using EGL. I have looked at some examples (mainly Kodi source code) but I'm not able to create the EGLImageKHR. The function eglCreateImageKHR returns 0, and when I check for errors, I get a EGL_BAD_ATTRIBUTE error but I don't understand why.
Below is how I'm converting the VAAPI surface.
During initialization, I set up EGL this way:
// currentDisplay comes from call to glXGetCurrentDisplay() and is also used when getting the VADisplay like this: vaGetDisplay(currentDisplay)
EGLint major, minor;
_eglDisplay = eglGetDisplay(currentDisplay);
eglInitialize(_eglDisplay, &major, &minor);
eglBindAPI(EGL_OPENGL_API);
Then later, to create my EGL image, this is what I do:
// _vaapiContext.vaDisplay comes from vaGetDisplay(currentDisplay)
// surface is the VASurfaceID of the surface I want to use in OpenGL
vaDeriveImage(_vaapiContext.vaDisplay, surface, &_vaapiContext.vaImage);
VABufferInfo buf_info;
memset(&buf_info, 0, sizeof(buf_info));
buf_info.mem_type = VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME;
vaAcquireBufferHandle(_vaapiContext.vaDisplay, _vaapiContext.vaImage.buf, &buf_info);
EGLint attribs[] = {
EGL_WIDTH, _vaapiContext.vaImage.width,
EGL_HEIGHT, _vaapiContext.vaImage.height,
EGL_LINUX_DRM_FOURCC_EXT, fourcc_code('R', '8', ' ', ' '),
EGL_DMA_BUF_PLANE0_FD_EXT, buf_info.handle,
EGL_DMA_BUF_PLANE0_OFFSET_EXT, _vaapiContext.vaImage.offsets[0],
EGL_DMA_BUF_PLANE0_PITCH_EXT, _vaapiContext.vaImage.pitches[0],
EGL_NONE
};
EGLImageKHR eglImage = eglCreateImageKHR(_eglDisplay, EGL_NO_CONTEXT, EGL_LINUX_DMA_BUF_EXT, (EGLClientBuffer)NULL, attribs);
Looking at what could cause this error in the following document https://www.khronos.org/registry/egl/extensions/EXT/EGL_EXT_image_dma_buf_import.txt, I also tried to add the following options which should not matter since my format is not planar
EGL_YUV_COLOR_SPACE_HINT_EXT, EGL_ITU_REC601_EXT,
EGL_SAMPLE_RANGE_HINT_EXT, EGL_YUV_FULL_RANGE_EXT,
EGL_YUV_CHROMA_HORIZONTAL_SITING_HINT_EXT, EGL_YUV_CHROMA_SITING_0_EXT,
EGL_YUV_CHROMA_VERTICAL_SITING_HINT_EXT, EGL_YUV_CHROMA_SITING_0_EXT
The code that I'm using is similar to all the examples I've seen so I'm not sure what the error is.
Note that I have removed all the error checks for this post. All the calls above are successful except for eglCreateImageKHR.

Answer 1

After turning the egl log level to debug, I was able to get more information about the error and pinpointed where in the egl source code this error happened. It turns out that the format fourcc_code('R', '8', ' ', ' ') was not supported because my mesa version was too old. You need to have mesa 11.0.0 or above installed. After recompiling mesa (I'm running Ubuntu 15.04) and installing the 11.0.0 version, I'm finally getting an EGL image.