The SDL forums have moved to discourse.libsdl.org.
This is just a read-only archive of the previous forums, to keep old links working.

SDL Forum Index
SDL
Simple Directmedia Layer Forums
CreateTextureFromOpaque (prototype patch attached)...
oviano


Joined: 05 Mar 2016
Posts: 24
Post Posted: Mon Dec 26, 2016 11:15 am
First some background information...

I've been working on integrating libVLC into an iOS SDL application.

To begin with I was using libVLC's "vmem" video output module which delivers the frames as simple I420 or NV12 planes in main memory. I would then use SDL_UpdateTexture to copy these into my texture. It worked but you could only get limited performance using this method on iOS. I couldn't get 1080p60 playing smoothly for example.

I then started experimenting with modifying vmem in libVLC to be able to deal with frames produced by the iOS videotoolbox hardware decoder. This delivers frames as zero-copied CVPixelBufferRef objects which was more efficient, but still I had to then lock this object, and copy the data into a texture using SDL_UpdateTexture so although the hardware decoding was now accelerated I still had to do the copy in the SDL app which is slow due to the use of glTextSubImage2D. Still, 1080p60 was now possible albeit using more CPU than I would like.

I read up on it some more, looked at some of the iOS VLC code and discovered that the key to solving this is to be able to take a CVPixelBufferRef object and use the iOS GLES2 texture cache functionality to create the texture. Since the CVPixelBuffer objects delivered by libVLC via videotoolbox are IO-surface backed it makes the copy very efficient.

This is where CreateTextureOpaque comes from. This basically goes through the usual method of creating an SDL texture but at the point at which it would normally call glGenTextures to create the textures, it instead calls a user supplied callback to create the actual GL texture. Similarly when the texture is destroyed in SDL_DestroyTexture it calls a user supplied callback (which it saves at the time the texture was created) which can do any deallocation required.

I then use a callback similar to that below to supply a texture created using the CVOpenGLESTextureCacheCreate instead.

It works very well, and eliminates all copying - video is decoded by videotoolbox and efficiently ends up in an SDL_Texture and as far as I can tell SDL is perfectly happy with this arrangement.

The patch is here, but I've only implemented the functionality in the GLES2 part of it at this stage pending further discussion but maybe this or something like it could allow similar optimisations for other platforms/versions in scenarios where a an SDL_Texture needs to be created out of an already-existing one created by another library or something?

https://www.dropbox.com/s/2mrewz3ug6syp5h/create-texture-from-opaque-source-prototype.patch?dl=0

Code:


void VLC_VIDEO_PLAYER::create_cvpx_texture(void)
{
            if (!(cvpx_texture = SDL_CreateTextureOpaque(renderer, SDL_PIXELFORMAT_NV12, SDL_TEXTUREACCESS_STREAMING, pixels_pitch, pixels_lines, texture_cb, NULL, this)))
}

void *VLC_VIDEO_PLAYER::texture_cb(int plane_index, void *opaque)
{
   @autoreleasepool
   {
      VLC_VIDEO_PLAYER* vp = (VLC_VIDEO_PLAYER*)opaque;

      assert(vp != NULL);

      // texture cache not valid?
       if (!vp->texture_cache)
      {
         // create texture cache
         EAGLContext *context = (__bridge EAGLContext *) SDL_GL_GetCurrentContext();
           CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, context, NULL, &vp->texture_cache);
           if (err != noErr)
         {
            OUTPUT_ERROR("VLC_VIDEO_PLAYER: Error creating texture cache\n");
         
            return NULL;
           }
       }
   
      // set pixel buffer
      CVPixelBufferRef pixel_buffer = (CVPixelBufferRef)vp->pixels;

      // set width/height
      int width = (int)CVPixelBufferGetWidth(pixel_buffer);
       int height = (int)CVPixelBufferGetHeight(pixel_buffer);

      // luma plane?
      if (plane_index == 0)
      {
         // create luma texture
           CVReturn err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, vp->texture_cache, pixel_buffer, NULL, GL_TEXTURE_2D,  GL_LUMINANCE, width, height, GL_LUMINANCE, GL_UNSIGNED_BYTE, 0, &vp->luma_texture);
           if (err != noErr)
         {
            OUTPUT_ERROR("VLC_VIDEO_PLAYER: Error creating luma texture\n");
         
            return NULL;
           }

         return (void*)(uintptr_t)CVOpenGLESTextureGetName(vp->luma_texture);
      }
      
      // chroma plane?
      else
      {
         // create chroma texture
           CVReturn err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, vp->texture_cache, pixel_buffer, NULL, GL_TEXTURE_2D, GL_LUMINANCE_ALPHA, width / 2, height / 2, GL_LUMINANCE_ALPHA, GL_UNSIGNED_BYTE, 1, &vp->chroma_texture);
           if (err != noErr)
         {
            OUTPUT_ERROR("VLC_VIDEO_PLAYER: Error creating chroma texture\n");
         
            return 0;
           }

         return (void*)(uintptr_t)CVOpenGLESTextureGetName(vp->chroma_texture);
      }
   }
   
   return NULL;
}