In the waave engine, a streaming object describe the methods to stream video frames to a specified target. To understand how to write one, we will look at the internal standard streaming object provided with the library that stream video on SDL surfaces. The source code can be found in the stream_overlay.c file.
First look at the object structure given in streaming_object.h :
/*******************/ /* the main object */ /*******************/ typedef struct WVStreamingObject{ /* the src format, initialized by the decoder */ /* init can use it to adjust his buffers */ int srcWidth; int srcHeight; enum PixelFormat srcFormat; //ffmpeg src format /* the size of the list where we decode frames */ int nbSlots; /****************************/ /* streaming object methods */ /****************************/ /* method type */ int getBufferMethod; //STATIC or DYNAMIC int GLRMethod; //SYNC or ASYNC get/lock/release int getThreadSafety; //get thread safety int LRThreadSafety; //Lock and release thread safety /* init the object */ int (*init)(struct WVStreamingObject* streamObj); /* get a frame buffer */ /* if we have a static get, this is done one time per slot */ /* if we have a dynamic get, we get each time we decode a new frame in a slot */ /* !!! NULL if useless !!!*/ WVStreamingBuffer (*getBuffer)(struct WVStreamingObject* streamObj, int slotIdx); /* lock the buffer before writing */ /* !!! NULL if useless !!! */ int (*lockBuffer)(struct WVStreamingObject* streamObj, int slotIdx); /* filter the buffer with the user method */ /* !!! need to be thread safe !!! */ /* else filter in releaseBuffer */ int (*filterBuffer)(struct WVStreamingObject* streamObj, int slotIdx); /* release the buffer after writing */ /* !!! NULL if useless !!! */ int (*releaseBuffer)(struct WVStreamingObject* streamObj, int slotIdx); /* refresh the frame by displaying his content */ int (*refreshFrame)(struct WVStreamingObject* streamObj, int slotIdx); /* close the object */ int (*close)(struct WVStreamingObject* streamObj); /****************/ /* private data */ /****************/ void* objPrivate; }WVStreamingObject;
The first three fields, srcWidth, srcheight and srcFormat are set by the decoder and give the video native format. This will be read by the object's method to build decoding buffers. All the others field must be filled at the object construction or with the init() method. Check the documentation to understand the meaning of each field. nbSlots is an very important field that give the number of simultaneous decoding buffer builded in the object. We look now to stream_overlay.c to see how the object is builded.
First we prepare a structure to put all the object private parameters. Our object stream video on sdl surface so we need to store the surface handle. The video frame are stored in a list of yuv overlay that we need to build. And we need to adjust our video format to the surface size so we store the target rectangle dimensions.
typedef struct StreamOverlayPrivate{ /* user params */ SDL_Surface* targetSurface; /* the object overlays */ SDL_Rect targetRect; SDL_Overlay** slotOverlay; }StreamOverlayPrivate;
The targetSurface will be set at object creation and init() will fill the others fields.
We start by writing a setOutputRect() function that will adjust the video frames to target surface conserving aspect ratio. Check source code for details. We can then start writing the init() method.
static int init_streamOverlay(WVStreamingObject* streamObj) { StreamOverlayPrivate* objPrivate = (StreamOverlayPrivate*)streamObj->objPrivate; /* set output rect */ setOutputRect(streamObj); /* alloc space for the overlays */ objPrivate->slotOverlay = (SDL_Overlay**)malloc(streamObj->nbSlots*sizeof(SDL_Overlay*)); int i; for(i=0; i<streamObj->nbSlots; i++) objPrivate->slotOverlay[i] = NULL; return 0; }
We set the output rectangle and allocate space to store the sdl yuv overlay's handles. Note that nbSlots will be set at object creation.
The object will be builded with a static getBuffer function. This mean that the function will be called just one time per slots to initialize the frame buffers. The decoder will request one buffer for each slots. Here the structure that we need to return :
/***************************/ /* frame buffer descriptor */ /***************************/ typedef struct WVStreamingBuffer{ /* the frame size and format */ int width; int height; enum PixelFormat format; //see ffmpeg format /* the frame plane */ uint8_t* data[4]; /* the corresponding plane linesize */ int linesize[4]; }WVStreamingBuffer;
We start to create a sdl overlay for the slot :
static WVStreamingBuffer getBuffer_streamOverlay(WVStreamingObject* streamObj, int slotIdx) { StreamOverlayPrivate* objPrivate = (StreamOverlayPrivate*)streamObj->objPrivate; /* create an overlay */ /* we use native size */ SDL_Overlay* newOverlay; if(!objPrivate->slotOverlay[slotIdx]){ newOverlay = SDL_CreateYUVOverlay(streamObj->srcWidth, streamObj->srcHeight, \ SDL_YV12_OVERLAY, objPrivate->targetSurface); /* save */ objPrivate->slotOverlay[slotIdx] = newOverlay; } else{ newOverlay = objPrivate->slotOverlay[slotIdx]; }
Next we fill the frame buffer dercriptor that we return to the decoder :
/* create the buffer */ WVStreamingBuffer newBuff; newBuff.width = streamObj->srcWidth; newBuff.height = streamObj->srcHeight; newBuff.format = PIX_FMT_YUV420P; /* fill frame plane */ newBuff.data[0] = newOverlay->pixels[0]; newBuff.data[1] = newOverlay->pixels[2]; newBuff.data[2] = newOverlay->pixels[1]; /* fill plane linesize */ newBuff.linesize[0] = newOverlay->pitches[0]; newBuff.linesize[1] = newOverlay->pitches[2]; newBuff.linesize[2] = newOverlay->pitches[1]; /* return the buffer */ return newBuff; }
The next two methods are trivials. We just need to call the corresponding sdl functions. Sdl overlays are locked with SDL_LockYUVOverlay and released with SDL_UnlockYUVOverlay.
static int lockBuffer_streamOverlay(WVStreamingObject* streamObj, int slotIdx) { StreamOverlayPrivate* objPrivate = (StreamOverlayPrivate*)streamObj->objPrivate; /* lock the overlay */ /* seems thread safe */ SDL_LockYUVOverlay(objPrivate->slotOverlay[slotIdx]); return 0; } static int releaseBuffer_streamOverlay(WVStreamingObject* streamObj, int slotIdx) { StreamOverlayPrivate* objPrivate = (StreamOverlayPrivate*)streamObj->objPrivate; /* release the overlay */ /* seems thread safe */ SDL_UnlockYUVOverlay(objPrivate->slotOverlay[slotIdx]); return 0; }
We give now the sdl function that blit an sdl yuv overlay on a sdl surface :
static int refreshFrame_streamOverlay(WVStreamingObject* streamObj, int slotIdx) { StreamOverlayPrivate* objPrivate = (StreamOverlayPrivate*)streamObj->objPrivate; /* blit the overlay */ return SDL_DisplayYUVOverlay(objPrivate->slotOverlay[slotIdx], &(objPrivate->targetRect)); }
Finally we give the close method that free all the yuv overlays and the overlay list :
static int close_streamOverlay(WVStreamingObject* streamObj) { StreamOverlayPrivate* objPrivate = (StreamOverlayPrivate*)streamObj->objPrivate; /* close all the overlays */ int i; for(i=0; i<streamObj->nbSlots; i++){ if(objPrivate->slotOverlay[i]){ SDL_FreeYUVOverlay(objPrivate->slotOverlay[i]); objPrivate->slotOverlay[i] = NULL; } } /* free the overlays buffer */ free(objPrivate->slotOverlay); return 0; }
We write now the function that will create the object for the user. We start by allocating memory for the object and it's private data :
WVStreamingObject* WV_getStreamOverlayObj(SDL_Surface* targetSurface) { /* alloc the struct */ WVStreamingObject* streamObj; streamObj = (WVStreamingObject*)malloc(sizeof(WVStreamingObject) + sizeof(StreamOverlayPrivate)); void* structP = (void*)streamObj; structP += sizeof(WVStreamingObject); streamObj->objPrivate = structP;
We have just to fill the struct fields. Note that we always use three slots. This may be set by a constructor parameters.
/* save user params */ StreamOverlayPrivate* objPrivate = (StreamOverlayPrivate*)streamObj->objPrivate; objPrivate->targetSurface = targetSurface; /**********************/ /* fill object params */ /**********************/ /* we use 3 slots */ streamObj->nbSlots = 3; /* methods params */ streamObj->getBufferMethod = WV_STATIC_GET; //get the buffers one time streamObj->GLRMethod = WV_ASYNC_GLR; //async streamObj->getThreadSafety = WV_NO_THREAD_SAFE; streamObj->LRThreadSafety = WV_THREAD_SAFE; //assume that lockSurface is thread safe /* methods */ streamObj->init = &init_streamOverlay; streamObj->getBuffer = &getBuffer_streamOverlay; streamObj->lockBuffer = &lockBuffer_streamOverlay; streamObj->filterBuffer = NULL; streamObj->releaseBuffer = &releaseBuffer_streamOverlay; streamObj->refreshFrame = &refreshFrame_streamOverlay; streamObj->close = &close_streamOverlay; /*********************/ /* return the object */ /*********************/ return streamObj; }
The destructor is very simple because the structure was allocated in one time :
void WV_freeStreamOverlayObj(WVStreamingObject* streamObj) { free(streamObj); }