.rst .pdf

open issue suggest edit

Contents

Encoding Procedures

There are two methods for shared memory allocation and handling in oneVPL: external and internal.

External Memory

The following pseudo code shows the encoding procedure with external memory (legacy mode):

MFXVideoENCODE_QueryIOSurf(session, &init_param, &request);
allocate_pool_of_frame_surfaces(request.NumFrameSuggested);
MFXVideoENCODE_Init(session, &init_param);
sts=MFX_ERR_MORE_DATA;
for (;;) {
   if (sts==MFX_ERR_MORE_DATA && !end_of_stream()) {
      find_unlocked_surface_from_the_pool(&surface);
      fill_content_for_encoding(surface);
   }
   surface2=end_of_stream()?NULL:surface;
   sts=MFXVideoENCODE_EncodeFrameAsync(session,NULL,surface2,bits,&syncp);
   if (end_of_stream() && sts==MFX_ERR_MORE_DATA) break;
   // Skipped other error handling
   if (sts==MFX_ERR_NONE) {
      MFXVideoCORE_SyncOperation(session, syncp, INFINITE);
      do_something_with_encoded_bits(bits);
   }
}
MFXVideoENCODE_Close(session);
free_pool_of_frame_surfaces();

Note the following key points about the example:

• The application uses the MFXVideoENCODE_QueryIOSurf() function to obtain the number of working frame surfaces required for reordering input frames.

• The application calls the MFXVideoENCODE_EncodeFrameAsync() function for the encoding operation. The input frame must be in an unlocked frame surface from the frame surface pool. If the encoding output is not available, the function returns the mfxStatus::MFX_ERR_MORE_DATA status code to request additional input frames.

• Upon successful encoding, the MFXVideoENCODE_EncodeFrameAsync() function returns mfxStatus::MFX_ERR_NONE. At this point, the encoded bitstream is not yet available because the MFXVideoENCODE_EncodeFrameAsync() function is asynchronous. The application must use the MFXVideoCORE_SyncOperation() function to synchronize the encoding operation before retrieving the encoded bitstream.

• At the end of the stream, the application continuously calls the MFXVideoENCODE_EncodeFrameAsync() function with a NULL surface pointer to drain any remaining bitstreams cached within the oneVPL encoder, until the function returns mfxStatus::MFX_ERR_MORE_DATA.

Note

It is the application’s responsibility to fill pixels outside of the crop window when it is smaller than the frame to be encoded, especially in cases when crops are not aligned to minimum coding block size (16 for AVC and 8 for HEVC and VP9).

Internal Memory

The following pseudo code shows the encoding procedure with internal memory:

MFXVideoENCODE_Init(session, &init_param);
sts=MFX_ERR_MORE_DATA;
for (;;) {
   if (sts==MFX_ERR_MORE_DATA && !end_of_stream()) {
      MFXMemory_GetSurfaceForEncode(session,&surface);
      fill_content_for_encoding(surface);
   }
   surface2=end_of_stream()?NULL:surface;
   sts=MFXVideoENCODE_EncodeFrameAsync(session,NULL,surface2,bits,&syncp);
   if (surface2) surface->FrameInterface->Release(surface2);
   if (end_of_stream() && sts==MFX_ERR_MORE_DATA) break;
   // Skipped other error handling
   if (sts==MFX_ERR_NONE) {
      MFXVideoCORE_SyncOperation(session, syncp, INFINITE);
      do_something_with_encoded_bits(bits);
   }
}
MFXVideoENCODE_Close(session);

There are several key differences in this example, compared to external memory (legacy mode):

• The application does not need to call the MFXVideoENCODE_QueryIOSurf() function to obtain the number of working frame surfaces since allocation is done by oneVPL.

• The application calls the MFXMemory_GetSurfaceForEncode() function to get a free surface for the subsequent encode operation.

• The application must call the mfxFrameSurfaceInterface::Release function to decrement the reference counter of the obtained surface after the call to the MFXVideoENCODE_EncodeFrameAsync() function.

Configuration Change

The application changes configuration during encoding by calling the MFXVideoENCODE_Reset() function. Depending on the difference in configuration parameters before and after the change, the oneVPL encoder will either continue the current sequence or start a new one. If the encoder starts a new sequence, it completely resets internal state and begins a new sequence with the IDR frame.

The application controls encoder behavior during parameter change by attaching the mfxExtEncoderResetOption structure to the mfxVideoParam structure during reset. By using this structure, the application instructs the encoder to start or not start a new sequence after reset. In some cases, the request to continue the current sequence cannot be satisfied and the encoder will fail during reset. To avoid this scenario, the application may query the reset outcome before the actual reset by calling the MFXVideoENCODE_Query() function with the mfxExtEncoderResetOption attached to the mfxVideoParam structure.

The application uses the following procedure to change encoding configurations:

1. The application retrieves any cached frames in the oneVPL encoder by calling the MFXVideoENCODE_EncodeFrameAsync() function with a NULL input frame pointer until the function returns mfxStatus::MFX_ERR_MORE_DATA.

2. The application calls the MFXVideoENCODE_Reset() function with the new configuration:

   • If the function successfully sets the configuration, the application can continue encoding as usual.

   • If the new configuration requires a new memory allocation, the function returns mfxStatus::MFX_ERR_INCOMPATIBLE_VIDEO_PARAM. The application must close the oneVPL encoder and reinitialize the encoding procedure with the new configuration.

External Bitrate Control

The application can make the encoder use the external Bitrate Control (BRC) instead of the native bitrate control. To make the encoder use the external BRC, the application should attach the mfxExtCodingOption2 structure with ExtBRC = MFX_CODINGOPTION_ON and the mfxExtBRC callback structure to the mfxVideoParam structure during encoder initialization. The Init, Reset, and Close callbacks will be invoked inside their corresponding functions: MFXVideoENCODE_Init(), MFXVideoENCODE_Reset(), and MFXVideoENCODE_Close(). The following figure shows asynchronous encoding flow with external BRC (using GetFrameCtrl and Update):

Asynchronous encoding flow with external BRC

Note

IntAsyncDepth is the oneVPL max internal asynchronous encoding queue size. It is always less than or equal to mfxVideoParam::AsyncDepth.

The following pseudo code shows use of the external BRC:

   #include "mfxvideo.h"
   #include "mfxbrc.h"

   typedef struct {
      mfxU32 EncodedOrder;
      mfxI32 QP;
      mfxU32 MaxSize;
      mfxU32 MinSize;
      mfxU16 Status;
      mfxU64 StartTime;
      // ... skipped
   } MyBrcFrame;

   typedef struct {
      MyBrcFrame* frame_queue;
      mfxU32 frame_queue_size;
      mfxU32 frame_queue_max_size;
      mfxI32 max_qp[3]; //I,P,B
      mfxI32 min_qp[3]; //I,P,B
      // ... skipped
   } MyBrcContext;

   void* GetExtBuffer(mfxExtBuffer** ExtParam, mfxU16 NumExtParam, mfxU32 bufferID)
   {
       int i=0;
       for(i = 0; i < NumExtParam; i++) {
           if(ExtParam[i]->BufferId == bufferID) return ExtParam[i];
       }
       return NULL;
   }

   static int IsParametersSupported(mfxVideoParam *par)
   {
       // do some checks
       return 1;
   }

   static int IsResetPossible(MyBrcContext* ctx, mfxVideoParam *par)
   {
       // do some checks
       return 1;
   }

   static MyBrcFrame* GetFrame(MyBrcFrame *frame_queue, mfxU32 frame_queue_size, mfxU32 EncodedOrder)
   {
       //do some logic
       if(frame_queue_size) return &frame_queue[0];
       return NULL;
   }

   static mfxU32 GetFrameCost(mfxU16 FrameType, mfxU16 PyramidLayer)
   {
       // calculate cost
       return 1;
   }

   static mfxU32 GetMinSize(MyBrcContext *ctx, mfxU32 cost)
   {
       // do some logic
       return 1;
   }

   static mfxU32 GetMaxSize(MyBrcContext *ctx, mfxU32 cost)
   {
       // do some logic
       return 1;
   }

   static mfxI32 GetInitQP(MyBrcContext *ctx, mfxU32 MinSize, mfxU32 MaxSize, mfxU32 cost)
   {
       // do some logic
       return 1;
   }

   static mfxU64 GetTime()
   {
       mfxU64 wallClock;
       return wallClock;
   }

   static void UpdateBRCState(mfxU32 CodedFrameSize, MyBrcContext *ctx)
   {
       return;
   }

   static void RemoveFromQueue(MyBrcFrame* frame_queue, mfxU32 frame_queue_size, MyBrcFrame* frame)
   {
       return;
   }

   static mfxU64 GetMaxFrameEncodingTime(MyBrcContext *ctx)
   {
       return 2;
   }

   mfxStatus MyBrcInit(mfxHDL pthis, mfxVideoParam* par) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;
      mfxI32 QpBdOffset;
      mfxExtCodingOption2* co2;
      mfxI32 defaultQP;

      if (!pthis || !par)
         return MFX_ERR_NULL_PTR;

      if (!IsParametersSupported(par))
         return MFX_ERR_UNSUPPORTED;

      ctx->frame_queue_max_size = par->AsyncDepth;
      ctx->frame_queue = (MyBrcFrame*)malloc(sizeof(MyBrcFrame) * ctx->frame_queue_max_size);

      if (!ctx->frame_queue)
         return MFX_ERR_MEMORY_ALLOC;

      co2 = (mfxExtCodingOption2*)GetExtBuffer(par->ExtParam, par->NumExtParam, MFX_EXTBUFF_CODING_OPTION2);
      QpBdOffset = (par->mfx.FrameInfo.BitDepthLuma > 8) ? (6 * (par->mfx.FrameInfo.BitDepthLuma - 8)) : 0;

      ctx->max_qp[0] = (co2 && co2->MaxQPI) ? (co2->MaxQPI - QpBdOffset) : defaultQP;
      ctx->min_qp[0] = (co2 && co2->MinQPI) ? (co2->MinQPI - QpBdOffset) : defaultQP;

      ctx->max_qp[1] = (co2 && co2->MaxQPP) ? (co2->MaxQPP - QpBdOffset) : defaultQP;
      ctx->min_qp[1] = (co2 && co2->MinQPP) ? (co2->MinQPP - QpBdOffset) : defaultQP;

      ctx->max_qp[2] = (co2 && co2->MaxQPB) ? (co2->MaxQPB - QpBdOffset) : defaultQP;
      ctx->min_qp[2] = (co2 && co2->MinQPB) ? (co2->MinQPB - QpBdOffset) : defaultQP;

      // skipped initialization of other other BRC parameters

      ctx->frame_queue_size = 0;

      return MFX_ERR_NONE;
   }

   mfxStatus MyBrcReset(mfxHDL pthis, mfxVideoParam* par) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;

      if (!pthis || !par)
         return MFX_ERR_NULL_PTR;

      if (!IsParametersSupported(par))
         return MFX_ERR_UNSUPPORTED;

      if (!IsResetPossible(ctx, par))
         return MFX_ERR_INCOMPATIBLE_VIDEO_PARAM;

      // reset here BRC parameters if required

      return MFX_ERR_NONE;
   }

   mfxStatus MyBrcClose(mfxHDL pthis) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;

      if (!pthis)
         return MFX_ERR_NULL_PTR;

      if (ctx->frame_queue) {
         free(ctx->frame_queue);
         ctx->frame_queue = NULL;
         ctx->frame_queue_max_size = 0;
         ctx->frame_queue_size = 0;
      }

      return MFX_ERR_NONE;
   }

   mfxStatus MyBrcGetFrameCtrl(mfxHDL pthis, mfxBRCFrameParam* par, mfxBRCFrameCtrl* ctrl) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;
      MyBrcFrame* frame = NULL;
      mfxU32 cost;

      if (!pthis || !par || !ctrl)
         return MFX_ERR_NULL_PTR;

      if (par->NumRecode > 0)
         frame = GetFrame(ctx->frame_queue, ctx->frame_queue_size, par->EncodedOrder);
      else if (ctx->frame_queue_size < ctx->frame_queue_max_size)
         frame = &ctx->frame_queue[ctx->frame_queue_size++];

      if (!frame)
         return MFX_ERR_UNDEFINED_BEHAVIOR;

      if (par->NumRecode == 0) {
         frame->EncodedOrder = par->EncodedOrder;
         cost = GetFrameCost(par->FrameType, par->PyramidLayer);
         frame->MinSize = GetMinSize(ctx, cost);
         frame->MaxSize = GetMaxSize(ctx, cost);
         frame->QP = GetInitQP(ctx, frame->MinSize, frame->MaxSize, cost); // from QP/size stat
         frame->StartTime = GetTime();
      }

      ctrl->QpY = frame->QP;

      return MFX_ERR_NONE;
   }

   #define DEFAULT_QP_INC 4
   #define DEFAULT_QP_DEC 4

   mfxStatus MyBrcUpdate(mfxHDL pthis, mfxBRCFrameParam* par, mfxBRCFrameCtrl* ctrl, mfxBRCFrameStatus* status) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;
      MyBrcFrame* frame = NULL;
      mfxU32 panic = 0;

      if (!pthis || !par || !ctrl || !status)
         return MFX_ERR_NULL_PTR;

      frame = GetFrame(ctx->frame_queue, ctx->frame_queue_size, par->EncodedOrder);
      if (!frame)
         return MFX_ERR_UNDEFINED_BEHAVIOR;

      // update QP/size stat here

      if (   frame->Status == MFX_BRC_PANIC_BIG_FRAME
        || frame->Status == MFX_BRC_PANIC_SMALL_FRAME)
         panic = 1;

      if (panic || (par->CodedFrameSize >= frame->MinSize && par->CodedFrameSize <= frame->MaxSize)) {
         UpdateBRCState(par->CodedFrameSize, ctx);
         RemoveFromQueue(ctx->frame_queue, ctx->frame_queue_size, frame);
         ctx->frame_queue_size--;
         status->BRCStatus = MFX_BRC_OK;

         // Here update Min/MaxSize for all queued frames

         return MFX_ERR_NONE;
      }

      panic = ((GetTime() - frame->StartTime) >= GetMaxFrameEncodingTime(ctx));

      if (par->CodedFrameSize > frame->MaxSize) {
         if (panic || (frame->QP >= ctx->max_qp[0])) {
            frame->Status = MFX_BRC_PANIC_BIG_FRAME;
         } else {
            frame->Status = MFX_BRC_BIG_FRAME;
            frame->QP = DEFAULT_QP_INC;
         }
      }

      if (par->CodedFrameSize < frame->MinSize) {
         if (panic || (frame->QP <= ctx->min_qp[0])) {
            frame->Status = MFX_BRC_PANIC_SMALL_FRAME;
            status->MinFrameSize = frame->MinSize;
         } else {
            frame->Status = MFX_BRC_SMALL_FRAME;
            frame->QP = DEFAULT_QP_DEC;
         }
      }

      status->BRCStatus = frame->Status;

      return MFX_ERR_NONE;
   }

   void EncoderInit()
   {
        //initialize encoder
        MyBrcContext brc_ctx;
        mfxExtBRC ext_brc;
        mfxExtCodingOption2 co2;
        mfxExtBuffer* ext_buf[2] = {&co2.Header, &ext_brc.Header};
        mfxVideoParam vpar;

        memset(&brc_ctx, 0, sizeof(MyBrcContext));
        memset(&ext_brc, 0, sizeof(mfxExtBRC));
        memset(&co2, 0, sizeof(mfxExtCodingOption2));

        vpar.ExtParam = ext_buf;
        vpar.NumExtParam = sizeof(ext_buf) / sizeof(ext_buf[0]);

        co2.Header.BufferId = MFX_EXTBUFF_CODING_OPTION2;
        co2.Header.BufferSz = sizeof(mfxExtCodingOption2);
        co2.ExtBRC = MFX_CODINGOPTION_ON;

        ext_brc.Header.BufferId = MFX_EXTBUFF_BRC;
        ext_brc.Header.BufferSz = sizeof(mfxExtBRC);
        ext_brc.pthis           = &brc_ctx;
        ext_brc.Init            = MyBrcInit;
        ext_brc.Reset           = MyBrcReset;
        ext_brc.Close           = MyBrcClose;
        ext_brc.GetFrameCtrl    = MyBrcGetFrameCtrl;
        ext_brc.Update          = MyBrcUpdate;

        sts = MFXVideoENCODE_Query(session, &vpar, &vpar);
        if (sts == MFX_ERR_UNSUPPORTED || co2.ExtBRC != MFX_CODINGOPTION_ON)
            // unsupported case
            sts = sts;
        else
            sts = MFXVideoENCODE_Init(session, &vpar);
   }

JPEG

The application can use the same encoding procedures for JPEG/motion JPEG encoding, as shown in the following pseudo code:

// encoder initialization
MFXVideoENCODE_Init (...);
// single frame/picture encoding
MFXVideoENCODE_EncodeFrameAsync (...);
MFXVideoCORE_SyncOperation(...);
// close down
MFXVideoENCODE_Close(...);

The application may specify Huffman and quantization tables during encoder initialization by attaching mfxExtJPEGQuantTables and mfxExtJPEGHuffmanTables buffers to the mfxVideoParam structure. If the application does not define tables, then the oneVPL encoder uses tables recommended in ITU-T* Recommendation T.81. If the application does not define a quantization table it must specify the mfxInfoMFX::Quality parameter. In this case, the oneVPL encoder scales the default quantization table according to the specified mfxInfoMFX::Quality parameter value.

The application should properly configure chroma sampling format and color format using the mfxFrameInfo::FourCC and mfxFrameInfo::ChromaFormat fields. For example, to encode a 4:2:2 vertically sampled YCbCr picture, the application should set mfxFrameInfo::FourCC to MFX_FOURCC_YUY2 and mfxFrameInfo::ChromaFormat to MFX_CHROMAFORMAT_YUV422V. To encode a 4:4:4 sampled RGB picture, the application should set mfxFrameInfo::FourCC to MFX_FOURCC_RGB4 and mfxFrameInfo::ChromaFormat to MFX_CHROMAFORMAT_YUV444.

The oneVPL encoder supports different sets of chroma sampling and color formats on different platforms. The application must call the MFXVideoENCODE_Query() function to check if the required color format is supported on a given platform and then initialize the encoder with proper values of mfxFrameInfo::FourCC and mfxFrameInfo::ChromaFormat.

The application should not define the number of scans and number of components. These numbers are derived by the oneVPL encoder from the mfxInfoMFx::Interleaved flag and from chroma type. If interleaved coding is specified, then one scan is encoded that contains all image components. Otherwise, the number of scans is equal to number of components. The encoder uses the following component IDs: “1” for luma (Y), “2” for chroma Cb (U), and “3” for chroma Cr (V).

The application should allocate a buffer that is big enough to hold the encoded picture. A rough upper limit may be calculated using the following equation where Width and Height are width and height of the picture in pixel and BytesPerPx is the number of bytes for one pixel:

BufferSizeInKB = 4 + (Width * Height * BytesPerPx + 1023) / 1024;

The equation equals 1 for a monochrome picture, 1.5 for NV12 and YV12 color formats, 2 for YUY2 color format, and 3 for RGB32 color format (alpha channel is not encoded).

Multi-view Video Encoding

Similar to the decoding and video processing initialization procedures, the application attaches the mfxExtMVCSeqDesc structure to the mfxVideoParam structure for encoding initialization. The mfxExtMVCSeqDesc structure configures the oneVPL MVC encoder to work in three modes:

• Default dependency mode: The application specifies mfxExtMVCSeqDesc::NumView and all other fields to zero. The oneVPL encoder creates a single operation point with all views (view identifier 0 : NumView-1) as target views. The first view (view identifier 0) is the base view. Other views depend on the base view.

• Explicit dependency mode: The application specifies mfxExtMVCSeqDesc::NumView and the view dependency array, and sets all other fields to zero. The oneVPL encoder creates a single operation point with all views (view identifier View[0 : NumView-1].ViewId) as target views. The first view (view identifier View[0].ViewId) is the base view. View dependencies are defined as mfxMVCViewDependency structures.

• Complete mode: The application fully specifies the views and their dependencies. The oneVPL encoder generates a bitstream with corresponding stream structures.

During encoding, the oneVPL encoding function MFXVideoENCODE_EncodeFrameAsync() accumulates input frames until encoding of a picture is possible. The function returns mfxStatus::MFX_ERR_MORE_DATA for more data at input or mfxStatus::MFX_ERR_NONE if it successfully accumulated enough data for encoding a picture. The generated bitstream contains the complete picture (multiple views). The application can change this behavior and instruct the encoder to output each view in a separate bitstream buffer. To do so, the application must turn on the mfxExtCodingOption::ViewOutput flag. In this case, the encoder returns mfxStatus::MFX_ERR_MORE_BITSTREAM if it needs more bitstream buffers at output and mfxStatus::MFX_ERR_NONE when processing of the picture (multiple views) has been finished. It is recommended that the application provide a new input frame each time the oneVPL encoder requests a new bitstream buffer. The application must submit view data for encoding in the order they are described in the mfxExtMVCSeqDesc structure. Particular view data can be submitted for encoding only when all views that it depends upon have already been submitted.

The following pseudo code shows the encoding procedure:

mfxExtBuffer *eb;
mfxExtMVCSeqDesc  seq_desc;
mfxVideoParam init_param;

init_param.ExtParam=(mfxExtBuffer **)&eb;
init_param.NumExtParam=1;
eb=(mfxExtBuffer *)&seq_desc;

/* init encoder */
MFXVideoENCODE_Init(session, &init_param);

/* perform encoding */
for (;;) {
    MFXVideoENCODE_EncodeFrameAsync(session, NULL, surface2, bits,
                                    &syncp);
    MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
}

/* close encoder */
MFXVideoENCODE_Close(session);

Decoding Procedures Video Processing Procedures