Video Processing Procedures

The following pseudo code shows the video processing procedure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
MFXVideoVPP_QueryIOSurf(session, &init_param, response);
allocate_pool_of_surfaces(in_pool, response[0].NumFrameSuggested);
allocate_pool_of_surfaces(out_pool, response[1].NumFrameSuggested);
MFXVideoVPP_Init(session, &init_param);
mfxFrameSurface1 *in=find_unlocked_surface_and_fill_content(in_pool);
mfxFrameSurface1 *out=find_unlocked_surface_from_the_pool(out_pool);
for (;;) {
   sts=MFXVideoVPP_RunFrameVPPAsync(session,in,out,NULL,&syncp);
   if (sts==MFX_ERR_MORE_SURFACE || sts==MFX_ERR_NONE) {
      MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
      process_output_frame(out);
      out=find_unlocked_surface_from_the_pool(out_pool);
   }
   if (sts==MFX_ERR_MORE_DATA && in==NULL)
      break;
   if (sts==MFX_ERR_NONE || sts==MFX_ERR_MORE_DATA) {
      in=find_unlocked_surface_from_the_pool(in_pool);
      fill_content_for_video_processing(in);
      if (end_of_stream())
         in=NULL;
   }
}
MFXVideoVPP_Close(session);
free_pool_of_surfaces(in_pool);
free_pool_of_surfaces(out_pool);

Note the following key points about the example:

  • The application uses the MFXVideoVPP_QueryIOSurf() function to obtain the number of frame surfaces needed for input and output. The application must allocate two frame surface pools: one for the input and one for the output.

  • The video processing function MFXVideoVPP_RunFrameVPPAsync() is asynchronous. The application must use the MFXVideoCORE_SyncOperation() function to synchronize in order to make the output result ready.

  • The body of the video processing procedure covers the following three scenarios:

    • If the number of frames consumed at input is equal to the number of frames generated at output, VPP returns mfxStatus::MFX_ERR_NONE when an output is ready. The application must process the output frame after synchronization, as the MFXVideoVPP_RunFrameVPPAsync() function is asynchronous. The application must provide a NULL input at the end of the sequence to drain any remaining frames.

    • If the number of frames consumed at input is more than the number of frames generated at output, VPP returns mfxStatus::MFX_ERR_MORE_DATA for additional input until an output is ready. When the output is ready, VPP returns mfxStatus::MFX_ERR_NONE. The application must process the output frame after synchronization and provide a NULL input at the end of the sequence to drain any remaining frames.

    • If the number of frames consumed at input is less than the number of frames generated at output, VPP returns either mfxStatus::MFX_ERR_MORE_SURFACE (when more than one output is ready), or mfxStatus::MFX_ERR_NONE (when one output is ready and VPP expects new input). In both cases, the application must process the output frame after synchronization and provide a NULL input at the end of the sequence to drain any remaining frames.

Configuration

oneVPL configures the video processing pipeline operation based on the difference between the input and output formats, specified in the mfxVideoParam structure. The following list shows several examples:

  • When the input color format is YUY2 and the output color format is NV12, oneVPL enables color conversion from YUY2 to NV12.

  • When the input is interleaved and the output is progressive, oneVPL enables deinterlacing.

  • When the input is single field and the output is interlaced or progressive, oneVPL enables field weaving, optionally with deinterlacing.

  • When the input is interlaced and the output is single field, oneVPL enables field splitting.

In addition to specifying the input and output formats, the application can provide hints to fine-tune the video processing pipeline operation. The application can disable filters in the pipeline by using the mfxExtVPPDoNotUse structure, enable filters by using the mfxExtVPPDoUse structure, and configure filters by using dedicated configuration structures. See the Configurable VPP Filters table for a complete list of configurable video processing filters, their IDs, and configuration structures. See the ExtendedBufferID enumerator for more details.

oneVPL ensures that all filters necessary to convert the input format to the output format are included in the pipeline. oneVPL may skip some optional filters even if they are explicitly requested by the application, for example due to limitations of the underlying hardware. To notify the application about skipped optional filters, oneVPL returns the mfxStatus::MFX_WRN_FILTER_SKIPPED warning. The application can retrieve the list of active filters by attaching the mfxExtVPPDoUse structure to the mfxVideoParam structure and calling the MFXVideoVPP_GetVideoParam() function. The application must allocate enough memory for the filter list.

See the Configurable VPP Filters table for a full list of configurable filters.

Configurable VPP Filters

Filter ID

Configuration Structure

MFX_EXTBUFF_VPP_DENOISE

mfxExtVPPDenoise

MFX_EXTBUFF_VPP_DENOISE2

mfxExtVPPDenoise2

MFX_EXTBUFF_VPP_MCTF

mfxExtVppMctf

MFX_EXTBUFF_VPP_DETAIL

mfxExtVPPDetail

MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION

mfxExtVPPFrameRateConversion

MFX_EXTBUFF_VPP_IMAGE_STABILIZATION

mfxExtVPPImageStab

MFX_EXTBUFF_VPP_PROCAMP

mfxExtVPPProcAmp

MFX_EXTBUFF_VPP_FIELD_PROCESSING

mfxExtVPPFieldProcessing

The following example shows video processing configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
/* enable image stabilization filter with default settings */
mfxExtVPPDoUse du;
mfxU32 al=MFX_EXTBUFF_VPP_IMAGE_STABILIZATION;

du.Header.BufferId=MFX_EXTBUFF_VPP_DOUSE;
du.Header.BufferSz=sizeof(mfxExtVPPDoUse);
du.NumAlg=1;
du.AlgList=&al;

/* configure the mfxVideoParam structure */
mfxVideoParam conf;
mfxExtBuffer *eb=(mfxExtBuffer *)&du;

memset(&conf,0,sizeof(conf));
conf.IOPattern=MFX_IOPATTERN_IN_SYSTEM_MEMORY | MFX_IOPATTERN_OUT_SYSTEM_MEMORY;
conf.NumExtParam=1;
conf.ExtParam=&eb;

conf.vpp.In.FourCC=MFX_FOURCC_YV12;
conf.vpp.Out.FourCC=MFX_FOURCC_NV12;
conf.vpp.In.Width=conf.vpp.Out.Width=1920;
conf.vpp.In.Height=conf.vpp.Out.Height=1088;

/* video processing initialization */
MFXVideoVPP_Init(session, &conf);

Region of Interest

During video processing operations, the application can specify a region of interest for each frame as shown in the following figure:

VPP region of interest operation

VPP region of interest operation

Specifying a region of interest guides the resizing function to achieve special effects, such as resizing from 16:9 to 4:3, while keeping the aspect ratio intact. Use the CropX, CropY, CropW, and CropH parameters in the mfxVideoParam structure to specify a region of interest.

The VPP Region of Interest Operations table shows examples of VPP operations applied to a region of interest.

VPP Region of Interest Operations
Operation


VPP Input
Width X Height

VPP Input
CropX, CropY,
CropW, CropH
VPP Output
Width X Height

VPP Output
CropX, CropY,
CropW, CropH

Cropping

720 x 480

16, 16, 688, 448

720 x 480

16, 16, 688, 448

Resizing

720 x 480

0, 0, 720, 480

1440 x 960

0, 0, 1440, 960

Horizontal stretching

720 x 480

0, 0, 720, 480

640 x 480

0, 0, 640, 480

16:9 4:3 with letter boxing at the top and bottom

1920 x 1088

0, 0, 1920, 1088

720 x 480

0, 36, 720, 408

4:3 16:9 with pillar boxing at the left and right

720 x 480

0, 0, 720, 480

1920 x 1088

144, 0, 1632, 1088

Multi-view Video Processing

oneVPL video processing supports processing multiple views. For video processing initialization, the application needs to attach the mfxExtMVCSeqDesc structure to the mfxVideoParam structure and call the MFXVideoVPP_Init() function. The function saves the view identifiers. During video processing, oneVPL processes each view individually. oneVPL refers to the FrameID field of the mfxFrameInfo structure to configure each view according to its processing pipeline. If the video processing source frame is not the output from the oneVPL MVC decoder, then the application needs to fill the the FrameID field before calling the MFXVideoVPP_RunFrameVPPAsync() function. This is shown in the following pseudo code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
mfxExtBuffer *eb;
mfxExtMVCSeqDesc  seq_desc;
mfxVideoParam init_param;

init_param.ExtParam = &eb;
init_param.NumExtParam=1;
eb=(mfxExtBuffer *)&seq_desc;

/* init VPP */
MFXVideoVPP_Init(session, &init_param);

/* perform processing */
for (;;) {
    MFXVideoVPP_RunFrameVPPAsync(session,in,out,NULL,&syncp);
    MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
}

/* close VPP */
MFXVideoVPP_Close(session);