The GStreamer VA-API Report

Víctor Jáquez

<vjaquez@igalia.com> irc:ceyusa

GStreamer Conference 2017

21-22 October (Prague)

Agenda

• What's VA-API?
• What's new?
• What's next?

What is VA-API?

An API specification

A library implementation

https://github.com/01org/libva

Back-ends

Intel driver

https://github.com/01org/intel-vaapi-driver

Mesa drivers

Others (?)

Intel driver on Kayblake

$ vainfo
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.0 (libva 2.0.1.pre1)
vainfo: Driver version: Intel i965 driver for Intel(R) Kaby Lake - 2.0.1.pre1 (1.8.3-156-g37613a5)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileH264MultiviewHigh      : VAEntrypointVLD
      VAProfileH264MultiviewHigh      : VAEntrypointEncSlice
      VAProfileH264StereoHigh         : VAEntrypointVLD
      VAProfileH264StereoHigh         : VAEntrypointEncSlice
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSlice
      VAProfileVP9Profile2            : VAEntrypointVLD

GStreamer-VAAPI

One Year (more or less)

20 contributors

411 commits

~119 bugs

Releases

13 + next stable + master

• 1.12.3
• 1.12.2
• 1.12.1
• 1.12.0
• 1.11.91
• 1.11.90
• 1.11.2
• 1.11.1
• 1.10.5
• 1.10.4
• 1.10.3
• 1.10.2
• 1.10.1

Contributors

Thank you all

• Dominique Leuenberger (1)
• Edward Hervey (1)
• Hyunjun Ko (82)
• Jagyum Koo (1)
• Jan Schmidt (4)
• Julien Isorce (17)
• Matt Staples (1)
• Nicolas Dufresne (1)
• Orestis Floros (9)
• Scott D Phillips (6)
• Sebastian Dröge (12)
• Sreerenj Balachandran (23)
• Thomas Petazzoni (2)
• Thibault Saunier (1)
• Tim-Philipp Müller (8)
• Tomas Rataj (1)
• Ullysses Artie Eoff (2)
• Victor Toso (1)
• Yi A Wang (2)

What's new?

GstObject-ification

GstVaapiDisplay ✓

GstVaapi{Filter, Decoder, Encoder, …}

(leak tracer for the worth!)

Register properties at class initialization

Direct Rendering and Upload

Use vaDeriveImage when possible

(problems with Mesa backend)

exporting environment variable GST_VAAPI_ENABLE_DIRECT_RENDERING

DMABuf

Improved of DMABuf's usage both upstream and dowstream

Negotiate memory:DMABuf caps feature

VA Display initialization

GstGL decides the type of GstVaapiDisplay instance

Fixed initialization for headless systems

VA Display sharing

Improved GstVaapiDisplay spread among pipeline

External VA display via gst.vaapi.app.Display context

VA Display cache

Removed!

Build

Support meson

Support libva 2.0 (VA-API 1.0)

Coverity

Fixed all detected issues by static analyzer

GstValidate

Normally™ passes all the gst-validate tests

Misc

Redirect libva's messages into GStreamer logs

Decoders

Faster upstream re-negotiation

low-latency mode for H.264 decoding (flush DPB for streaming)

Google Summer of Code 2017

Drops MVC & SVC frames when

back-end doesn't support it

base-only property is on

vaapidecodebin

vaapipostproc is plugged "dynamically"

You can control it by exporting the environment variable GST_VAAPI_DISABLE_VPP

Encoders

Query possible upload formats

Merge Gst Tags for downstream info

Support upstream reconfiguration

Encoders

Profile in caps negotiation (H.264 & H.265)

Handle Regions-Of-Interest (H.264)

Support H.265 main-10 profile

Encoders

Constant bit-rate mode VP8 & VP9

Variable bit-rate mode H.265

Added quality level parameter

Encoders

AU delimeter in H.264 (streaming)

H.264 macro-block bit-rate control

H.264 & H.265 multi-reference

H.264 & H.265 QP setting either for I/P/B

Sink

Automatic color transformation if VPP available

Revamp Wayland backend (Weston)

Flexible Encoding Infrastructure

It's an extension to VA-API

Allows more control over different phases of encoding

Trade off quality for speed with their on IPs.

https://bugzilla.gnome.org/show_bug.cgi?id=784667

Flexible Encoding Infrastructure

Split the encoding in two phases

1. ENC motion vector calculation and prediction
2. PAK transformations and entropy coding

Flexible Encoding Infrastructure

Without FEI:

ENC+PAK is a black box to middle-ware,

With FEI:

Extract the output of ENC and feed PAK

(custom motion vectors and macro-block prediction modes)

Flexible Encoding Infrastructure

vaapih264feienc

H.264 encoder provides low-level per macroblock tuning parameters

vaapih264feipreenc (TBI Shall we?)

Pre-processing element which can extract MVs and statistics before calling encode function

Flexible Encoding Infrastructure

Highly experimental & unstable

Skylake only

H.264 only

What's next?

libva's pipeline

FEI

DMABuf

Multi frame processing (?)

Encoding statistics

bugzilla's pipeline

Ton of pending tasks and bugs

Questions?