Who is this dude?

  • Born in the Basque Country
  • Grown and living in Galicia
    • One of the seven Celtic Nations, like Scotland
  • Studied Computer Science at Universidade da Coruña
  • Partner and developer at Igalia
    • Working in Multimedia and WebKit

From html5demos.com

Index

  • Quick review of WebKit
  • WebKit Multimedia
  • Power features
    • WebAudio
    • Hardware acceleration
    • Accelerated video compositing

Quick Review

WebKit

  • Web content engine
  • Processes and renders web content
  • Started as a fork of KTHML and KJS
  • Open source since 2005
  • Goals: open source, compatibility, compliance, stability, performance, security, portability, usability, hackability
  • Non-goals: being a full web browser, being a science project, having reusable components, unlimited scope
  • Forked by Google as Blink
  • Split into ports: GTK+, Qt, EFL, Mac, Chromium, Windows

Architecture

WebKitGTK+

  • Each WebKit port is composed of
    • Platform interfaces
    • API layer
  • WebKitGTK+ platform layer:
    • libsoup for networking
    • cairo for rasterization
    • OpenGL for making the scene graph and WebGL
    • GStreamer for media
    • Various GTK+ APIs for talking with the system
  • API layer is a GtkWidget and a set of GObject APIs
  • WebKitGTK+ is used by Epiphany, Midori, yelp, devhelp

WebKit2

  • Give the web rendering parts of WebKit their own process
    • Page crashes don't crash the browser
    • Can put vulnerable data into a separate address space
  • Sandbox web rendering
    • Prevent pages from accessing the disk and operating system interface

WebKit2 Architecture

The Near Future

WebKit Multimedia

WebKit Multimedia

  • Mac port: QuickTime, QtKit, AVFoundations
  • Apple Win port: QuickTime
  • GTK+ and EFL ports: GStreamer 1.0 backend
  • Blackberry port: Bridges over to its own Media playback framework
  • Chromium: bridges over to WebKit/Chromium backend which relays to FFMPeg implementations
  • Qt port: GStreamer on Linux, QtKit on Mac, QuickTime on Windows

WebKit Multimedia Architecture

GStreamer MediaPlayer - Overview

  • Mainly developed and tested on Linux platform
  • Successfully ported to MacOS/Win platfoms (not fully upstream yet though)
  • Playbin-based: the best GStreamer element for playback use-cases
    • Playbin2 with GStreamer 0.10
  • MediaPlayer registers and uses two WebKit-specific GStreamer elements: webkitwebsrc and a video-sink

GStreamer MediaPlayer - Main Features

  • On-disk-buffering
  • Frame-accurate seeking
  • Basic metrics reporting
  • Trick-modes
  • WebAudio
  • Hardware decoding support with VA-API (gst 1.2.x)
  • Video accelerated compositing
  • Codec installer support

WebKit and GStreamer Architecture

GStreamer MediaPlayer - HTTP source element

  • a GStreamer bin implementing the URI handler interface for http(s):// and blob:// uris.
  • A single source pad provides raw data access to downstream elements.
  • appsrc is used internally to interface with the WebCore SubResourceLoader object
  • Element is registered with high rank in the GStreamer registry so playbin is more prone to pick it up to read http:// URIs.

GStreamer MediaPlayer - Video rendering

  • Element inheriting from GstVideoSink
    • In practical sense, it is a GstAppSink with video capabilities negotiation
  • Takes video GstBuffers as input
  • Pass those to the MediaPlayerPrivateGStreamer via a signal

GStreamer MediaPlayer - Audio rendering

  • Use autoaudiosink
  • Check for pulseaudio for playback request messages
  • Pitch preservation
  • Preserve audio passthrough when possible
  • Volume managed by the sink

GStreamer MediaPlayer - Next steps

  • Improvements and bug fixes in the player:
    • Store on-disk buffered media in WebCore's cache directory
    • Make the webkitwebsrc element private to WebKit
  • MediaStream (getUserMedia) and WebRTC spec implementation
  • MediaSource adaptive streaming API spec implementation
  • Integration of WebAudio with media elements and mediastream stack
  • Finish porting to 1.2
  • Remove GStreamer 0.10 code path

WebKit Multimedia Power Features

WebAudio

  • A JavaScript API allowing to create a new generation of Web games and applications
  • Effects, convolution engine for room simulation, etc
  • Realtime analysis, playback and visualization
  • W3C spec driven and implemented mainly by Chromium and Firefox
  • In WebKit, currently implemented for:
    • Mac
    • Chromium
    • GStreamer based Ports: GTK+, EFL, Qt

WebAudio GStreamer - Reading Audio

  • Spec allows two ways to read audio data:
    • from memory: we can use giostreamsrc GStreamer element
    • from a file path: we can use the filesrc GStremer element
  • Pipeline: src ! decodebin2 ! audioconvert ! audioresample ! capsfilter ! deinterleave ! queue ! appsink
  • deinterleave separates multi-channel audio to n audio mono channels (Pipeline above can contain multiple queue ! appsink chains)
  • appsink used to collect mono audio buffers, store them in a list and convert data to floating point values
  • Data is stored in a WebCore::AudioBus, containing one or multiple channel Float arrays

WebAudio GStreamer - Playing Audio

  • The WebAudio stack processed or generated some audio data to be sent to the sound card
  • Similarly to the "Reading Data" step, a WebCore::AudioBus is used to store the data
  • Custom GStreamer element to convert an AudioBus to an interleaved audio data flow: webkitwebaudiosrc
  • webkitwebaudiosrc: a bin with n internal pads pushing to an interleave element. Source pad outputs a single, interleave, wav-encoded data flow
  • Pipeline: webkitwebaudiosrc ! wavparse ! audioconvert ! autoaudiosink

WebAudio GStreamer - Work in Progress

  • Some layout tests don't pass yet
  • Not yet enabled by default in release builds
  • Support for stereo+ streams in the reader side. Playback side already handles those.
  • Integration with <video> and <audio> elements: WebAudio MediaElementSourceNode

Hardware Accelerated Video

  • We use GStreamer VA-API. (1.2.x)
  • That's it! Hardcore comes when compositing that.

Accelerated Video Compositing - Acceleration in WebKit

Accelerated Video Compositing

  • Element inheriting from GstVideoSink
    • In practical sense, it is a GstAppSink with video capabilities negotiation
  • Takes video GstBuffers as input.
  • Video frame is transferred to the MediaPlayerPrivateGStreamer via a GLib signal.
  • MediaPlayerPrivateGStreamer uploads it to a GL texture and renders it asynchronously in the WebKit context.

Accelerated Video Compositing - Architecture

Accelerated Video Compositing - Rendering paths

  • Software rendering path
    • It uses Cairo
  • Accelerated compositing paths
    • TextureMapperLayer:
      • Light-weight implementation of a scene-graph
      • The TextureMapperLayer tree is equivalent to the GraphicsLayer tree.
    • TextureMapperGL
      • Provides a GPU accelerated implementation of the drawing primitives, using shaders compatible with GL/ES 2.0

Accelerated Video Compositing - Next steps

  • More video decoders with GstVideoGLTextureUploadMeta support
  • Support EGLImage in WebKit as video frames handlers
    • And video decoders that deliver this kind of capabilities.
  • Handle the vertical blank interval in WebKit
    • GTK+ 3.8 seems to fix this issue.

From html5demos.com

Thank You!

(q&a)