Snabb

A toolkit for building user-space network functions

About Igalia

  • Consultancy specialized in open-source
  • Base in Coruña but distributed all over the world (>60 people working from 15 different countries)
  • Contributors to projects such as WebKit, Chromium V8, etc
  • Other areas: Graphics, Multimedia, Networking

https://www.igalia.com/networking

Agenda

  • What is Snabb?
  • How it works?
  • Catalog and programs
  • Use case: lwAFTR

What is Snabb?

Snabb is a toolkit for developing high-performance network functions in user-space

What is a Network Function?

  • A program that manipulates traffic data
  • Basic operations: read, forward, drop, modify, create...
  • Combining these primitives we can build any network function

Examples

  • Firewall: read incoming packets, compare to table of rules and execute an action(forward or drop)
  • NAT: read incoming packets, modify headers and forward packet
  • Tunelling: read incoming packets, create a new packet, embed packet into new one and send it

Why Snabb?

  • Increasing improvement of commodity hardware: 10Gbps NICs at very affordable prices
  • High-performance equipment is still very expensive
  • Idea: build an analog high-performance router using commodity hardware

Why Snabb?

  • What software to put into this hardware?
  • Common intuition: Linux
  • Drawback: Linux is not suitable for high-performance networking

Why not Linux?

  • General-purpose operating system
  • An OS abstracts hw resources to offer high-level interfaces: filesystems, processes, sockets...
  • Our network function will be divided into two lands: user-space and kernel-space
  • Colorary: processing a packet has an inheritent cost => the cost of the OS

High-performance networking

  • NIC: 10Gbps
  • Avg Packet-size: 550-byte
  • PPS: 2272727,27
  • 1 packet every 440ns ((1/2272727,27)*10^9)
  • CPU: 2,5 Ghz
  • 1100 cycles to process one packet (2,5 cycles/sec * 440 ns)

High-performance networking

  • Packet-size: 64-byte: 51 ns per packet; 128 cycles per packet
  • Lock/Unlock: 16ns; Cache-miss: 32 ns
  • Source: Jonathan Corbet's "Improving Linux networking performance"
  • Small packet size => More packets per second => worse
  • Faster CPU => better

User-space driver

  • Do a kernel by-pass and manage the hardware directly from user-space:
    • Tell Linux not to manage the PCI device (unbind)
    • Do a mmap of the registers of the PCI device into addressable memory
    • Whenever we read/write the addressable memory, we're actually poking the registers of the NIC
    • Follow the NIC's datasheet to implement operations such as initialize, receive, transmit, etc

User-space networking

  • Snabb is not an isolated case of user-space networking:
    • Snabb (2012)
    • DPDK (2012)
    • VPP/fd.io (2016)
  • DPDK (Data-plane Development Kit, Intel)
  • VPP (Vector Packet Processing, Cisco)

Ring-buffer

Very important to avoid packet drops

Inside Snabb

Snabb

  • Project started by Luke Gorrie
  • User-space networking benefit: freedom of programming language
  • Snabb is mostly written in Lua
  • Network functions are also written in Lua
  • Fast to run, fast to develop
  • Snabb means fast in Swedish :)

About Lua

  • Started in 1993 at University of Rio de Janeiro (PUC Rio)
  • Very similar to JavaScript, easy to learn
  • Very small and compact, it's generally embeded in other systems
  • Use cases: microcontrollers (NodeMCU), videogames (Grim Fandango), IA (Torch7)

About LuaJIT

  • Just-in-time compiler for Lua
  • Extremely fast virtual machine!!
  • Very good integration with C thanks to FFI (Foreign Function Interface)

FFI: example


ffi.cdef[[
  void syslog(int priority, const char*format, ...);
]]
ffi.C.syslog(2, "error:...");
          

local ether_header_t = ffi.typeof [[
/* All values in network byte order.  */
struct {
   uint8_t  dhost[6];
   uint8_t  shost[6];
   uint16_t type;
} __attribute__((packed))
]]
          

Snabb in a nutshell

  • A snabb program is an app graph
  • Apps are conected together via links
  • Snabb processes the program in units called breadths

NF: App graph

Breadths

  • A breadth has two steps:
    • inhale a batch of packets into the graph
    • process those packets
  • To inhale, the method pull of the apps is executed (if defined)
  • To process, the method push of the apps is executed (if defined)

  # Pull function of included Intel 82599 driver
  function Intel82599:pull ()
    for i = 1, engine.pull_npackets do
      if not self.dev:can_receive() then break end
      local pkt = self.dev:receive()
      link.transmit(self.output.tx, pkt)
    end
  end

    # Push function of included PcapFilter
    function PcapFilter:push ()
      while not link.empty(self.input.rx) do
        local p = link.receive(self.input.rx)
        if self.accept_fn(p.data, p.length) then
          link.transmit(self.output.tx, p)
        else
          packet.free(p)
        end
      end
    end

Packet processing

  • Normally only one app of the app graph introduces packets into the graph
  • The method push gives an opportunity to every app to do something with a packet

App graph definition


    local c = config.new()

    -- App definition.
    config.add(c, "nic", Intel82599, {
      pci = "0000:04:00.0"
    })
    config.add(c, "filter", PcapFilter, "src port 80")
    config.add(c, "writer", Pcap.PcapWriter, "output.pcap")

    -- Link definition.
    config.link(c, "nic.tx        -> filter.input")
    config.link(c, "filter.output -> writer.input")

    engine.configure(c)
    engine.main({duration=1})
  

Packets


              struct packet {
                uint16_t length;
                unsigned char data[10*1024];
              };
            

Links


              struct link {
                struct packet *packets[1024];
                // the next element to be read
                int read;
                // the next element to be written
                int write;
              };
            

Snabb: App catalog and programs

Inventary

  • apps: software components that developers combine together to build network functions
  • programs: complete network functions

Apps I/O

  • Intel i210/i350/82599/XL710
  • Mellanox Connectx-4/5
  • Virtio host y guest
  • UNIX socket
  • Linux: tap and "raw" (e.g: eth0)
  • Pcap files

Apps L2

  • Flooding and learning bridge
  • VLAN insert/remove
  • ARP/NDP

Apps L3

  • IPv4/v6 fragmentation and reassembly
  • IPv4/v6 splitter
  • ICMPv4/v6 echo responder
  • Control-plane delegation (nh_fwd)

Apps L4

  • IPsec ESP
  • Lightweight 4-over-6 AFTR
  • Keyed IPv6 Tunnel

Apps monitoring

  • IPFix capturer and exporter
  • L7 monitor/filtering (libndpi)
  • Pcap expressions filter (with own backend for code generation)

Apps testing

  • Lots of load generators: loadgen, packetblaster, loadbench...

Use case: lwAFTR

Context

  • 2012-2014: Several RIRs run out of IPv4 public addresses
  • 2008: IPv6 adoption starts to peak up
  • Still big dependency on IPv4: services, websites, programs, etc

Solutions

  • Carrier-Grade NAT: temporal solution for IPv4 address exhaustion problem
  • Deployment of Dual-Stack networks (IPv4 e IPv6)
  • Dual-Stack implies increasing complexity and costs (maintenance of two separated networks)
  • Dual-Stack Lite (IPv6-only network which also offers IPv4 connectivity relying on CGN)
  • Lightweight 4over6: iteration over Dual-Stack

Lightweight 4over6

Lw4o6 - Goals

  • RFC7596 fully complaint (lwAFTR part)
  • Performance: 2MPPS; 550-byte (packet-size); Binding-table: 1M subscribers.
  • No packet drops

Lw4o6 - Development

  • Version 1:
    • Prototype
    • Basic functionality (encapsulating/decapsulating)
    • Small binding-table (own format)
    • Development of tools to measure performance

Lw4o6 - Development

  • Version 2
    • Production quality
    • Fully standard compliant
    • Big binding-table: 1M subscribers (still customized format but much closer to standard)
    • Add support for other necessary protocols: ARP, NDP, fragmentation, reassembly, ping
    • Tons of optimizations (use of AVX instructions to speed up lookups)

Lw4o6 - Development

  • Version 3:
    • Added YANG support to Snabb
    • Support binding-table format according to standard
    • Support of execution as leader/worker (leader: control-plane/worker: data-plane)

Lw4o6 - Development

  • Version 4:
    • Multiprocess (one leader, multiple workers)
    • Improvement of the Intel 10Gbps driver (added support for RSS, Received Side Scaling)
    • Added alarms support according to latest draft

Lightweight 4over6 - Talks

Other programs

Program: packet blaster

  • Generally useful tool: fill TX buffer of NIC with packets and transmit them over and over again
  • Measures received traffic too
  • Easily saturates 10Gbps links
  • 
    	snabb packetblaster replay packets.pcap 82:00.1
    

Program: Snabbwall

  • L7 firewall that optionally uses nDPI
  • Collaboration betwen Igalia and NLnet Foundation
  • Landed upstream in 2017
  • Website: http://snabbwall.org

Program: IPFix

  • NETFLOW collector and exporter (v9 and IPFIX)
  • Line-rate speed on a single core. Further improvement: parallel processing via RSS
  • Landed upstream very recently

Program: l2VPN

  • L2VPN over IPv6 (developed by Alexander Gall from SWITCH)
  • Pending to land upstream; used in production
  • Ideal Snabb use case: programmer/operator builds bespoke tool

Program: Your VNF

  • Snabb upstream open to include new network functions
  • Repository will grow as people will build new things
  • Igalia can build one for you

Last notes about performance

Considerations

  • Isolcpus: Prevents the kernel to take a CPU to schedule processes
  • Dishable HyperThreading
  • Use HugePages (2MB) (Linux default is 4Kb)
  • Do not neglect NUMA when launching programs
  • Make use of SIMD instructions (AVX, AVX2) to speed up computations (checksum)
  • Keep an eye on regressions: profile often

Summary

  • Toolkit for developing high-performance network functions in user-space
  • Snabb provides apps which can be combined together forming a graph (network function)
  • Snabb provides programs, complete network functions ready to use
  • Snabb provides libraries, to easy the development of new network functions
  • Completely written in Lua: easy to extend
  • Fast: kernel-by pass + high-level language + fast VM (LuaJIT)

Thanks!

  • Email: dpino@igalia.com
  • Twitter: @diepg


  $ git clone https://github.com/snabbco/snabb.git
  $ cd snabb
  $ make