Inputs

An input is the data stream a policy analyses. Each input is declared once on the agent as a tap, under orb.backends.pktvisor.taps, and a policy then selects a tap by name or by tag.

Input type	Purpose
`pcap`	Live packet capture from a network interface, or a pcap file.
`flow`	sFlow and Netflow records received on a UDP port.
`dnstap`	dnstap stream from a DNS server, over a unix socket or TCP.
`netprobe`	Active probes against a list of targets: ICMP, TCP, HTTP and DNS over HTTPS.

sflow is also accepted as an input type and is handled by the same module as flow. A mock input exists for testing and is not documented here.

A tap declares the input type and its configuration; the policy that uses it may override parts of that configuration. See Input in a policy below and the policy structure for how the two fit together.

Input in a policy

A policy's input section names the data stream it analyses. It selects a tap, declares what type that tap is, and may narrow or adjust it.

Required

input_type - the type of input. It is validated against the type of the tap selected by tap or tap_selector; if they disagree, the policy fails.

tap - the name of a tap declared under orb.backends.pktvisor.taps, or tap_selector - tags to match against the declared taps. Exactly one of the two must be set, except for netprobe and flow policies, which may set neither. A selector uses either any or all, not both, and a selector that matches no tap is an error.

A netprobe or flow policy that sets neither runs on a tap the agent manages, fleet_netprobe or fleet_flow. Its settings then go in config, since the managed tap carries none. A flow policy without pcap_file must set both port and bind. Any other input type must name a tap. See Taps the agent manages and Policies from Fleet. Names starting with fleet_ are reserved, so tap cannot be set to one.

Optional

filter - what data to include from the input.

config - how the input is used. These keys are the same ones the tap accepts, documented per input type below.

A tap's configuration and the policy input's configuration are merged, with the policy input's values taking precedence for any key set in both.

Selecting a tap by name

input:
  tap: tap_name
  input_type: type_of_input
  filter:
    bpf: ...
  config:
    ...

Selecting taps by tag

Matching any of the tags:

input:
  tap_selector:
    any:
      - key1: value1
      - key2: value2
  input_type: type_of_input
  filter:
    bpf: ...
  config:
    ...

Matching all of them:

input:
  tap_selector:
    all:
      - key1: value1
      - key2: value2
  input_type: type_of_input
  filter:
    bpf: ...
  config:
    ...

A selector that matches more than one tap attaches the policy to each of them, which generates a separate set of metrics per tap. See merge_like_handlers for scraping them together.

Declaring a tap

A tap abstracts away host level details, such as the ethernet interface or the dnstap socket location, so that a policy can apply to a broad set of agents without naming them. Taps are declared under orb.backends.pktvisor.taps, and a single agent can declare as many as it needs.

A tap accepts input_type, config and tags only. It has no filter key, and one written there is ignored without an error: filters belong to the policy's input. A pcap tap can still carry a bpf expression in its config, which is where the example below puts it.

orb:
  backends:
    pktvisor:
      taps:
        first_tap_name:
          input_type: type
          config: ...
          tags:
            key1: value1
            key2: value2
        second_tap_name:
          input_type: type
          config: ...
          tags:
            key1: value1
            key3: value3

The following inputs are supported: pcap, flow, dnstap and netprobe. For each input type, specific configuration, filters and tags can be defined.

Packet Capture (pcap)

Example: Pktvisor PCAP Tap Configuration

orb:
  backends:
    pktvisor:
      taps:
        my_pcap_tap:
          input_type: pcap
          config:
            pcap_source: "libpcap"
            debug: true
            iface: auto
            host_spec: "192.168.0.1/24"
            bpf: "port 53"
          tags:
            pcap: true

Seeing the traffic

A pcap tap can only analyse what reaches its interface. For an agent that is not itself in the traffic path, that usually means mirroring the traffic of interest to the interface it listens on, or putting that interface into promiscuous mode. An agent with no visibility of the traffic produces an empty policy rather than an error.

pcap configuration

The following configurations are available for pcap inputs. bpf is both a config key and a filter, so a tap can carry it in its config where a tap cannot carry a filter.

Config	Type
pcap_file	str
pcap_source	str
iface	str
host_spec	str
debug	bool
tcp_packet_reassembly_cache_limit	int
bpf	str

pcap_file

Type: str

One option of using pktvisor is for reading existing network data files. In this case, the path to the file must be passed. This variable is dominant, so if a file is passed, pktvisor will do the entire process based on the file.

pcap_file: "path/to/file"

pcap_source

Type: str

pcap_source specifies the type of library to use. Default: libpcap. Options are libpcap, af_packet (Linux) and mock (for testing).

pcap_source: "af_packet"

iface

Type: str

Name of the interface to bind.

iface: str

Example:

iface: eth0

Tip:

You can use auto as iface. In this way, the network with the highest data flow will be analyzed.
iface: auto

host_spec

Type: str

The host_spec setting is useful to determine the direction of observed packets, once knowing the host ip, it is possible to determine the data flow direction, ie if they are being sent by the observed host (from host) or received (to host).

host_spec: str

Example:

host_spec: "192.168.0.1/24"

debug

Type: bool

When true activate debug logs. It applies to live capture only and is ignored when pcap_file is set

debug: true

tcp_packet_reassembly_cache_limit

Type: int

Sets the maximum number of TCP connections tracked for reassembly. The cache is keyed by connection, not by packet. Default value: 300000.

To remove limit set tcp_packet_reassembly_cache_limit to 0.

tcp_packet_reassembly_cache_limit: 300000

pcap filters

Filter	Type
`bpf`	str

bpf

Type: str

bpf filter data based on Berkeley Packet Filters (BPF).

bpf: str

Example:

bpf: "port 53"

sFlow/Netflow (flow)

Example: Pktvisor FLOW Tap Configuration

orb:
  backends:
    pktvisor:
      taps:
        my_flow_tap:
          input_type: flow
          config:
            port: 6343
            bind: 192.168.1.1
            flow_type: sflow
          tags:
            flow: true

Choosing a port and flow type

flow_type must match what the exporters send, and the port is whatever they are configured to send to. The conventional ports differ by format:

Datagram format	`flow_type`	Conventional port
sFlow	`sflow`	6343
Netflow	`netflow`	9996
IPFIX	`ipfix`	4739

ipfix and netflow select the same parser, so either value works for an IPFIX exporter.

bind is the local address to receive on, and both it and port must be set together. bind must be an IPv4 or IPv6 address, such as 0.0.0.0 or ::, written without brackets; a hostname such as localhost is refused. Any port from 1 to 65535 is accepted, so an exporter sending to a different one simply needs that number here. A bind and port that overlap another listener fail the input with unable to bind flow UDP server on \<bind>:\<port>: address already in use. They overlap when the port is the same and so is the address, or either side is the wildcard of the other's family: 0.0.0.0 for IPv4, :: for IPv6. Two different specific addresses can share a port. A dual-stack ::, described below, also overlaps IPv4 listeners on its port. Other bind failures end with their own reason, such as permission denied for a port below 1024 when pktvisord runs as a non-root user with --net=host; see Running as a non-root user.

Unless net.ipv6.bindv6only=1 is set where pktvisord runs, :: also receives IPv4 datagrams. With --net=host that is the host's setting; a bridge-mode container has its own, which docker run --sysctl net.ipv6.bindv6only=1 sets. A Netflow or IPFIX exporter that sends over IPv4 to a :: bind is identified by its IPv4 address, as it would be on 0.0.0.0, so key its device_map and only_device_interfaces entries by that address; entries written in IPv4-mapped form, such as ::ffff:192.0.2.10, still match. A dual-stack :: input and a 0.0.0.0 input cannot share a port, so to receive IPv4 and IPv6 exporters on one port, bind ::. With net.ipv6.bindv6only=1, the two can share it. For sFlow, the device is the agent address inside each datagram, whatever the bind.

flow configuration

The following configs are available for flow inputs. A flow input reads either from a capture file or from a UDP socket: set pcap_file, or set both port and bind. An input that sets neither fails to start.

Config	Type
pcap_file	str
port	int
bind	str
flow_type	str

pcap_file (flow)

Type: str

Reads flow records from a capture file instead of a socket. When set, port and bind are not used.

pcap_file: /path/to/flows.pcap

port (flow)

Type: int and bind: str

Receives flow records on a UDP socket. Both must be set together; only UDP bind is supported. bind is an IPv4 or IPv6 address and port is 1 to 65535; see Choosing a port and flow type.

port: int
bind: str

Example:

port: 6343
bind: 192.168.1.1

flow_type

Type: str

Default: sflow. Accepted values are sflow, netflow and ipfix.

flow_type: str

Example:

flow_type: netflow

flow filters

There are no specific filters for the FLOW input.

dnstap

Example: Pktvisor DNSTAP Tap Configuration

orb:
  backends:
    pktvisor:
      taps:
        my_dnstap_tap:
          input_type: dnstap
          config:
            socket: path/to/file.sock
          tags:
            dnstap: true

dnstap configuration

The 3 existing DNSTAP configurations (dnstap_file, socket and tcp) are mutually exclusive, that is, only one can be used in each input and one of them must exist. They are arranged in order of priority.

Config	Type
dnstap_file	str
socket	str
tcp	str

dnstap_file

Type: str

dnstap_file: path/to/file

socket

Type: str

Path of the Unix socket that pktvisor listens on, and that the DNS server sends its dnstap stream to.

socket: path/to/file.sock

When the input starts, a socket at that path that no server listens on, such as one left by a pktvisor that crashed or was killed, is replaced. The input fails to start instead, with an error that includes unable to bind dnstap server on \<path>: followed by the reason, and leaves the file in place, when:

a server already listens on the socket: address already in use;
the path exists and is not a socket: the path exists and is not a socket;
no server listens on the socket, but pktvisor cannot remove it: unable to remove the stale socket there, followed by the reason;
pktvisor cannot tell whether a server listens on it: unable to tell whether a server listens on the socket there, followed by the reason. A socket left by a pktvisor that ran as root is one case, because a non-root pktvisor cannot connect to it. Remove the file if nothing uses it.

A path whose directory does not exist fails with permission denied, not with a missing-file error.

tcp

Type: str

The other way to inform the ip and port to be monitored is through the 'tcp' configuration. Usage syntax is a string with ip:port (only ipv4 is supported for now). An address and port that overlap another listener, as for flow, fail the input with unable to listen with dnstap server on <ip>:<port>: address already in use.

tcp: ip:port

Example:

tcp: 192.168.8.2:235

dnstap filters

Filter	Type
`only_hosts`	str[]

only_hosts

Type: str[]

only_hosts filters data to the given hosts. It is read as a list, so a bare string is rejected. Like every filter it is set on the policy's input.filter, not on the tap.

input:
  input_type: dnstap
  tap: my_dnstap_tap
  filter:
    only_hosts:
      - 192.0.2.4/32
      - 198.51.100.0/24

Netprobe

The netprobe input actively probes a list of targets rather than observing traffic, and it has enough settings of its own, differing by test type, to carry its own page.

Example: Pktvisor Netprobe Tap Configuration

orb:
  backends:
    pktvisor:
      taps:
        default_netprobe:
          input_type: netprobe
          config:
            test_type: ping
            interval_msec: 2000
            timeout_msec: 1000
            packets_per_test: 10
            packets_interval_msec: 25
            packet_payload_size: 56
            targets:
              primary_site:
                target: www.example.com
              secondary_site:
                target: www.example.net
          tags:
            netprobe: true

See Netprobe input for the test types (ping, tcp, http, doh), the per-target keys, the timing settings and the HTTP response checks.

On this page