0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 ====================================
0004 Virtual Routing and Forwarding (VRF)
0005 ====================================
0006
0007 The VRF Device
0008 ==============
0009
0010 The VRF device combined with ip rules provides the ability to create virtual
0011 routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
0012 Linux network stack. One use case is the multi-tenancy problem where each
0013 tenant has their own unique routing tables and in the very least need
0014 different default gateways.
0015
0016 Processes can be "VRF aware" by binding a socket to the VRF device. Packets
0017 through the socket then use the routing table associated with the VRF
0018 device. An important feature of the VRF device implementation is that it
0019 impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
0020 (ie., they do not need to be run in each VRF). The design also allows
0021 the use of higher priority ip rules (Policy Based Routing, PBR) to take
0022 precedence over the VRF device rules directing specific traffic as desired.
0023
0024 In addition, VRF devices allow VRFs to be nested within namespaces. For
0025 example network namespaces provide separation of network interfaces at the
0026 device layer, VLANs on the interfaces within a namespace provide L2 separation
0027 and then VRF devices provide L3 separation.
0028
0029 Design
0030 ------
0031 A VRF device is created with an associated route table. Network interfaces
0032 are then enslaved to a VRF device::
0033
0034 +-----------------------------+
0035 | vrf-blue | ===> route table 10
0036 +-----------------------------+
0037 | | |
0038 +------+ +------+ +-------------+
0039 | eth1 | | eth2 | ... | bond1 |
0040 +------+ +------+ +-------------+
0041 | |
0042 +------+ +------+
0043 | eth8 | | eth9 |
0044 +------+ +------+
0045
0046 Packets received on an enslaved device and are switched to the VRF device
0047 in the IPv4 and IPv6 processing stacks giving the impression that packets
0048 flow through the VRF device. Similarly on egress routing rules are used to
0049 send packets to the VRF device driver before getting sent out the actual
0050 interface. This allows tcpdump on a VRF device to capture all packets into
0051 and out of the VRF as a whole\ [1]_. Similarly, netfilter\ [2]_ and tc rules
0052 can be applied using the VRF device to specify rules that apply to the VRF
0053 domain as a whole.
0054
0055 .. [1] Packets in the forwarded state do not flow through the device, so those
0056 packets are not seen by tcpdump. Will revisit this limitation in a
0057 future release.
0058
0059 .. [2] Iptables on ingress supports PREROUTING with skb->dev set to the real
0060 ingress device and both INPUT and PREROUTING rules with skb->dev set to
0061 the VRF device. For egress POSTROUTING and OUTPUT rules can be written
0062 using either the VRF device or real egress device.
0063
0064 Setup
0065 -----
0066 1. VRF device is created with an association to a FIB table.
0067 e.g,::
0068
0069 ip link add vrf-blue type vrf table 10
0070 ip link set dev vrf-blue up
0071
0072 2. An l3mdev FIB rule directs lookups to the table associated with the device.
0073 A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
0074 l3mdev rule for IPv4 and IPv6 when the first device is created with a
0075 default preference of 1000. Users may delete the rule if desired and add
0076 with a different priority or install per-VRF rules.
0077
0078 Prior to the v4.8 kernel iif and oif rules are needed for each VRF device::
0079
0080 ip ru add oif vrf-blue table 10
0081 ip ru add iif vrf-blue table 10
0082
0083 3. Set the default route for the table (and hence default route for the VRF)::
0084
0085 ip route add table 10 unreachable default metric 4278198272
0086
0087 This high metric value ensures that the default unreachable route can
0088 be overridden by a routing protocol suite. FRRouting interprets
0089 kernel metrics as a combined admin distance (upper byte) and priority
0090 (lower 3 bytes). Thus the above metric translates to [255/8192].
0091
0092 4. Enslave L3 interfaces to a VRF device::
0093
0094 ip link set dev eth1 master vrf-blue
0095
0096 Local and connected routes for enslaved devices are automatically moved to
0097 the table associated with VRF device. Any additional routes depending on
0098 the enslaved device are dropped and will need to be reinserted to the VRF
0099 FIB table following the enslavement.
0100
0101 The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
0102 addresses as VRF enslavement changes::
0103
0104 sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
0105
0106 5. Additional VRF routes are added to associated table::
0107
0108 ip route add table 10 ...
0109
0110
0111 Applications
0112 ------------
0113 Applications that are to work within a VRF need to bind their socket to the
0114 VRF device::
0115
0116 setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
0117
0118 or to specify the output device using cmsg and IP_PKTINFO.
0119
0120 By default the scope of the port bindings for unbound sockets is
0121 limited to the default VRF. That is, it will not be matched by packets
0122 arriving on interfaces enslaved to an l3mdev and processes may bind to
0123 the same port if they bind to an l3mdev.
0124
0125 TCP & UDP services running in the default VRF context (ie., not bound
0126 to any VRF device) can work across all VRF domains by enabling the
0127 tcp_l3mdev_accept and udp_l3mdev_accept sysctl options::
0128
0129 sysctl -w net.ipv4.tcp_l3mdev_accept=1
0130 sysctl -w net.ipv4.udp_l3mdev_accept=1
0131
0132 These options are disabled by default so that a socket in a VRF is only
0133 selected for packets in that VRF. There is a similar option for RAW
0134 sockets, which is enabled by default for reasons of backwards compatibility.
0135 This is so as to specify the output device with cmsg and IP_PKTINFO, but
0136 using a socket not bound to the corresponding VRF. This allows e.g. older ping
0137 implementations to be run with specifying the device but without executing it
0138 in the VRF. This option can be disabled so that packets received in a VRF
0139 context are only handled by a raw socket bound to the VRF, and packets in the
0140 default VRF are only handled by a socket not bound to any VRF::
0141
0142 sysctl -w net.ipv4.raw_l3mdev_accept=0
0143
0144 netfilter rules on the VRF device can be used to limit access to services
0145 running in the default VRF context as well.
0146
0147 Using VRF-aware applications (applications which simultaneously create sockets
0148 outside and inside VRFs) in conjunction with ``net.ipv4.tcp_l3mdev_accept=1``
0149 is possible but may lead to problems in some situations. With that sysctl
0150 value, it is unspecified which listening socket will be selected to handle
0151 connections for VRF traffic; ie. either a socket bound to the VRF or an unbound
0152 socket may be used to accept new connections from a VRF. This somewhat
0153 unexpected behavior can lead to problems if sockets are configured with extra
0154 options (ex. TCP MD5 keys) with the expectation that VRF traffic will
0155 exclusively be handled by sockets bound to VRFs, as would be the case with
0156 ``net.ipv4.tcp_l3mdev_accept=0``. Finally and as a reminder, regardless of
0157 which listening socket is selected, established sockets will be created in the
0158 VRF based on the ingress interface, as documented earlier.
0159
0160 --------------------------------------------------------------------------------
0161
0162 Using iproute2 for VRFs
0163 =======================
0164 iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
0165 section lists both commands where appropriate -- with the vrf keyword and the
0166 older form without it.
0167
0168 1. Create a VRF
0169
0170 To instantiate a VRF device and associate it with a table::
0171
0172 $ ip link add dev NAME type vrf table ID
0173
0174 As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
0175 covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
0176 device create.
0177
0178 2. List VRFs
0179
0180 To list VRFs that have been created::
0181
0182 $ ip [-d] link show type vrf
0183 NOTE: The -d option is needed to show the table id
0184
0185 For example::
0186
0187 $ ip -d link show type vrf
0188 11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
0189 link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
0190 vrf table 1 addrgenmode eui64
0191 12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
0192 link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
0193 vrf table 10 addrgenmode eui64
0194 13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
0195 link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
0196 vrf table 66 addrgenmode eui64
0197 14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
0198 link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
0199 vrf table 81 addrgenmode eui64
0200
0201
0202 Or in brief output::
0203
0204 $ ip -br link show type vrf
0205 mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
0206 red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
0207 blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
0208 green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
0209
0210
0211 3. Assign a Network Interface to a VRF
0212
0213 Network interfaces are assigned to a VRF by enslaving the netdevice to a
0214 VRF device::
0215
0216 $ ip link set dev NAME master NAME
0217
0218 On enslavement connected and local routes are automatically moved to the
0219 table associated with the VRF device.
0220
0221 For example::
0222
0223 $ ip link set dev eth0 master mgmt
0224
0225
0226 4. Show Devices Assigned to a VRF
0227
0228 To show devices that have been assigned to a specific VRF add the master
0229 option to the ip command::
0230
0231 $ ip link show vrf NAME
0232 $ ip link show master NAME
0233
0234 For example::
0235
0236 $ ip link show vrf red
0237 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
0238 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
0239 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
0240 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
0241 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
0242 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
0243
0244
0245 Or using the brief output::
0246
0247 $ ip -br link show vrf red
0248 eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
0249 eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
0250 eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST>
0251
0252
0253 5. Show Neighbor Entries for a VRF
0254
0255 To list neighbor entries associated with devices enslaved to a VRF device
0256 add the master option to the ip command::
0257
0258 $ ip [-6] neigh show vrf NAME
0259 $ ip [-6] neigh show master NAME
0260
0261 For example::
0262
0263 $ ip neigh show vrf red
0264 10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
0265 10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
0266
0267 $ ip -6 neigh show vrf red
0268 2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
0269
0270
0271 6. Show Addresses for a VRF
0272
0273 To show addresses for interfaces associated with a VRF add the master
0274 option to the ip command::
0275
0276 $ ip addr show vrf NAME
0277 $ ip addr show master NAME
0278
0279 For example::
0280
0281 $ ip addr show vrf red
0282 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
0283 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
0284 inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
0285 valid_lft forever preferred_lft forever
0286 inet6 2002:1::2/120 scope global
0287 valid_lft forever preferred_lft forever
0288 inet6 fe80::ff:fe00:202/64 scope link
0289 valid_lft forever preferred_lft forever
0290 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
0291 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
0292 inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
0293 valid_lft forever preferred_lft forever
0294 inet6 2002:2::2/120 scope global
0295 valid_lft forever preferred_lft forever
0296 inet6 fe80::ff:fe00:203/64 scope link
0297 valid_lft forever preferred_lft forever
0298 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
0299 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
0300
0301 Or in brief format::
0302
0303 $ ip -br addr show vrf red
0304 eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
0305 eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
0306 eth5 DOWN
0307
0308
0309 7. Show Routes for a VRF
0310
0311 To show routes for a VRF use the ip command to display the table associated
0312 with the VRF device::
0313
0314 $ ip [-6] route show vrf NAME
0315 $ ip [-6] route show table ID
0316
0317 For example::
0318
0319 $ ip route show vrf red
0320 unreachable default metric 4278198272
0321 broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2
0322 10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2
0323 local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2
0324 broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2
0325 broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2
0326 10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2
0327 local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2
0328 broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2
0329
0330 $ ip -6 route show vrf red
0331 local 2002:1:: dev lo proto none metric 0 pref medium
0332 local 2002:1::2 dev lo proto none metric 0 pref medium
0333 2002:1::/120 dev eth1 proto kernel metric 256 pref medium
0334 local 2002:2:: dev lo proto none metric 0 pref medium
0335 local 2002:2::2 dev lo proto none metric 0 pref medium
0336 2002:2::/120 dev eth2 proto kernel metric 256 pref medium
0337 local fe80:: dev lo proto none metric 0 pref medium
0338 local fe80:: dev lo proto none metric 0 pref medium
0339 local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium
0340 local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium
0341 fe80::/64 dev eth1 proto kernel metric 256 pref medium
0342 fe80::/64 dev eth2 proto kernel metric 256 pref medium
0343 ff00::/8 dev red metric 256 pref medium
0344 ff00::/8 dev eth1 metric 256 pref medium
0345 ff00::/8 dev eth2 metric 256 pref medium
0346 unreachable default dev lo metric 4278198272 error -101 pref medium
0347
0348 8. Route Lookup for a VRF
0349
0350 A test route lookup can be done for a VRF::
0351
0352 $ ip [-6] route get vrf NAME ADDRESS
0353 $ ip [-6] route get oif NAME ADDRESS
0354
0355 For example::
0356
0357 $ ip route get 10.2.1.40 vrf red
0358 10.2.1.40 dev eth1 table red src 10.2.1.2
0359 cache
0360
0361 $ ip -6 route get 2002:1::32 vrf red
0362 2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium
0363
0364
0365 9. Removing Network Interface from a VRF
0366
0367 Network interfaces are removed from a VRF by breaking the enslavement to
0368 the VRF device::
0369
0370 $ ip link set dev NAME nomaster
0371
0372 Connected routes are moved back to the default table and local entries are
0373 moved to the local table.
0374
0375 For example::
0376
0377 $ ip link set dev eth0 nomaster
0378
0379 --------------------------------------------------------------------------------
0380
0381 Commands used in this example::
0382
0383 cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
0384 1 mgmt
0385 10 red
0386 66 blue
0387 81 green
0388 EOF
0389
0390 function vrf_create
0391 {
0392 VRF=$1
0393 TBID=$2
0394
0395 # create VRF device
0396 ip link add ${VRF} type vrf table ${TBID}
0397
0398 if [ "${VRF}" != "mgmt" ]; then
0399 ip route add table ${TBID} unreachable default metric 4278198272
0400 fi
0401 ip link set dev ${VRF} up
0402 }
0403
0404 vrf_create mgmt 1
0405 ip link set dev eth0 master mgmt
0406
0407 vrf_create red 10
0408 ip link set dev eth1 master red
0409 ip link set dev eth2 master red
0410 ip link set dev eth5 master red
0411
0412 vrf_create blue 66
0413 ip link set dev eth3 master blue
0414
0415 vrf_create green 81
0416 ip link set dev eth4 master green
0417
0418
0419 Interface addresses from /etc/network/interfaces:
0420 auto eth0
0421 iface eth0 inet static
0422 address 10.0.0.2
0423 netmask 255.255.255.0
0424 gateway 10.0.0.254
0425
0426 iface eth0 inet6 static
0427 address 2000:1::2
0428 netmask 120
0429
0430 auto eth1
0431 iface eth1 inet static
0432 address 10.2.1.2
0433 netmask 255.255.255.0
0434
0435 iface eth1 inet6 static
0436 address 2002:1::2
0437 netmask 120
0438
0439 auto eth2
0440 iface eth2 inet static
0441 address 10.2.2.2
0442 netmask 255.255.255.0
0443
0444 iface eth2 inet6 static
0445 address 2002:2::2
0446 netmask 120
0447
0448 auto eth3
0449 iface eth3 inet static
0450 address 10.2.3.2
0451 netmask 255.255.255.0
0452
0453 iface eth3 inet6 static
0454 address 2002:3::2
0455 netmask 120
0456
0457 auto eth4
0458 iface eth4 inet static
0459 address 10.2.4.2
0460 netmask 255.255.255.0
0461
0462 iface eth4 inet6 static
0463 address 2002:4::2
0464 netmask 120
