# What is ikvswitch
## What this package does

This package emulates 3 racks of a datacenter, and its matching
switch infrastructure with a BGP-2-the-host setup.

Its goal is to make it possible to deploy VMs the same way as one
would in a real production environment, with IPv6 un-numbered
connectivity between the spine and the leaf switches in the racks,
and between the leaf switches and the servers.

Each rack has 2 switches, each of them connected to 2 spines over
BGP, using FRR and the bgp-2-the-host over IPv6 un-numbered routing.

The number of racks is limited to 3, as this is enough for a full
HA redundancy, however, the number of ports in leaf switches (ie:
the number of U in your virtual racks) is configurable and unlimited
(the only limits are those of the Linux kernel).

## Designed to let your host server configuration untouched

To avoid making any (possibly distrubing) changes to the host
networking, there's an "internet" VM that is connected to the host
using a very simple L2 connectivity. The only thing that is added
to the host server for routing, is a simple SNAT rule:

```
iptables -t nat -A POSTROUTING -s 192.168.96.0/19 ! -d 192.168.96.0/19 -j SNAT --to-source ${MY_IP}
```

Therefore, if your test server is using BGP-to-the-host itself for
its outbound connectivity (or any other type of setup), there's no
need to add a special configuration to your local BGP daemon, or
any complex routing rules. The above iptables rules (and that rule
only) is enough for a full bi-directional connectivity from your
host server to all switches and hosted VMs.

## Designed for very low number of dependencies

There are a number of other tools designed to setup complex
networking setups, which are probably best fit for testing
a datacenter infrastructure. However, these are often non-free,
and bring a lot of dependencies that need to be installed in
your host server.

Instead, ikvswitch approach is to have a very low amount of
dependency packages. Basically, it needs Qemu, a few networking
utility like bridge-utils, iptables and so on, openstack-debian-images
to create the operating system images of your switches, and that's
about it.

## Designed for application developers, not for network engineers

The goal of ikvswitch is not to help designing a network infrastructure,
but rather to help developers test clustering application (like
OpenStack, but maybe something else...) using a "standard"
bgp-to-the-host L3 network fabric. The only thing that one
needs to do is to "plug VMs in the virtual racks" in a kind
of standard KVM way, knowing that ikvswitch will provide
BGP routing and DHCP.

## Network schema of the virtual infrastructure

What's below is probably the easiest way to understand what the
heck ikvswitch does.

```
  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ host server                  ┃
  ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
   │ ikhn ((dummy net) host nick)
   │
   │ ikhb (host bridge) doing an L2 connectivity to the host server
   │ 192.168.96.1
   │
   │ ikht (host (internet VM) tap)
   │ 192.168.96.2
  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ internet                     ┃
  ┃ 192.168.98.1                 ┃
  ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
   │ ikin1                   │ ikin2 (internet (VM) nic (TAP))
   │                         │
   │ ikib1                   │ ikib2 (internet bridge)
   │                         │
   │ ikit1                   │ ikit2 (internet (VM) TAP)
  ┏━━━━━━━━━━━━┓            ┏━━━━━━━━━━━━┓
  ┃1           ┃            ┃1           ┃
  ┃ spine1     ┃            ┃ spine 2    ┃
  ┃192.168.98.2┃            ┃192.168.98.3┃
  ┃2 3 4 5 6 7 ┃            ┃2 3 4 5 6 7 ┃
  ┗━━━━━━━━━━━━┛            ┗━━━━━━━━━━━━┛
   │ │ │ │ │ │     ╭─────────┘ │ │ │ │ │ ifnames (spine tap): ikstXY, where X is 1 (for spine 1) or 2, and Y 2 to 7
   │ │ │ │ │ │     │           │ │ │ │ │  
   │ └─│─│─│─│───────────┐     │ └─│─│─│─────────────┐ brige names: iksbXY, where X is 1 or 2, and Y 2 to 7
   │   │ │ │ │     │     │     │   │ │ └─────────────│─────────────────────────────────────────────────────┐
   │   │ │ │ │     │     │     │   │ └───────────────│──────────────────────────────────┐                  │
   │   └─│─│─│─────│─────│─────│───│────────┐        │                                  │                  │
   │     │ │ │     │     │     │   └────────│────────│─────────────────┐                │                  │
   │     └─│─│─────│─────│─────│────────────│────────│────────┐        │                │                  │
   │       │ │     │     │     │            │        │        │        │                │                  │
   │       └─│─────│─────│─────│────────────│────────│────────│────────│────────┐       │                  │
   │         │     │     │     │            │        │        │        │        │       │                  │
   │         └─────│─────│─────│────────────│────────│────────│────────│────────│───────│─────────┐        │
   │               │     │     │            │        │        │        │        │       │         │        │
╭--│---------------│-----│-----│-------┐ ╭--│--------│--------│--------│---┐ ╭--│-------│---------│--------│---┐
|  │               │     │     │       | |  │        │        │        │   | | ifnames: ikltXY, where X is the leaf number from 1 to 6, and Y is the port number, 1 or 2
| ╭─────────────────┐   ╭─────────────┐| |╭─────────────┐   ╭─────────────┐| |╭─────────────┐   ╭─────────────┐|
| │1               2│   │1     2      │| |│ 1        2  │   │1        2   │| |│1       2    │   │1        2   │|
| │ rack1 leaf1     │   │ rack1 leaf2 │| |│ rack2 leaf1 │   │ rack2 leaf2 │| |│ rack3 leaf1 │   │ rack3 leaf2 │|
| │ hostname: leaf1 │   │ hn: leaf2   │| |│ hn: leaf3   │   │ hn: leaf4   │| |│ hn: leaf5   │   │ hn: leaf6   │|
| │ 192.168.98.4    │   │ 192.168.98.5│| |│ 192.168.98.6│   │ 192.168.98.7│| |│192.168.98.8 │   │ 192.168.98.9│|
| │3 4 5 6 7 9 10 ..│   │3 4 5 6 7... │| |│ 3 4...      │   │ 3 4...      │| |│ 3 4...      │   │ 3 4...      │|
| └─────────────────┘   └─────────────┘| |└─────────────┘   └─────────────┘| |└─────────────┘   └─────────────┘|
|  │ │                   │ │           | |  │ │               │ │ ifnames: iklsX-Y, where X is the leaf switch number from 1 to 6, and Y is the port number, from 3 to 15
|  │ └───────┐           │ │           | |  │ └───────┐       │ │          | |  │ └───────┐       │ │          |
|  │         │           │ │ (end) brige names: ikebXYY, where X is the number of the rack switch, and YY the interface number
|  │     ╭───│───────────┘ │           | |  │     ╭───│───────┘ │          | |  │     ╭───│───────┘ │          |
|  │     │   │     ╭───────┘           | |  │     │   │     ╭───┘          | |  │     │   │     ╭───┘          |
|  │     │   │     │                   | |  │     │   │     │              | |  │     │   │     │              |
|  │     │   │     │                   | |  │     │   │     │              | |  │     │   │     │              |
|  │     │   │     │                   | |  │     │   │     │              | |  │     │   │     │              |
|  │     │   │     │ ifnames (tap): ikvmX-Y-Z where X is the rack number from 1 to 3, and Y the VM number (or position in the rack) from 3 to 15, and last is the NIC number (1 or 2)
| ┏━━━━━━━┓ ┏━━━━━━━┓                  | | ┏━━━━━━━┓ ┏━━━━━━━┓             | | ┏━━━━━━━┓ ┏━━━━━━━┓             |
| ┃1     2┃ ┃1     2┃                  | | ┃1     2┃ ┃1     2┃             | | ┃1     2┃ ┃1     2┃             |
| ┃ VM  1 ┃ ┃ VM  2 ┃  .....           | | ┃ VM  3 ┃ ┃ VM  4 ┃  .....      | | ┃ VM  5 ┃ ┃ VM  6 ┃  .....      |
| ┃ U3    ┃ ┃ U4    ┃                  | | ┃ U3    ┃ ┃ U4    ┃             | | ┃ U3    ┃ ┃ U4    ┃             |
| ┗━━━━━━━┛ ┗━━━━━━━┛                  | | ┗━━━━━━━┛ ┗━━━━━━━┛             | | ┗━━━━━━━┛ ┗━━━━━━━┛             |
|                               vrack 1| |                          vrack 2| |                          vrack 3|
└--------------------------------------┘ └---------------------------------┘ └---------------------------------┘

```

## What is ran when starting the infrastructure

Each switch is emulated by a virtual machine (see below), that is connected
to its neighbouring switch or server over IPv6 un-numbered (ie:
link-local IPv6 connectivity only). On the local loopback of each
switch, the management IPv4 is added so it is possible to ssh into
them traversing all the infrastructure, thanks to BGP routing.

## What is routed (bridges and VM TAPs)

All of the 192.168.96.0/19 network is routed with NAT on the
host server. All normally physical wires are being virtualized
using a Linux bridge. Every VM interface has its counterpart
"vm tap" in the host server. Then of course, each of these TAP
is then connected to the bridge.

## Nice property with link local

Since ikvswitch sets predictable MAC addresses for the swtich
ports, the IPv6 link-local is always the same, and therefore,
it is rather easy to check for IPv6 link-local connectivity
between switch ports. For example, to check if spine2 is well
connected to the leaf3 switch, we use the port 2 IPv6 of the
leaf3 switch (ie: fe80::a00:27ff:fe06:ac52), and the port
ens7 of the spine2 switch:

```
host-server> # ssh 192.168.97.2
spine1> # ping fe80::a00:27ff:fe06:ac52%ens7
```

The same way, even without FRR BGP routing working, it is
possible to ssh leaf3 from spine1 using the link-local address:

```
spine1> # ssh fe80::a00:27ff:fe06:ac52%ens7
```

## included DHCP server

ikvswitch provides a DHCP server that runs on each odd leaf
switches. That's leaf1, 3 and 5 on the above schematic. These
are by default setting the "next-server" to 192.168.100.2.
Typically, a virtual machine would be setup using that IP
address, connected to any of the leaf switches. This may
later be used to serve as a PXE server, to boot (and install)
your other virtual machines, just like you would do in the
real world with physical servers.

## Target audiance for this package

Anyone that wishes to use a complex BGP to the host setup with
VMs connected to 2 leaf switches, itself connected to 2 spines,
and test that environment.

I wrote this to test a deployment of OpenStack, but any deployment
(for example Kubernetes) may use this infrastructure, if setup
inside VMs.

## What this package does not provide

Even though it prepares the tap interface for VMs, this package
does *not* start VMs that are to be connected to the virtual
networking infrastructure that ikvswitch provides.

# How to use
## Configuration

Edit /etc/ikvswitch/ikvswitch.conf to your liking. The most
important value is the MY_IP one on top of the file, as this
is used for the NAT of what's inside the virtual network.

It's also probably a good idea to edit VM_ROOT_PASS at the
very end of the file.

It isn't needed to configure anything else, unless there is
a conflict with your local network (for example, if the LAN
where your host server is, is on the 192.168.96.0/19 range).

Absolutely all IP subnets, MAC addresses, interface and
bridge names can be configured and named the way you like.

## Starting-up the host networking

Before starting-up all the 9 VMs that will host the virtual
switches, it is mandatory to configure the host server.
Simply do:

```
# ikvswitch-host-networking start
```

to configure the host. This will create nearly 200 virtual NICs
by default, and nearly 100 bridges, so it takes a bit
of time to start.

## Starting-up the switches and connecting to them

It is as simple as:

```
# ikvswitch-setup start
```

That's all there is to it! :)

After that command, it should be possible to ssh into
the "internet" virtual switch:

```
host-server> # ssh 192.168.96.2
```

The ssh authorized_keys inside the switches have been
copied from your host's /root/.ssh/authorized_keys, and
the password is set to what's in /etc/ikvswitch/ikvswitch.conf
(it case you need to debug with VNC).

If everything goes well, from any VM, or even from the
host server, it's possible to reach any of the virtual
switches:

```
spine1> # ssh leaf3
```

or from the host server:

```
host-server> # ssh 192.168.98.6
```

## Debugging over VNC

Each switch is binding a VNC console on ports 5950 to
5958 (by default). So to connect to the leaf1 switch,
here's the command:

```
laptop> # vncviewver host-server:5953
```

then login as root, with the password matching what
is setup in /etc/ikvswitch/ikvswitch.conf (changeme
is the default).

# Spawning a VM connected to a switch

To create a VM just like "VM 1" in the schematic above (ie: port 3
of both rs1 and rs2), use this in the KVM command line for the
network interfaces:

```
 -device virtio-net-pci,netdev=net0,mac=MAC_ADDR_OF_THE_VM -netdev tap,id=net0,ifname=ikvm1-3-1 \
 -device virtio-net-pci,netdev=net1,mac=MAC_ADDR_OF_THE_VM -netdev tap,id=net1,ifname=ikvm1-3-2
```

The last parameter "ifname=ikvm1-3-1" at the end of the command line
contains the rack number (the first "1"), then the position in the
rack (the number "3" in the middle), and the last number should be
set to "1" for the first NIC, and "2" for the 2nd NIC as described
below:

```
 ifname=ikvm1-3-2
            │ │ │
            │ │ └─ Leaf switch 1 or 2
            │ └─── Port number of switches in the rack (from 3 to 15)
            └───── Rack number (from 1 to 3)
```

Now, as another example, let's say the VM is in rack 2, position 6:

```
-device virtio-net-pci,netdev=net0,mac=VM_NIC1_MAC_ADDR -netdev tap,id=net0,ifname=ikvm1-3-1 \
-device virtio-net-pci,netdev=net1,mac=VM_NIC2_MAC_ADDR -netdev tap,id=net1,ifname=ikvm1-3-2
```

So only the ifname needs to be modified to change the location
of a VM in the datacenter virtual aisle. If we want to "rack
the VM" in position 5, in rack 3:

```
-device virtio-net-pci,netdev=net0,mac=VM_NIC1_MAC_ADDR -netdev tap,id=net0,ifname=ikvm3-5-1 \
-device virtio-net-pci,netdev=net1,mac=VM_NIC2_MAC_ADDR -netdev tap,id=net1,ifname=ikvm3-5-2
```

As we're having a DHCP server on port ens6 to ens18 on the
leaf1, leaf3 and leaf5 switches, it could conflict with the
BGP-2-host setup. Therefore, on leaf switches, we're having
a VLAN on each of the interface, which is where the final
VMs must connect to. By default it's the VLAN number 10
(this can be changed in ikvswitch.conf).
