How Re6st works globally?
Re6st is a software that creates a resilient, scalable, ipv6 network on top of an existing ipv4 network, by creating tunnels on the fly and routing traffic through them.
A re6st network consists of one server (re6st-registry) and many nodes (re6stnet).
Every node is affiliated to a registry. It creates and accepts tunnels to and from some nodes in the network, these nodes are its neighbors.
Packets are routed through these tunnels, and the tunnels of a Re6st network form a mesh network.
If a node wants to contact a machine outside the Re6st network, packets can exit the network through specific nodes that use BGP (Border Gateway Protocol) and interface a Re6st network with the internet.
- Registry (re6st-resgistry)
Re6st-registry manages its Re6st network, its main tasks are:
- registering nodes in the network
- attributing IP addresses to nodes
- revoking nodes if necessary
- delivering certificates to nodes allowing them to be part of the network
- defining and distributing the network parameters to all nodes
Babel
Babel is the routing protocol used by re6st-nodes.
The daemon Babeld communicates with other Babeld and decide the best routes (i.e lowest metrics) to install in the kernel for each node, therefore, it's up to Babel to give efficient routes to Re6st.
Re6st uses the routes installed by Babel to create tunnels and route its traffic.
HMAC (key-hashed message authentication code) consists of adding a key to every Babel message exchanged mainly to ensure that Babel communication is only made between two machines of the same Re6st network.
Without HMAC, nodes on a same LAN but not in the same re6st network could establish Babel communication through local links.
This unwanted behavior can lead to nodes from different re6st networks to exchange their routes via Babel and therefore to bypass BGP by having direct routes to another Re6st network.
When a network is created, a HMAC key is randomly generated on 16 bytes, a notification is then propagated to all nodes and the new key is communicated to them when they ask for network parameters.
The registry manages this key and can update it whenever it wants via the RPC updateHMAC, that modifies the database of the registry and changes the version of network parameters.
Because it's Re6st that launches Babel, it will restart it upon receiving new HMAC changes to take it into account by modifying the babeld process call.
If a node was down when the parameters were propagated, the node will get them when restarting because nodes check their versions of network parameters every 5 minutes.
The database of the registry can contain 3 different values for HMAC: babel_hmac0, babel_hmac1, babel_hmac2, it's this database that dictates the HMAC state on a Re6st network and nodes should stay synchronized with it.
The HMAC configuration follows this cycle of changes that allows us to perform steps that do not block communication (state N is compatible with state N-1):
Because we only need to store a maximum of two keys at the same time, we can just have two variables stored in the database of nodes: babel_hmac_sign and babel_hmac_accept, the logic is always that the lowest index of babel_hmac stored on the registry will be assigned to babel_hmac_sign, and the second index will be assigned to babel_hmac_accept.
When a key is assigned to babel_hmac_sign, babel will use that key to sign outgoing packets, and will also accept incoming packets signed with it.
When a key is assigned to babel_hmac_accept, babel will simply accept incoming packets signed with this key.
To introduce HMAC on a Re6st network that does not use HMAC for Babel yet, a trick was added in Babel: we jump directly from nothing related to HMAC in the database of the registry to state 2 (where babel_hmac1 will be our new key and babel_hmac2 will be an empty string)
This empty strings means that the nodes will ignore packets that don't carry a HMAC key, this way even the introduction of the first key on a network that does not use HMAC for Babel will not block any communication with a possibly heterogeneous update to state 2 (where we can have on the same networks machines that sign with the new HMAC key and machines that don't sign packets with HMAC yet)
Tunnels management
A node has several types of interfaces:
re6stnet1, re6stnet2, ...: these correspond to the client tunnels that link this node to another node.
re6stnet-tcp, re6stnet-udp...: these correspond the server tunnels that allow several incoming connections from several nodes to this node.
These interfaces are created by OpenVPN, which is a subprocess managed by Re6st that is used for instantiation and destruction of tunnels.
Destruction of tunnels is delicate because machines at both ends of it must agree on its destruction, therefore, the tunnels destruction process works like a state machine:
- softLocking: the tunnel is marked as expensive from a routing point of view, this allows to not be too brutal and let the time to nodes to change their routes.
- hardLocking: the tunnel is marked as "unusable", after 60s of inactivity, we go to the next state
- locked: a request to close the tunnel has been sent to remote end-point that will kill the tunnel upon receiving this request
-> if tunnel is killed, there are no more states
- unlocking: the tunnel cannot be deleted, its cost is restored to its previous value
You can find a sample of interfaces for a machine that listens for connections in udp4, udp6, tcp, and that has currently 2 clients.
*NETNSif... are interfaces created for the demo
Communication between nodes and the registry
The registry has a RPC (Remote Procedure Call) server that contains methods callable remotely by every node in its Re6st network. Here are a few important ones that shape the global operation of Re6st:
hello is a function in Re6st initiating a handshake between the registry and a node:
Node->Registry: Common Name of the node
Registry->Node: X ( = a newly generated secret, encrypted with the client's certificate) + Sign(Certificate Authority, X)
Once a connection between a node and the registry has been initiated like seen above, it should stay authenticated while none of them restart Re6st, and communication is using the following secret scheme:
| # |
secret used* |
direction |
content |
| 1 |
secret 1 |
Node->Registry |
path_infoX?query_stringX |
| 2 |
secret 2 |
Registry->Node |
resultX |
| 3 |
secret 3 |
Node->Registry |
path_infoY?query_stringY |
| 4 |
secret 4 |
Registry->Node |
resultY |
| ... |
|
|
|
| N |
secret N |
Node->Registry |
path_infoZ?query_stringZ |
| N+1 |
secret N+1 |
Registry->Node |
resultZ |
* secret+1 = SHA1(secret) to protect from replay attacks.
When a node starts Re6st without anything cached, the only information it has is the address of the registry, this function returns a peer that this node can contact to fetch configuration and create routes.
The registry stores some parameters that define the network config in its database, these parameters are used by nodes in their configuration and should be consistent with the registry.
Nodes keep their network config synchronized with the registry's configuration with this method.
Some fields of the network config require re6st to reboot when updated because they change the executions of currently running processes such as OpenVPN or Babel, a few examples of these fields are hello, encrypt, ipv4, babel_hmac_accept, babel_hmac_sign.
Provides a certificate to a node that calls this method, a valid certificates is mandatory for a node to be part of a Re6st network.
The node must have a token allowing it to join the Re6st network, unless the registry is allowed to deliver certificates to anonymous clients.
The nodes certificates last for a year and they call the function renewCertificate of the registry a month before the expiration date of their certificate to renew it.
If a node fails to renew its certificate, it will be expelled from the Re6st network.
Communication between nodes
Nodes should not communicate with the registry too often because with large Re6st networks, the registry would receive too much traffic.
So nodes communicate with each other via Re6st for several purposes:
- Create tunnels
- Kill tunnels
- Get versions of the network parameters
- Get node information
Connection between two nodes (A─>B) must be previously established via a handshake:
- A->B: 0, public certificate key of A
- B->A: 1, fingerprint(public certificate key of B), public certificate key of A
- A->B: 2, X ( = a newly generated secret, encrypted with public certificate key of B), Sign(private certificate key of A, X)
The fingerprint is only used to know if peer's certificate has changed
Once the connection is established, packets can be exchanged like this:
| # * |
Direction |
Content |
| 3 |
B->A |
dataX (= 3, typeX, valueX), hmac(secret, dataX) |
| 4 |
A->B |
dataY (= 4, typeY, valueY), hmac(secret, dataY) |
| ... |
|
|
| n |
B->A |
dataZ (= n, typeZ, valueZ), hmac(secret, dataZ) |
* reject messages with # smaller or equal than previously processed to avoid replay attacks.
Try Re6st
You can download Re6st here https://lab.nexedi.com/nexedi/re6stnet (same as http://re6st.net).
You can experiment on Re6st with the demo that uses Nemu to emulate different Re6st nodes with their own network configuration using network namespaces.
The usual delays used on real networks are shorten via a wrapper to better observe the results of actions we take on machines.
Here are the main features of the demo:
- Display a graph of the network via the port of our choice.
- Verify that communication between machines is working properly via ping (option
--ping).
- Verify that HMAC for babel is working as expected and that its transmission on nodes is achieved promptly (option
--hmac)
- Test the overall connectivity and performance of a Re6st network
More information can be found in the script of the demo: re6stnet/demo/demo.
References: