Galactic VPC

This guide provides background on the architecture, design decisions, and installation of the components that make up Galactic VPC.

Galactic VPC is a Kubernetes-native overlay that enables the creation of Virtual Private Cloud (VPC) networks between containers. It allows termination of directly attached interfaces as well as routed network prefixes across a given VPC. This is achieved using an SRv6 (Segment Routing IPv6)-enabled underlay network through which the necessary encapsulated packets are routed.

Our galactic-operator is responsible for two major tasks.

It translates our CRDs into the necessary native or downstream manifests to provide the required functionality. Primarily, this means creating Multus NetworkAttachmentDefinition manifests that configure the relevant CNI plugins.

Additionally, the operator assigns unique identifiers for VPC and VPCAttachment objects. These identifiers are important, as they are used to encode all the relevant endpoint information for the SRv6 targets.

DeepWiki.

galactic-cni is our Kubernetes CNI plugin. It is responsible for creating the necessary low-level Linux kernel resources that tie the container into the relevant SRv6 endpoint. The CNI plugin also assigns a unique local VRF on each host where termination is performed.

DeepWiki.

galactic-agent runs on each Kubernetes node that hosts containers with terminated endpoints. It does not need to run on control-plane-only nodes if they do not host such workloads. It is usually deployed as a DaemonSet to ensure this.

The primary purpose of the galactic-agent is to receive and configure routing instructions from the central routing control plane (aka galactic-router). It connects to an MQTT broker, through which the necessary messages are passed using Protobuf-encoded messages.

In addition, the galactic-agent listens on a host-local Unix socket for gRPC messages from containers created by the galactic-cni plugin. These messages register and deregister network segments and terminations with the central control plane.

DeepWiki.

galactic-router is abstracted behind an MQTT-based pub-sub queue, where it can receive register and deregister events for workloads. Based on these events, the galactic-router builds knowledge of the VPCs and their attached workloads. From this, it creates the necessary routing instructions for each Kubernetes node and dispatches them accordingly.

It is written in Python to allow for higher-level interaction with the Galactic infrastructure. We believe Python is a better match for this component, as its ecosystem for data processing tasks is very rich. That said, there is no fundamental requirement for it to be written in Python. Any language could connect to the MQTT broker and perform the necessary computations. It is also possible to remove the MQTT abstraction entirely and listen for manifest changes inside the cluster to issue the relevant routing instructions. This side of the system is intentionally kept flexible, as we expect to innovate around how routing is performed.

DeepWiki.

Galactic Custom Resource Definitions are specified in the galactic-operator repository. The repository also contains samples for all the resources the operator works with.

In the following example, we showcase a VPC scenario where we connect workloads across three Datum Points of Presence.

Each of the three workloads directly connects to the VPC within the subnets defined for the VPC using both IPv4 and IPv6.

In addition, the workloads in SJC (San Jose) and AMS (Amsterdam) route their own prefixes across this VPC to each other. This is a scenario often encountered in more complex network setups and is often not supported by other overlay technologies.

For those prefixes to work, the container must set up the relevant interface and addresses inside itself. This is similar to what might be done by something like WireGuard.

If you just want to test the basic function, you can simply set them up manually:

# SJC
ip link add name test0 type dummy
ip link set dev test0 up
ip addr add 192.168.1.1/24 dev test0
ip addr add 2001:1::1/64 dev test0

# AMS
ip link add name test0 type dummy
ip link set dev test0 up
ip addr add 192.168.2.1/24 dev test0
ip addr add 2001:2::1/64 dev test0

---
apiVersion: galactic.datumapis.com/v1alpha
kind: VPC
metadata:
  name: my-test-vpc
spec:
  networks:
    - 10.1.1.0/24
    - 2001:10:1:1::/64

---
apiVersion: galactic.datumapis.com/v1alpha
kind: VPCAttachment
metadata:
  name: my-test-vpc-sjc
spec:
  vpc:
    apiVersion: galactic.datumapis.com/v1alpha
    kind: VPC
    name: my-test-vpc
    namespace: default
  interface:
    name: galactic0
    addresses:
      - 10.1.1.1/24
      - 2001:10:1:1::1/64
  routes:
    - destination: 192.168.1.0/24
      via: 10.1.1.1
    - destination: 2001:1::/64
      via: 2001:10:1:1::1
    - destination: 192.168.2.0/24
      via: 10.1.1.3
    - destination: 2001:2::/64
      via: 2001:10:1:1::3

---
apiVersion: v1
kind: Pod
metadata:
  name: my-test-pod-sjc
  annotations:
    k8s.v1alpha.galactic.datumapis.com/vpc-attachment: my-test-vpc-sjc
spec:
  containers:
  - name: test-container
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
    securityContext:
      capabilities:
        add:
          - NET_ADMIN
  nodeSelector:
    topology.kubernetes.io/region: sjc

---
apiVersion: galactic.datumapis.com/v1alpha
kind: VPCAttachment
metadata:
  name: my-test-vpc-iad
spec:
  vpc:
    apiVersion: galactic.datumapis.com/v1alpha
    kind: VPC
    name: my-test-vpc
    namespace: default
  interface:
    name: galactic0
    addresses:
      - 10.1.1.2/24
      - 2001:10:1:1::2/64

---
apiVersion: v1
kind: Pod
metadata:
  name: my-test-pod-iad
  annotations:
    k8s.v1alpha.galactic.datumapis.com/vpc-attachment: my-test-vpc-iad
spec:
  containers:
  - name: test-container
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
    securityContext:
      capabilities:
        add:
          - NET_ADMIN
  nodeSelector:
    topology.kubernetes.io/region: iad


---
apiVersion: galactic.datumapis.com/v1alpha
kind: VPCAttachment
metadata:
  name: my-test-vpc-ams
spec:
  vpc:
    apiVersion: galactic.datumapis.com/v1alpha
    kind: VPC
    name: my-test-vpc
    namespace: default
  interface:
    name: galactic0
    addresses:
      - 10.1.1.3/24
      - 2001:10:1:1::3/64
  routes:
    - destination: 192.168.2.0/24
      via: 10.1.1.3
    - destination: 2001:2::/64
      via: 2001:10:1:1::3
    - destination: 192.168.1.0/24
      via: 10.1.1.1
    - destination: 2001:1::/64
      via: 2001:10:1:1::1

---
apiVersion: v1
kind: Pod
metadata:
  name: my-test-pod-ams
  annotations:
    k8s.v1alpha.galactic.datumapis.com/vpc-attachment: my-test-vpc-ams
spec:
  containers:
  - name: test-container
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
    securityContext:
      capabilities:
        add:
          - NET_ADMIN
  nodeSelector:
    topology.kubernetes.io/region: ams