The RPi Swarm
Update 2021.07.28 Meet the newest teensy weensy cluster member
It might be smallest node but she opens the door to thousands just like her. The Raspberry Pi Zero W running Docker-ce and managed by Portainer. This tiny unit is equipped with the PiMoroni Enviro hat capable of displaying temperature graphs, measuring sound pressure levels, temperature, barometric pressure and light intensity.
The plan is to feed this information back into the Analytics with the aid of ElasticSearch infrastructure running in the cluster.
Shown in the picture is the video output from the PiZeroW (courtesy of the multi-input PIP monitor) and Portainer which will directly manage this tiny nodelett (is that a new word "Nodelett" ?). It's network name is hostzero :-)
The original page follow, which was written prior to the discovery of Portainer......
Finally the 64 bit RPis are configured in "swarm mode". Docker's swarm mode provides functionality similar to Kubernetes and RedHat's OpenShift. Essentially this a cluster of computers where the swarm controller is capable of autonomously allocating containers to nodes and then starting them up. In the process it allocates and attaches resources such as networking, storage and memory to those containers. Should a container die for whatever reason, monitoring alerts the controller and a new instance can be kicked off.
Docker containers can be connected to an "overlay network", an encapsulation type transport which provides virtual networks across hosts. Readers are probably familiar with tun/tap bridge networks which operate on a single computer. These are great for connecting virtual machines on the same host, but can't span multiple computers. Docker's overlay network driver is a similar concept, however the single computer restriction is removed and containers can talk across machines on a private virtual network. The basis of the overlay network is Linux's built in VXLAN which has been there since kernel version 3.7. (A detailed description is available here: https://success.docker.com/article/networking )
At electricbrain all nodes have access to a shared NFS volume (see: NFS NAS). Manually managed applications have directories which hold run scripts for the containers (as well as systemd service files where that's needed) which live out on the NFS server. Any node can launch any container and attach NFS storage. If the container isn't on the node, it is automatically fetched from local Docker repository prior to startup.
ElectricBrain website running on Docker Swarm using an overlay Private Virtual Network
After starting up on which ever node is manually chosen on the cluster, all of the container's services just appear on the virtual overlay network. Containers can communicate with each other simply by using a network name since the swarm controller's DNS server tracks who is where and what IP address was allocated on the private network.
Link to official Docker Swarm Mode page
Update: The Swarm has expanded and gone multi-architecture too. Contrary to what the next couple of paragraphs state. Nodes are now arm32v7, arm64v8, Intel64 and Ryzen with a CPU count of 32 coupled to 81GB RAM in total.
The swarm currently extends to three computers (hosts 5,7 and 8) resulting in 12 CPUs and 6GB of RAM. The boards are 2xRPi3 and 1xRPi4x4GB all of which are 64 bit quad core arm64v8.
# docker node list ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION 514olajmzj7dceuoxlj2ndbsc host5.localdomain Ready Active 19.03.3 3q50edjfh1bb4jpuu6h9cxs3j host7.localdomain Ready Active 19.03.3 62i0m7y83i0hnh0vdsznqtn08 * host8.localdomain Ready Active Leader 19.03.2
Hosts 1, 2, 3, 6 and TVHeadEnd still all do their own thing for now since I'm not game to try a multi-architecture swarm just yet (if it can even be done - the remaining hosts are a mixture of arm32v6, arm32v7, arm64v8 and intel 64).
Update: The swarm is now managed by Portainer as well as manually, contrary to next couple of paragraphs.
At this point in the journey the configs are not swarm managed. This where the manager decides where and when to run containers.
The operating systems on the swarm hosts are a bit of a mixture too. This was essentially caused by the Raspberry Pi 4 coming out ahead of schedule. The Fedora project was not able to support it in either Fedora 30 or 31. That means RPi3 hosts 5 and 7 are running Fedora 31 while host8 (the swarm controller) is running the James A Chamber's Ubuntu arm64 image.
Also it turned out during the OS refresh process, that Fedora changed over to cgroup V2 in Fedora 31 causing mass confusion. The stated aim of that was to accelerate adoption of the new V2 standard and to attempt to get users to move over to PODMAN docker architecture. Well, I spat the dummy as everything just seemed to be getting too hard. As a result docker-ce is now the electricbrain standard. All other forms of docker either have been or will be removed from all hosts.
Once all hosts were on the same version of docker-ce the swarm commands just worked irrespective of the underlying distro.
Update: I have since learned that IBM Redhat is moving everything over to PodMan and RedShift. I was fortunate enough to get to play with a 16 node Redshift cluster in an banking environment. It's conceptually very close to what's running here, although obviously way more sophisticated. Installing the official ElasticSearch docker containers was a breeze thanks to experience gained on the Swarm. Essentially Kubernetes (k8s)/RedShift can do way more, but Docker Swarm is really easy to get going in comparison. You also don't need a department full of computer engineers.
Portainer Swarm Management
Added Portainer. Needed help with one thing which was obtained via Twitter. These guys are great! I love Portainer. Portainer is an evolving story here at ElectricBrain.
Shown below is the Swarm Visualizer with the Portainer controlled containers:
Some of the containers running on the swarm.