TL;DR: This page is used to be quite short. The OS of choice at ElectricBrain is Fedora Ubuntu.

2024-04-10 Update Ubuntu 23.10 on NVMe Gen 3 SSD via Pi 5 pcie

Only the boot loader firmware can turn on the pcie connector on a Raspberry Pi 5 or so it seems. And the only way (I could find) to configure the boot loader is with RaspiOS. Lots and lots of posts were read to arrive at this point. Thankyou to everyone who posted stuff about this issue all over the place. Hopefully this helps someone. BTW - it's jolly fast!

The issue: booting on the Ubuntu SD-Card then performing an lsblk or an lspci command, one cannot see any evidence of an NVMe SSD. As far as I understand it this is essentially because the pcie connector is not enabled. The connector only becomes available when the bootloader goes looking for a pcie M.2 boot device. These steps are what I did to get Ubuntu booted from NVMe on the Pi 5

• Attach NVMe M.2 SSD to Pi 5. I used the Crucial P3 500GB on the GeekWorm X1002 - low power was the most important selection criteria.
• Create a bootable Ubuntu 23.10 SD-Card and verify it boots and works.
• Next boot RaspiOS (yes they renamed it) on another SD-Card.
• Make changes to /boot/firmware/config.txt to enable the PCIE connector (cannot be done if you boot from the Ubuntu SD-card) and reboot. https://www.jeffgeerling.com/blog/2023/testing-pcie-on-raspberry-pi-5
• Attach your Ubuntu SD-Card via a USB3 adaptor such that /dev/sda appears in the lsblk list. https://www.pcguide.com/reviews/best-sd-card-reader/
• Block copy the entire Ubuntu SD-Card's contents (/dev/sda) to the NVMe SSD (/dev/nvme0n1) with dd if=/dev/sda of=/dev/nvme0n1 bs=1m
• Use parted to resize the resulting nvme0n1p2 to take the rest of the NVMe drive. Use resize2fs to get the ext4 filesystem to take up all of the partition, which is waaaay bigger than the SD-Card.
• Change the boot order to use the NVMe SSD first. Enjoy. https://forums.raspberrypi.com/viewtopic.php?t=366106
• Caveat - kernel must be at least 6.5.0-1013-raspi (the pcie enumeration bug was fixed in a very slightly earlier version like 1012.15 or something).

2021-07-18 Update Free yourself from OS lock-in with help from Portainer

As a complete computer geek I feel compelled to talk about how easy it was to completely change the operating system over on a cluster node leading to all nodes on the entire cluster (completed today). The key point here is that it's now possible to be free from the shackles imposed by your operating system provider.

Changing over the OS was a by-product of containerizing all applications.

Only a couple of years ago such a thing would've been considered more or less impossible without days if not weeks of disruption. Essentially the changeover, sparked by IBMs aquisition of Redhat (I'll spare you the somewhat philosophical details), was to remove Fedora and replace it with Ubuntu. The target node was host9 in a cluster of 10 or so computers.

The technical marvel was how easy it was to do. Using Portainer (from New Zealand) to manage things, the node was "drained" of containers. Then it was shutdown, had its storage replaced with the new OS on it, brought up again and configured as a replacement host9. The new OS was complete with Docker ready to join the cluster.

Once back in the swarm the node was "reactived" and bingo. Portainer set about redeploying application containers to it. Zero reinstallation of any applications. Everything pretty much just upped and ran with more or less zero downtime.

With containerization cloud hopping also becomes a reality. Even hopping in and out of clouds to on premisis. Also to a lesser extent, hopping over to power saving ARM cpus (as per Apple) and back again, via multi-architecture containers saving on electricity. All in the same cluster.

2020-10-24 Update Install rpi-imager OS installer

Ubuntu is winning: These guys are just getting better and better. Just select the OS flavour. Choose among lots OSes too, not just Ubuntu. Then tell rpi-imager where to stick it. Then click write. Done!

I had to install snapd on the Fedora 31 desktop first and then install rpi-imager. Rpi-Imager is available for Windows, Mac and as shown here pretty much all variants of Linux. Amazing really.

Lastest round of hardware updates (2 x Rpi4B8G) are on SanDisk Extreme Pro 400GB MicroSD from Amazon.

2020-06-21 Update Abandon Fedora

The new Raspberry RPi4b8gb along with the RPi4b4gb both have USB issues. The latest machine came up without a keyboard. Combined with a perceived lack of support for DockerCE.... I give up!

Enter Ubuntu 20.04 LTS (Long Term Support): What a breath of fresh air. It just works! The swarm controller is an RPi4b4gb which received Ubuntu 18.04 LTS courtesy of James A Chamber's Ubuntu arm64 image when the Pi4 first appeared (since there were no 64-bit OSes available at the time).

Essentially the swarm controller machine has only ever received upgrades via the official OS mechansim. Apart from running out of memory due to too many containers (and a lack of resource management on the part of the operations department - i.e. me), it has never failed. Everything just works for me.

OS configs will be added to the OS Things to remember page as the new 8gb node comes online.

Consideration is being given to abandoning Fedora on all computers.

2020-05-19 Update Fedora 32 - things did not go well

Fedora 32 installed on host9.localdomain. Things did not go well. The swarm essentially did not like it one little bit. Things went wrong, some expected some not.

Firstly, you have do disable the systemd.unified_cgroup_hierarchy=0 thing - expected with DockerCE.

Secondly, the firewalld settings no longer work with Swarm mode and Portainer. This is because Fedora32 now uses nf-tables. This is philosophically good since ipv4 and ipv6 are now managed at driver level as a single thing. It is no longer possible to disable ipv6, because the whole computer just won't work - so DON'T DO IT. All computers here have/had ipv6 disabled because of historical holes in security. This is now fixed.

However, Docker (Swarm at least) doesn't like the new firewall that much. The fix currently deployed is to add the docker0 and docker_gwbridge to the trusted zone. The consequences of doing so are not clear at the time of writing, so it could be risky, but it works. Portainer is running and everything I've checked so far is working (although this doesn't pass as serious testing).

Rolling this out to further nodes is currently on hold until a recommended supported solution becomes available.

There are comments to be found out there where the suggestion is made that Fedora (and hence RedHat (who we all know is really IBM) ) is no longer "supporting" Docker. However, Fedora does have its own port of Docker known officially as the Fedora moby-engine RPM package. It may be necessary to examine this to see how these issues are resolved (if at all).

The conspiracy theory offered for the opinion is that IBM really wants us all to take up OpenShift and Podman with the Swarm stack equivalent being an OpenShift Pod. OpenShift ofcourse is k8s.

FYI Portainer now supports some k8s functionality and says full Kubernetes support is coming. So this possibly opens the door to a (hopefully seamless) transition to the more sophisticated k8s should the need be felt.

2019-11-22 Update Fedora 31 and still no RPi4 support

The latest incarnation at the time of writing is Fedora 31. This is currently installed on the new Ryzen 3700X, 64GB desktop, but unfortunately it's still not available for the RPi4B+. That Pi is still running on the James A Chamber's Ubuntu arm64 image.

Peter Robinson, Thu, 19 Sep 2019 01:51:46 -0700

It's not currently supported. It won't be until likely F-32,
 there's
issues with DMA on aarch64 and the patch sets for support are basic.
As the person that supports the RPi on Fedora the minimum viable
support that needs to happen for me to enable it is display output,
keyboard (ie USB),
 network and storage. Without those I get way too
many support requiests and it's actually just better not to have it
supported at all.

Update:2019.09.21: Operating system dilema

So the RPi4 4GB has arrived at ElectricBrain ... but ... the promised 64 bit operating systems have not appeared. Not only that, kernel hackers around the world are lamenting the RPi4's arrival is some months ahead of schedule AND there are heaps of breaking hardware changes (a.k.a. improvements). Chief among them is Pi4 boots in a different way to its predecessors.

Anyway to cut a long story short, Gentoo 64 was the first working arm64 OS installed (Sakaki's Rpi4 Gentoo 64) but was later abandoned in favour of James A Chamber's Ubuntu arm64 image. Ubuntu's packaging is closer to Fedora than Gentoo. When it comes to the systemd networking implementation there's no NetworkManager to remove, however there is netplan.io and cloudinit to contend with, both of which had to be removed before the preferred option of systemd-networkd was operational.

There is still no sign of Fedora 31 arm64 with the beta failing to boot on RPi4 here. Raspbian's recently announced arm64 kernel boots here, however the userland is still 32 bit (which seems somewhat pointless?).

Update:2019.06.30: Fedora 30 aarch64: With the announcement of the Raspberry Pi 4 comes the demise of one of the last bastions of the 32 bit operating system. The Pi foundation is migrating to the 64 bit OS universe with 8GB pi variants spotted (which cannot be serviced by a 32 bit OS).

2019-06-01 Update Fedora 30 RPi3

The latest incarnation at the time of writing is Fedora 30. During this refresh lots of annoying things have been fixed. No longer is it necessary to hack out the swap partition. It's now pretty much a simple matter of downloading, extracting, block copying and resizing a partition and you're done.

The steps here also include setting up a dedicated LVM partition for docker.

• Download Fedora Minimal

• Verify the checksum using sha256sum
• Connect the pi's MicroSD to your PC with a cheap adaptor off eBay (or MSY Computers).
• Uncompress and pipe the resulting binary blocks to the SDCARD in one command (be sure to identify the sdcard correctly in the output part of the dd command of=/dev/sd? .... otherwise... :-(
  unxz -c Fedora-Minimal-30-1.2.aarch64.raw.xz | dd of=/dev/sdi bs=16384
• user your favourite partition tool to change the size of /dev/sdi3 to a big new size. KDE Parition Manager was able to easily change the partition size and resize the filesystem in one step. Simple and cool.
• Take the sdcard and insert it in to the Pi 3b+. Connect everything up, being monitor, beyboard and ethernet (you can also either/or do WiFi [including 5Ghz on this model], but ethernet done here). Finally connect the power and watch it boot (eventually) to the setup screen.
• Do setup stuff. Reboot. The basic system is now up and running.
• Add the 4th partition which holds an LVM volume for the exclusive use of docker.

• Create a Volume Group using the new 4th partition

[root@host7 ~]# pvcreate /dev/mmcblk0p4
  Physical volume "/dev/mmcblk0p4" successfully created.
[root@host7 ~]# vgcreate docker /dev/mmcblk0p4
  Volume group "docker" successfully created
[root@host7 ~]# vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  docker   1   0   0 wz--n- 85.89g 85.89g

• Stop docker (if it's not installed, then install it now)
• vi /etc/sysconfig/docker-storage-setup

# Edit this file to override any configuration options specified in
# /usr/share/container-storage-setup/container-storage-setup.
#
# For more details refer to "man container-storage-setup"
#STORAGE_DRIVER=overlay2
STORAGE_DRIVER=devicemapper
VG=docker

• Run the command to change the storage: docker-storage-setup
• Start docker with the new storage setup and enable it to autostart on boot:
  systemctl start docker
  systemctl enable docker

Update: 2019.03.11: The Raspberry Pi 2/3 is now supported in in all stable Fedora releases. The new Raspberry Pi 3 B+ also has support in Fedora 29+.
For more information see https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi

Fedora 27 aarch64: https://arm.fedoraproject.org/

Essentially download the minimal image and put it on a microSD card.

The cards used here are the SanDisk Extreme 64GB. These guys are available in 512GB now :-)

Put the SD card in the Rasberry Pi 3 and powerup with a monitor and keyboard plugged in. Answer the questions, and you're done.

Then repartition everything! The default configuration has a swap parition (which I deleted) and an extended partition containing a single parition with the root file system.

1) Change /etc/fstab and delete the swap mount line.

2) Take the microSD card out of the Raspberry Pi 3 and place it in the adaptor that came with the SanDisk. Place that in to your PC.

3) copy the contents of the extended partition /dev/mmcblk0p5 to your HD using dd: i.e.
dd if=/dev/mmcblk0p5 of=mysavefile.dd bs=32768

Specify a blocksize because the OS default is 4K (or at least it used to be) and that will cause read-modify-write cycles (when writing) to the SD cards native block size.

The SD's blocksize is usually way bigger than 4K which will read the same block lots of times then change each 4K OS block and write the same bigger SD block back over and over. This is bad because SDs have a lifetime which is essentially limited by write cycles.

4) Use gparted to delete the swap partition and the extended partition and the partition it contains.

5) Create a new 16GB partition for the root.

6) Create an LVM parition for the remainder of the SD (to be used by docker containers and images).

7) dd the saved image back into partition 3:
dd if=mysavedpartition.dd of=/dev/mmcblk0p3 bs=32768

8) fsck -f /dev/mmcblk0p3

9) resize2fs /dev/mmcblk0p3

10) remove the sd card from the PC and place it back in the Raspberry Pi 3. Powerup again.

Notes:

1) If your screen goes blank, don't worry. There is a kernel bug which causes the vertical refresh wait timer to timeout - sometimes. You can still ping the machine and login in headless mode. Power cycling eventually gets it going again if it's a concern.