TL;DR: This page is used to be quite short. The OS of choice at ElectricBrain is Fedora Ubuntu.

2024-05-24 Update Ubuntu 24.04.10 on NVMe Gen 3 SSD via Pi 5 pcie

Ubuntu 24.04 deprecated and even removed some package relied on here at ElectricBrain. Stuff that worked only recently on 23.10 no longer works on 24.04 due to package being removed entirely.

Most of the GeekWorm X1002 boards with Crucial P3 SSDs attached worked except for one.

If this happens to you then try these tips:

  • https://forums.raspberrypi.com/viewtopic.php?t=363401
    Add this to /boot/firmware/config.txt

    dtparam=nvme
    #dtparam=pciex1_gen=2
    dtparam=pciex1_gen=3

     
  • Add this to eeprom config

    PCIE_PROBE=1
     
  • Add this to /boot/firmware/cmdline.txt

    nvme_core.default_ps_max_latency_us=0 pcie_aspm=off iommu=soft
     
  • If the drive "goes to sleep" due to inactivity then try chirping the disk every minute with a cron job.
    (The Crucial P3 does indeed go to sleep, to save power, causing all sorts of errors due to the drive disappearing giving the impression there is a hardware fault. This completely fixed it):

    * * * * * /bin/dd if=/dev/nvme0n1 of=/dev/null count=1 > /dev/null 2>&1
     

It may happen that the PCIE controller needs a full reset. Use these commands to do that:

echo 1 >/sys/bus/pci/devices/0000:00:00.0/remove
echo 1 >/sys/bus/pci/rescan

2024-04-10 Update Ubuntu 23.10 on NVMe Gen 3 SSD via Pi 5 pcie

Only the boot loader firmware can turn on the pcie connector on a Raspberry Pi 5 or so it seems. And the only way (I could find) to configure the boot loader is with RaspiOS. Lots and lots of posts were read to arrive at this point. Thankyou to everyone who posted stuff about this issue all over the place. Hopefully this helps someone. BTW - it's jolly fast!

The issue: booting on the Ubuntu SD-Card then performing an lsblk or an lspci command, one cannot see any evidence of an NVMe SSD. As far as I understand it this is essentially because the pcie connector is not enabled. The connector only becomes available when the bootloader goes looking for a pcie M.2 boot device. These steps are what I did to get Ubuntu booted from NVMe on the Pi 5

  • Attach NVMe M.2 SSD to Pi 5. I used the Crucial P3 500GB on the GeekWorm X1002 - low power was the most important selection criteria.
  • Create a bootable Ubuntu 23.10 SD-Card and verify it boots and works.
  • Next boot RaspiOS (yes they renamed it) on another SD-Card.
  • Make changes to /boot/firmware/config.txt to enable the PCIE connector (cannot be done if you boot from the Ubuntu SD-card) and reboot. https://www.jeffgeerling.com/blog/2023/testing-pcie-on-raspberry-pi-5
  • Attach your Ubuntu SD-Card via a USB3 adaptor such that /dev/sda appears in the lsblk list. https://www.pcguide.com/reviews/best-sd-card-reader/
  • Block copy the entire Ubuntu SD-Card's contents (/dev/sda) to the NVMe SSD (/dev/nvme0n1) with dd if=/dev/sda of=/dev/nvme0n1 bs=1m
  • Use parted to resize the resulting nvme0n1p2 to take the rest of the NVMe drive. Use resize2fs to get the ext4 filesystem to take up all of the partition, which is waaaay bigger than the SD-Card.
  • Change the boot order to use the NVMe SSD first. Enjoy. https://forums.raspberrypi.com/viewtopic.php?t=366106
  • Caveat - kernel must be at least 6.5.0-1013-raspi (the pcie enumeration bug was fixed in a very slightly earlier version like 1012.15 or something).

2021-07-18 Update Free yourself from OS lock-in with help from Portainer

As a complete computer geek I feel compelled to talk about how easy it was to completely change the operating system over on a cluster node leading to all nodes on the entire cluster (completed today). The key point here is that it's now possible to be free from the shackles imposed by your operating system provider.

Changing over the OS was a by-product of containerizing all applications.

Only a couple of years ago such a thing would've been considered more or less impossible without days if not weeks of disruption. Essentially the changeover, sparked by IBMs aquisition of Redhat (I'll spare you the somewhat philosophical details), was to remove Fedora and replace it with Ubuntu. The target node was host9 in a cluster of 10 or so computers.

The technical marvel was how easy it was to do. Using Portainer (from New Zealand) to manage things, the node was "drained" of containers. Then it was shutdown, had its storage replaced with the new OS on it, brought up again and configured as a replacement host9. The new OS was complete with Docker ready to join the cluster.

Once back in the swarm the node was "reactived" and bingo. Portainer set about redeploying application containers to it. Zero reinstallation of any applications. Everything pretty much just upped and ran with more or less zero downtime.

With containerization cloud hopping also becomes a reality. Even hopping in and out of clouds to on premisis. Also to a lesser extent, hopping over to power saving ARM cpus (as per Apple) and back again, via multi-architecture containers saving on electricity. All in the same cluster.

2020-10-24 Update Install rpi-imager OS installer

Ubuntu is winning: These guys are just getting better and better. Just select the OS flavour. Choose among lots OSes too, not just Ubuntu. Then tell rpi-imager where to stick it. Then click write. Done!

I had to install snapd on the Fedora 31 desktop first and then install rpi-imager. Rpi-Imager is available for Windows, Mac and as shown here pretty much all variants of Linux. Amazing really.

Lastest round of hardware updates (2 x Rpi4B8G) are on SanDisk Extreme Pro 400GB MicroSD from Amazon.

2020-06-21 Update Abandon Fedora

The new Raspberry RPi4b8gb along with the RPi4b4gb both have USB issues. The latest machine came up without a keyboard. Combined with a perceived lack of support for DockerCE.... I give up!

Enter Ubuntu 20.04 LTS (Long Term Support): What a breath of fresh air. It just works! The swarm controller is an RPi4b4gb which received Ubuntu 18.04 LTS courtesy of James A Chamber's Ubuntu arm64 image when the Pi4 first appeared (since there were no 64-bit OSes available at the time).

Essentially the swarm controller machine has only ever received upgrades via the official OS mechansim. Apart from running out of memory due to too many containers (and a lack of resource management on the part of the operations department - i.e. me), it has never failed. Everything just works for me.

OS configs will be added to the OS Things to remember page as the new 8gb node comes online.

Consideration is being given to abandoning Fedora on all computers.

2020-05-19 Update Fedora 32 - things did not go well

Fedora 32 installed on host9.localdomain. Things did not go well. The swarm essentially did not like it one little bit. Things went wrong, some expected some not.

Firstly, you have do disable the systemd.unified_cgroup_hierarchy=0 thing - expected with DockerCE.

Secondly, the firewalld settings no longer work with Swarm mode and Portainer. This is because Fedora32 now uses nf-tables. This is philosophically good since ipv4 and ipv6 are now managed at driver level as a single thing. It is no longer possible to disable ipv6, because the whole computer just won't work - so DON'T DO IT. All computers here have/had ipv6 disabled because of historical holes in security. This is now fixed.

However, Docker (Swarm at least) doesn't like the new firewall that much. The fix currently deployed is to add the docker0 and docker_gwbridge to the trusted zone. The consequences of doing so are not clear at the time of writing, so it could be risky, but it works. Portainer is running and everything I've checked so far is working (although this doesn't pass as serious testing).

Rolling this out to further nodes is currently on hold until a recommended supported solution becomes available.

There are comments to be found out there where the suggestion is made that Fedora (and hence RedHat (who we all know is really IBM) ) is no longer "supporting" Docker. However, Fedora does have its own port of Docker known officially as the Fedora moby-engine RPM package. It may be necessary to examine this to see how these issues are resolved (if at all).

 

Name         : moby-engine
Version      : 19.03.8
Release      : 1.ce.gitafacb8b.fc31
Architecture : x86_64
Size         : 41 M
Source       : moby-engine-19.03.8-1.ce.gitafacb8b.fc31.src.rpm
Repository   : updates
Summary      : The open-source application container engine
URL          : https://www.docker.com
License      : ASL 2.0
Description  : Docker is an open source project to build, ship and run any application as a
             : lightweight container.
             :
             : Docker containers are both hardware-agnostic and platform-agnostic. This means
             : they can run anywhere, from your laptop to the largest EC2 compute instance and
             : everything in between - and they don't require you to use a particular
             : language, framework or packaging system. That makes them great building blocks
             : for deploying and scaling web apps, databases, and backend services without
             : depending on a particular stack or provider.

The conspiracy theory offered for the opinion is that IBM really wants us all to take up OpenShift and Podman with the Swarm stack equivalent being an OpenShift Pod. OpenShift ofcourse is k8s.

FYI Portainer now supports some k8s functionality and says full Kubernetes support is coming. So this possibly opens the door to a (hopefully seamless) transition to the more sophisticated k8s should the need be felt.

2019-11-22 Update Fedora 31 and still no RPi4 support

The latest incarnation at the time of writing is Fedora 31. This is currently installed on the new Ryzen 3700X, 64GB desktop, but unfortunately it's still not available for the RPi4B+. That Pi is still running on the James A Chamber's Ubuntu arm64 image.

Peter Robinson, Thu, 19 Sep 2019 01:51:46 -0700

It's not currently supported. It won't be until likely F-32,
 there's
issues with DMA on aarch64 and the patch sets for support are basic.
As the person that supports the RPi on Fedora the minimum viable
support that needs to happen for me to enable it is display output,
keyboard (ie USB),
 network and storage. Without those I get way too
many support requiests and it's actually just better not to have it
supported at all.
https://www.mail-archive.com/arm@lists.fedoraproject.org/msg11461.html

Update:2019.09.21: Operating system dilema

So the RPi4 4GB has arrived at ElectricBrain ... but ... the promised 64 bit operating systems have not appeared. Not only that, kernel hackers around the world are lamenting the RPi4's arrival is some months ahead of schedule AND there are heaps of breaking hardware changes (a.k.a. improvements). Chief among them is Pi4 boots in a different way to its predecessors.

Anyway to cut a long story short, Gentoo 64 was the first working arm64 OS installed (Sakaki's Rpi4 Gentoo 64) but was later abandoned in favour of James A Chamber's Ubuntu arm64 image. Ubuntu's packaging is closer to Fedora than Gentoo. When it comes to the systemd networking implementation there's no NetworkManager to remove, however there is netplan.io and cloudinit to contend with, both of which had to be removed before the preferred option of systemd-networkd was operational.

There is still no sign of Fedora 31 arm64 with the beta failing to boot on RPi4 here. Raspbian's recently announced arm64 kernel boots here, however the userland is still 32 bit (which seems somewhat pointless?).

Update:2019.06.30: Fedora 30 aarch64: With the announcement of the Raspberry Pi 4 comes the demise of one of the last bastions of the 32 bit operating system. The Pi foundation is migrating to the 64 bit OS universe with 8GB pi variants spotted (which cannot be serviced by a 32 bit OS).

2019-06-01 Update Fedora 30 RPi3

The latest incarnation at the time of writing is Fedora 30. During this refresh lots of annoying things have been fixed. No longer is it necessary to hack out the swap partition. It's now pretty much a simple matter of downloading, extracting, block copying and resizing a partition and you're done.

The steps here also include setting up a dedicated LVM partition for docker.

  • Download Fedora Minimal

  • Verify the checksum using sha256sum
  • Connect the pi's MicroSD to your PC with a cheap adaptor off eBay (or MSY Computers).
  • Uncompress and pipe the resulting binary blocks to the SDCARD in one command (be sure to identify the sdcard correctly in the output part of the dd command of=/dev/sd? .... otherwise... :-(
    unxz -c Fedora-Minimal-30-1.2.aarch64.raw.xz | dd of=/dev/sdi bs=16384
  • user your favourite partition tool to change the size of /dev/sdi3 to a big new size. KDE Parition Manager was able to easily change the partition size and resize the filesystem in one step. Simple and cool.
  • Take the sdcard and insert it in to the Pi 3b+. Connect everything up, being monitor, beyboard and ethernet (you can also either/or do WiFi [including 5Ghz on this model], but ethernet done here). Finally connect the power and watch it boot (eventually) to the setup screen.
  • Do setup stuff. Reboot. The basic system is now up and running.
  • Add the 4th partition which holds an LVM volume for the exclusive use of docker.

[root@host7 ~]# fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.33.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p
Disk /dev/mmcblk0: 119.1 GiB, 127865454592 bytes, 249737216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcf722e3f

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *       2048   391167   389120  190M  6 FAT16
/dev/mmcblk0p2       391168  2488319  2097152    1G 83 Linux
/dev/mmcblk0p3      2488320 69597183 67108864   32G 83 Linux

Command (m for help): n
Partition type
   p   primary (3 primary, 0 extended, 1 free)
   e   extended (container for logical partitions)

Select (default e): p

Selected partition 4
First sector (69597184-249737215, default 69597184):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (69597184-249737215, default 249737215):

Created a new partition 4 of type 'Linux' and of size 85.9 GiB.
Command (m for help): p

Disk /dev/mmcblk0: 119.1 GiB, 127865454592 bytes, 249737216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcf722e3f

Device         Boot    Start       End   Sectors  Size Id Type
/dev/mmcblk0p1 *        2048    391167    389120  190M  6 FAT16
/dev/mmcblk0p2        391168   2488319   2097152    1G 83 Linux
/dev/mmcblk0p3       2488320  69597183  67108864   32G 83 Linux
/dev/mmcblk0p4      69597184 249737215 180140032 85.9G 83 Linux

Command (m for help): t
Partition number (1-4, default 4):
Hex code (type L to list all codes): L

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris        
 1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden or  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx         
 5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data    
 6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility   
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt         
 9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access     
 a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O        
 b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor      
 c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi ea  Rufus alignment
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         eb  BeOS fs        
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a6  OpenBSD         ee  GPT            
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        ef  EFI (FAT-12/16/
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f0  Linux/PA-RISC b
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f1  SpeedStor      
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f4  SpeedStor      
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      f2  DOS secondary  
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fb  VMware VMFS    
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fc  VMware VMKCORE
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fd  Linux raid auto
1c  Hidden W95 FAT3 75  PC/IX           bc  Acronis FAT32 L fe  LANstep        
1e  Hidden W95 FAT1 80  Old Minix       be  Solaris boot    ff  BBT            
Hex code (type L to list all codes): 8e

Changed type of partition 'Linux' to 'Linux LVM'.

Command (m for help): p
Disk /dev/mmcblk0: 119.1 GiB, 127865454592 bytes, 249737216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcf722e3f

Device         Boot    Start       End   Sectors  Size Id Type
/dev/mmcblk0p1 *        2048    391167    389120  190M  6 FAT16
/dev/mmcblk0p2        391168   2488319   2097152    1G 83 Linux
/dev/mmcblk0p3       2488320  69597183  67108864   32G 83 Linux
/dev/mmcblk0p4      69597184 249737215 180140032 85.9G 8e Linux LVM

Command (m for help): w

The partition table has been altered.
Syncing disks.

  • Create a Volume Group using the new 4th partition
[root@host7 ~]# pvcreate /dev/mmcblk0p4
  Physical volume "/dev/mmcblk0p4" successfully created.
[root@host7 ~]# vgcreate docker /dev/mmcblk0p4
  Volume group "docker" successfully created
[root@host7 ~]# vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  docker   1   0   0 wz--n- 85.89g 85.89g
  • Stop docker (if it's not installed, then install it now)
  • vi /etc/sysconfig/docker-storage-setup
# Edit this file to override any configuration options specified in
# /usr/share/container-storage-setup/container-storage-setup.
#
# For more details refer to "man container-storage-setup"
#STORAGE_DRIVER=overlay2
STORAGE_DRIVER=devicemapper
VG=docker
  • Run the command to change the storage: docker-storage-setup
  • Start docker with the new storage setup and enable it to autostart on boot:
    systemctl start docker
    systemctl enable docker

Update: 2019.03.11: The Raspberry Pi 2/3 is now supported in in all stable Fedora releases. The new Raspberry Pi 3 B+ also has support in Fedora 29+.
For more information see
https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi

Fedora 27 aarch64: https://arm.fedoraproject.org/

Essentially download the minimal image and put it on a microSD card.

The cards used here are the SanDisk Extreme 64GB. These guys are available in 512GB now :-)


Put the SD card in the Rasberry Pi 3 and powerup with a monitor and keyboard plugged in. Answer the questions, and you're done.

Then repartition everything! The default configuration has a swap parition (which I deleted) and an extended partition containing a single parition with the root file system.

1) Change /etc/fstab and delete the swap mount line.

2) Take the microSD card out of the Raspberry Pi 3 and place it in the adaptor that came with the SanDisk. Place that in to your PC.

3) copy the contents of the extended partition /dev/mmcblk0p5 to your HD using dd: i.e.
dd if=/dev/mmcblk0p5 of=mysavefile.dd bs=32768


Specify a blocksize because the OS default is 4K (or at least it used to be) and that will cause read-modify-write cycles (when writing) to the SD cards native block size.

The SD's blocksize is usually way bigger than 4K which will read the same block lots of times then change each 4K OS block and write the same bigger SD block back over and over. This is bad because SDs have a lifetime which is essentially limited by write cycles.

4) Use gparted to delete the swap partition and the extended partition and the partition it contains.

5) Create a new 16GB partition for the root.

6) Create an LVM parition for the remainder of the SD (to be used by docker containers and images).

7) dd the saved image back into partition 3:
dd if=mysavedpartition.dd of=/dev/mmcblk0p3 bs=32768

8) fsck -f /dev/mmcblk0p3

9) resize2fs /dev/mmcblk0p3

10) remove the sd card from the PC and place it back in the Raspberry Pi 3. Powerup again.

Notes:

1) If your screen goes blank, don't worry. There is a kernel bug which causes the vertical refresh wait timer to timeout - sometimes. You can still ping the machine and login in headless mode. Power cycling eventually gets it going again if it's a concern.