Ubuntu Server Tips and Tricks

This page is a collection of settings and adjustments made to various aspect of the operating systems installed at ElectricBrain. They mostly apply to the host nodes.

Update 2024.05.18: Ubuntu 24.04 update/upgrade

Indeed - only just over a month since the last operating system upgrade.

The Pi 5's now have NVMe SSDs attached. The plastics have been modified to accomodate the additional height of the GeekWork X1002 M.2 adaptors.

The operating system upgrade sees the networking deprecation warnings come in to force - i.e. there a bits that have been removed which were relied upon, and the warnings have been ignored here.... needless to say, following the steps below will result in no network. Essentially the ifconfig command is now officially gone. All configuration is done via the ip command. This is currently a work in progress.

Stop kernel logging messages being blurted across the console:

Disable unattanded upgrades

  • vi /etc/apt/apt.conf.d/20auto-upgrades
  • APT::Periodic::Update-Package-Lists "1";
    APT::Periodic::Unattended-Upgrade "0";

Fixed address networking

  • edit /etc/networks
    • comment out #link-local 169.254.0.0
  • disable IPv6 on RPi5
    • vi /boot/firmware/cmdline.txt
    • Append "ipv6.disable=1" to the end of the line (no quotes)
    • Append (docker swarm hosts) "cgroup_memory=1 audit=1 cgroup_enable=memory swapaccount=1"
    • save and reboot
  • set the host name permanently
    • > hostnamectl set-hostname myhostname.localdomain
  • Use netplan now
    • Disable cloud-init
      • touch /etc/cloud/cloud-init.disable
      • apt-get purge cloud-init
      • rm -rf /etc/cloud/
      • rm -rf /var/lib/cloud/
    • Create /etc/netplan/99_config.yaml and add something like the following:
    • network:
       version: 2
       renderer: networkd
       ethernets:
         eth0:
           addresses:
             - 192.168.0.2/24
           routes:
             - to: default
               via: 192.168.0.1
           nameservers:
               search: [localdomain]
               addresses: [192.168.0.254]
  • Force OS to use /etc/resolv.conf file, disable and stop the systemd-resolved service
    • sudo systemctl disable systemd-resolved
    • sudo systemctl stop systemd-resolved
    • Delete the symlink /etc/resolv.conf
      rm /etc/resolv.conf
    • Restart NetworkManager if you are using it
    • Else add these lines to a new /etc/resolv.conf
      search     localdomain
      nameserver 192.168.0.254
  • Install timeserver software if this is a reference (i.e. on a firewall)
    • apt install ntp
    • adjust firewall rules to transproxy all time sync requests to this machine
    Else if not a time reference point timesyncd to the reference computer
  • Install Docker
  • Lots more to go here.

Update 2024.04.10: Ubuntu 23.10 update/upgrade

Yes two years to the day!! What a coincidence.

Raspberry Pi 5 is now here and it's coupled to a Geekworm X1002 pcie M.2 SSD card equipped with a Crucial memory P3 500GB SSD. It's currently running at Gen 3 speeds (which at the time of writing is jolly fast).

Getting the system to boot from this device is covered elsewhere. The changes needed to integrate the new node into the Docker Swarm are covered here. You still need to do stuff covered in the earlier releases with only the differences being noted here (before they are forgotten - it was pretty thick and fast with some all night sessions too). Some stuff just doesn't need to be done anymore because the distro has evolved over time.

  • After upgrade from 22.04LTS to 23.10 check/revert. iptables is nolonger compatible.
    • /etc/firewalld/firewalld.conf contains:
      FirewallBackend=iptables
      FirewallBackend=nft
    • NO LONGER REQUIRED. Just use the distro docker packages
      etc/apt/sources.list.d/docker.list contains:
      deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu jammy stable

Update 2022.04.10: Ubuntu 22.04LTS update/upgrade

"From the release notes:

Various kernel modules have been moved from the linux-modules-raspi package in order to reduce the initramfs size. This results in several applications (most notably Docker, bug 1947601 18) failing due to missing kernel modules. To work around this:
sudo apt install linux-modules-extra-raspi

Hopefully this helps."

..... Oops

This also effects ZRam. Note that special attention is required to disable the now default ZSwap.

Ubuntu 22.04 LTS

  • Upgrade from 20.04LTS to 22.04LTS
    • su -
    • apt update
    • apt upgrade
    • systemctl reboot
    • apt dist-upgrade
    • apt auto-remove
    • apt install update-manager-core
    • From the console
      do-release-upgrade -d
      from here on follow prompts. Good luck.
  • Disable zswap
    • Interferes with zram
    • vi /boot/firmware/cmdline.txt
    • Append "zswap.enabled=0" to the end of the line (no quotes)
    • save and reboot
  • Add kernel modules to make Docker, zram work
    • apt install linux-modules-extra-raspi
  • Adjust zram swap settings
    • vi /usr/bin/init-zram-swapping
      From there adjust the mem formula etc. (i.e. divide by 4 not 2)
  • Install ksysguardd (which is no longer available due to re-packaging)
    • cd /tmp
    • wget \
      http://ports.ubuntu.com/pool/universe/k/ksysguard/ksysguardd_5.18.4.1-0ubuntu1_arm64.deb
    • apt install ./ksysguardd_5.18.4.1-0ubuntu1_arm64.deb
  • After upgrade from 20.04LTS to 22.04LTS check
    • /etc/firewalld/firewalld.conf contains:
      FirewallBackend=iptables
    • /etc/apt/sources.list.d/docker.list contains:
      deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu jammy stable

Update 2021.07.19: Ubuntu everywhere

Now used on all RPi4 nodes (as in ALL nodes). Fedora has been completely replaced.

Ubuntu 20.04 LTS

Remove popularity-contest (spyware)

  • apt remove popularity-contest

Remove snapd

Fixed address networking. Systemd-networkd is the default for Ubuntu so there is very little to do

  • edit and setup /etc/systemd/network/20-wired.network
    • Fix network address
      [Match]
      Name=eth0

      [Network]
      LinkLocalAddressing=no
      Address=192.168.0.19/24
      Gateway=192.168.0.254
      DNS=192.168.0.254
  • edit /etc/networks
    • comment out #link-local 169.254.0.0
  • Remove network-manager on Odroid C4
    • > systemctl disable NetworkManager
    • > systemctl stop NetworkManager
    • > systemctl enable systemd-networkd
    • > apt remove network-manager network-manager-pptp
  • Remove netplan
    • > apt remove cloud-init libnetplan0 netplan.io
    • > systemctl reboot
  • disable IPv6 on RPi4b
    • vi /boot/firmware/cmdline.txt
    • Append "ipv6.disable=1" to the end of the line (no quotes)
    • Append (docker swarm hosts) "cgroup_memory=1 audit=1 cgroup_enable=memory swapaccount=1"
    • save and reboot
  • disable IPv6 on AAEON UP Board
    • vi /etc/default/grub
    • Append " ipv6.disable=1" to GRUB_CMDLINE_LINUX_DEFAULT
    • Append (docker swarm hosts) "cgroup_memory=1 audit=1 cgroup_enable=memory swapaccount=1"
    • save and run: update-grub
    • reboot
  • disable IPv6 on OdroidC4
    • vi /media/boot/boot.ini
    • Append " ipv6.disable=1" to setenv bootargs
    • Append (docker swarm hosts) "cgroup_memory=1 audit=1 cgroup_enable=memory swapaccount=1"
    • Save and reboot
  • Remove ModemManager (I don't have a modem) on OdroidC4
    • > systemctl stop ModemManager
    • > systemctl disable ModemManager
    • > apt remove modemmanager
    • > apt purge modemmanager
  • set the host name permanently
    • > hostnamectl set-hostname myhostname.localdomain
  • Force OS to use /etc/resolv.conf file, disable and stop the systemd-resolved service
    • sudo systemctl disable systemd-resolved
    • sudo systemctl stop systemd-resolved
    • Then put the following line in the [main] section of your /etc/NetworkManager/NetworkManager.conf
      dns=default
    • Delete the symlink /etc/resolv.conf
      rm /etc/resolv.conf
    • Restart NetworkManager if you are using it
    • Else add these lines to a new /etc/resolv.conf
      search     localdomain
      nameserver 192.168.0.254

Disable unattanded upgrades

  • vi /etc/apt/apt.conf.d/20auto-upgrades
  • APT::Periodic::Update-Package-Lists "1";
    APT::Periodic::Unattended-Upgrade "0";

Fix audit messages in journal. ElasticSIEM is operating here and no audit messages are wanted in the journal:

  • The socket is stopped:
    systemctl stop systemd-journald-audit.socket
  • The socket is masked:
    systemctl mask systemd-journald-audit.socket
  • The systemd-journald service is restarted:
    systemctl restart systemd-journald
  • Reference https://github.com/systemd/systemd/issues/959

Stop kernel logging messages being blurted across the console:

Force screen output on HDMI0 even when no monitor is connected

  • Edit /boot/firmware/usercfg.txt and add the following lines (4k@24hz)
    • hdmi_force_hotplug=1
      hdmi_group:0=1
      hdmi_mode:0=93

    • hdmi_mode+0=20 (for HD 1080)

    • for Pi3 and earlier hdmi_mode=16 for 1920x1080 (no :0 )

  • These settings are not used by the OS. They are passed through to the firmware prior to booting. Essentially they work for all operating systems on the Pi 4 (although the file is in different places). It means runtime resolution adjustments are not possible on Pi's video - bootime only :-(

Firewalld

  • apt-get install firewalld
  • systemctl enable firewalld
  • systemctl start firewalld
  • Setup your rules.

Make "docker" group local

  • A network group for docker causes docker.socket to be delayed. This causes docker.service to fail with dependency issues.
  • Create a local group with the same gid as the network group
  • Reference: https://github.com/moby/moby/issues/29179

Stop MOTD-NEWS from spying

  • edit /etc/default/motd-news
  • Change ENABLED from 1 to 0
    ENABLED=0
  • Alternatively disable MOTD entirely
    by commenting out two lines in /etc/pam.d/sshd:
    session    optional     pam_motd.so  motd=/run/motd.dynamic
    session    optional     pam_motd.so noupdate
  • Reference: Here
  • systemctl stop motd-news.timer
  • systemctl disable motd-news.timer
  • systemctl mask motd-news.timer
Install sssd to do centralised network LDAP user authentication
  • apt install sssd

Install autofs

  • apt install autofs
Make autofs use files for its maps
  • vi /etc/nsswitch.conf
  • change automount: sss to automount: files

Install docker-ce on Ubuntu Focal

Install zram for swapping

  • apt install zram-config
  • reboot to activate
  • warning - DO NOT enable on Elasticsearch nodes

Disable SSSD Apparmor policy when using LDAP

Disable FWUPD auto firmware updates

Disable daily download and upgrade activities

Disable ua Ubuntu-Advantage (no subscription here)

Disable whoopsie spyware

  • apt autopurge whoopsie

Install timeserver software if this is a reference (i.e. on a firewall)

  • apt install ntp
  • adjust firewall rules to transproxy all time sync requests to this machine

Else if not a time reference point timesyncd to the reference computer

  • edit /etc/systemd/timesyncd.conf
    Uncomment NTP= and add the IP address of the reference
  • timedatectl set-timezone Australia/Melbourne
  • systemctl restart systemd-timesyncd

Disable updatedb indexing containers

  • vi /etc/updatedb.conf
  • Add /var/lib/docker to PRUNEPATHS
  • save and exit

Fedora

  • Remove ModemManager (I don't have a modem):
    rpm -e ModemManager
  • Remove kdeconnect (phones are ios here) security risk (Fedora 31):
    rpm -e kdeconnectd kde-connect kde-connect-libs
  • Turn off that !@#$% dnf makecache timer!
    http://swapoff.blogspot.com.au/2015/09/disable- automatic-package-metadata.html
    This jolly timer and makecache cause all sorts of AVCs from inside containers.
  • Enable selinux 'container_manage_cgroup' which allows systemd to run inside containers
    • setsebool container_manage_cgroup 1
      • Using this less now with F29.
      • No longer used with Fedora 31
  • disable and remove NetworkManager on servers - replace with systemd-networkd
    • dnf remove NetworkManager
      Note: this removes heaps of stuff. It worked for me. Be brave!
    • edit and setup /etc/systemd/network/20-wired.network
      [Match]
      Name=eth0

      [Network]
      Address=192.168.0.19/24
      Gateway=192.168.0.254
      DNS=192.168.0.254
    • systemctl enable systemd-networkd

    • systemctl start systemd-networkd

  • Soundblaster Katana no mic

    • amixer -c2 set 'PCM Capture Source',0  'External Mic'

  • Nvidia Graphics

    • Downgraded hardware in latest desktop build from GT 1060 6GB -> GT 1030 2GB

    • GT 1030 2GB uses a maximum of 30 watts (no more PSU power cables)

    • Performance dictates using Nvidia propietary drivers. Massive speed difference.

  • Akonadi

    • dnf remove kf5-akonadi-server

MacOS

StackOverFlow - How to install Guest addition in Mac OS as guest and Windows machine as host

The answer about using csrutil to turn off MacOS driver signing just blows my mind. There are some super geniuses out there :-) This also works on a Linux host.