ElectricBrain | Ubuntu Server Tips

Ubuntu Server Tips and Tricks

This page is a collection of settings and adjustments made to various aspect of the operating systems installed at ElectricBrain. They mostly apply to the host nodes.

Update 2024.04.10: Ubuntu 23.10 update/upgrade

Yes two years to the day!! What a coincidence.

Raspberry Pi 5 is now here and it's coupled to a Geekworm X1002 pcie M.2 SSD card equipped with a Crucial memory P3 500GB SSD. It's currently running at Gen 3 speeds (which at the time of writing is jolly fast).

Getting the system to boot from this device is covered elsewhere. The changes needed to integrate the new node into the Docker Swarm are covered here. You still need to do stuff covered in the earlier releases with only the differences being noted here (before they are forgotten - it was pretty thick and fast with some all night sessions too). Some stuff just doesn't need to be done anymore because the distro has evolved over time.

After upgrade from 22.04LTS to 23.10 check/revert. iptables is nolonger compatible.
- /etc/firewalld/firewalld.conf contains:
  ~~FirewallBackend=iptables~~
  FirewallBackend=nft
- NO LONGER REQUIRED. Just use the distro docker packages
  etc/apt/sources.list.d/docker.list contains:
  deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu jammy stable

Update 2022.04.10: Ubuntu 22.04LTS update/upgrade

"From the release notes:

Various kernel modules have been moved from the linux-modules-raspi package in order to reduce the initramfs size. This results in several applications (most notably Docker, bug 1947601 18) failing due to missing kernel modules. To work around this:
sudo apt install linux-modules-extra-raspi

Hopefully this helps."

..... Oops

This also effects ZRam. Note that special attention is required to disable the now default ZSwap.

Ubuntu 22.04 LTS

Upgrade from 20.04LTS to 22.04LTS
- su -
- apt update
- apt upgrade
- systemctl reboot
- apt dist-upgrade
- apt auto-remove
- apt install update-manager-core
- From the console do-release-upgrade -d from here on follow prompts. Good luck.
Disable zswap
- Interferes with zram
- vi /boot/firmware/cmdline.txt
- Append "zswap.enabled=0" to the end of the line (no quotes)
- save and reboot
Add kernel modules to make Docker, zram work
- apt install linux-modules-extra-raspi
Adjust zram swap settings
- vi /usr/bin/init-zram-swapping
  From there adjust the mem formula etc. (i.e. divide by 4 not 2)
Install ksysguardd (which is no longer available due to re-packaging)
- cd /tmp
- wget \
  http://ports.ubuntu.com/pool/universe/k/ksysguard/ksysguardd_5.18.4.1-0ubuntu1_arm64.deb
- apt install ./ksysguardd_5.18.4.1-0ubuntu1_arm64.deb
After upgrade from 20.04LTS to 22.04LTS check
- /etc/firewalld/firewalld.conf contains:
  FirewallBackend=iptables
- /etc/apt/sources.list.d/docker.list contains:
  deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu jammy stable

Update 2021.07.19: Ubuntu everywhere

Now used on all ~~RPi4~~ nodes (as in ALL nodes). Fedora has been completely replaced.

Ubuntu 20.04 LTS

Remove popularity-contest (spyware)

apt remove popularity-contest

Remove snapd

> snap list
> snap remove lxd
> snap remove core18
> apt autoremove snapd
> apt purge snapd
Reference: https://www.kevin-custer.com/blog/disabling-snaps-in-ubuntu-20-04/

Fixed address networking. Systemd-networkd is the default for Ubuntu so there is very little to do

edit and setup /etc/systemd/network/20-wired.network
- Fix network address
  [Match]
  Name=eth0
  
  [Network]
  LinkLocalAddressing=no
  Address=192.168.0.19/24
  Gateway=192.168.0.254
  DNS=192.168.0.254
edit /etc/networks
- comment out #link-local 169.254.0.0
Remove network-manager on Odroid C4
- > systemctl disable NetworkManager
- > systemctl stop NetworkManager
- > systemctl enable systemd-networkd
- > apt remove network-manager network-manager-pptp
Remove netplan
- > apt remove cloud-init libnetplan0 netplan.io
- > systemctl reboot
disable IPv6 on RPi4b
- vi /boot/firmware/cmdline.txt
- Append "ipv6.disable=1" to the end of the line (no quotes)
- Append (docker swarm hosts) "cgroup_memory=1 audit=1 cgroup_enable=memory swapaccount=1"
- save and reboot
disable IPv6 on AAEON UP Board
- vi /etc/default/grub
- Append " ipv6.disable=1" to GRUB_CMDLINE_LINUX_DEFAULT
- Append (docker swarm hosts) "cgroup_memory=1 audit=1 cgroup_enable=memory swapaccount=1"
- save and run: update-grub
- reboot
disable IPv6 on OdroidC4
- vi /media/boot/boot.ini
- Append " ipv6.disable=1" to setenv bootargs
- Append (docker swarm hosts) "cgroup_memory=1 audit=1 cgroup_enable=memory swapaccount=1"
- Save and reboot
Remove ModemManager (I don't have a modem) on OdroidC4
- > systemctl stop ModemManager
- > systemctl disable ModemManager
- > apt remove modemmanager
- > apt purge modemmanager
set the host name permanently
- > hostnamectl set-hostname myhostname.localdomain
Force OS to use /etc/resolv.conf file, disable and stop the systemd-resolved service
- sudo systemctl disable systemd-resolved
- sudo systemctl stop systemd-resolved
- Then put the following line in the [main] section of your /etc/NetworkManager/NetworkManager.conf
  dns=default
- Delete the symlink /etc/resolv.conf
  rm /etc/resolv.conf
- Restart NetworkManager if you are using it
- Else add these lines to a new /etc/resolv.conf
  search localdomain
  nameserver 192.168.0.254

Disable unattanded upgrades

vi /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "0";

Fix audit messages in journal. ElasticSIEM is operating here and no audit messages are wanted in the journal:

The socket is stopped:
systemctl stop systemd-journald-audit.socket
The socket is masked:
systemctl mask systemd-journald-audit.socket
The systemd-journald service is restarted:
systemctl restart systemd-journald
Reference https://github.com/systemd/systemd/issues/959

Stop kernel logging messages being blurted across the console:

```
# echo 'kernel.printk = 2 4 1 7' >> /etc/sysctl.conf
# sysctl -p
```
Reference https://access.redhat.com/discussions/4905891 This works on Ubuntu and Fedora (and probz Oracle Linux too).
Alternatively
# dmesg --console-off

Force screen output on HDMI0 even when no monitor is connected

Edit /boot/firmware/usercfg.txt and add the following lines (4k@24hz)
- hdmi_force_hotplug=1
  hdmi_group:0=1
  hdmi_mode:0=93
- hdmi_mode+0=20 (for HD 1080)
- for Pi3 and earlier hdmi_mode=16 for 1920x1080 (no :0 )
These settings are not used by the OS. They are passed through to the firmware prior to booting. Essentially they work for all operating systems on the Pi 4 (although the file is in different places). It means runtime resolution adjustments are not possible on Pi's video - bootime only :-(

Firewalld

apt-get install firewalld
systemctl enable firewalld
systemctl start firewalld
Setup your rules.

Make "docker" group local

A network group for docker causes docker.socket to be delayed. This causes docker.service to fail with dependency issues.
Create a local group with the same gid as the network group
Reference: https://github.com/moby/moby/issues/29179

Stop MOTD-NEWS from spying

edit /etc/default/motd-news
Change ENABLED from 1 to 0
ENABLED=0
Alternatively disable MOTD entirely
by commenting out two lines in /etc/pam.d/sshd:
session optional pam_motd.so motd=/run/motd.dynamic
session optional pam_motd.so noupdate
Reference: Here
systemctl stop motd-news.timer
systemctl disable motd-news.timer
systemctl mask motd-news.timer

Install sssd to do centralised network LDAP user authentication

apt install sssd

Install autofs

apt install autofs

Make autofs use files for its maps

vi /etc/nsswitch.conf
change automount: sss to automount: files

Install docker-ce on Ubuntu Focal

Install Docker Engine on Ubuntu

Install zram for swapping

apt install zram-config
reboot to activate
warning - DO NOT enable on Elasticsearch nodes

Disable SSSD Apparmor policy when using LDAP

ln -sf /etc/apparmor.d/usr.sbin.sssd /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.sbin.sssd
Ref: https://bgstack15.wordpress.com/2020/12/03/disable-apparmor-for-sssd/

Disable FWUPD auto firmware updates

systemctl disable fwupd-refresh.timer
systemctl stop fwupd-refresh.timer
Ref: https://linoxide.com/how-to-update-firmware-on-ubuntu-using-fwupd/

Disable daily download and upgrade activities

systemctl disable apt-daily-upgrade.timer
systemctl disable apt-daily.timer
systemctl stop apt-daily-upgrade.timer
systemctl stop apt-daily.timer
apt autoremove unattended-upgrades
Ref: https://superuser.com/questions/1327884/how-to-disable-daily-upgrade-and-clean-on-ubuntu-16-04

Disable ua Ubuntu-Advantage (no subscription here)

apt autopurge ubuntu-advantage-tools
Ref: https://askubuntu.com/questions/1338724/ua-ubuntu-advantage-unit-installed-but-not-subscribed

Disable whoopsie spyware

apt autopurge whoopsie

Install timeserver software if this is a reference (i.e. on a firewall)

apt install ntp
adjust firewall rules to transproxy all time sync requests to this machine

Else if not a time reference point timesyncd to the reference computer

edit /etc/systemd/timesyncd.conf
Uncomment NTP= and add the IP address of the reference
timedatectl set-timezone Australia/Melbourne
systemctl restart systemd-timesyncd

Disable updatedb indexing containers

vi /etc/updatedb.conf
Add /var/lib/docker to PRUNEPATHS
save and exit

Fedora

Remove ModemManager (I don't have a modem):
rpm -e ModemManager
Remove kdeconnect (phones are ios here) security risk (Fedora 31):
rpm -e kdeconnectd kde-connect kde-connect-libs
Turn off that !@#$% dnf makecache timer!
http://swapoff.blogspot.com.au/2015/09/disable- automatic-package-metadata.html
This jolly timer and makecache cause all sorts of AVCs from inside containers.
Enable selinux 'container_manage_cgroup' which allows systemd to run inside containers
- setsebool container_manage_cgroup 1
  - Using this less now with F29.
  - No longer used with Fedora 31
disable and remove NetworkManager on servers - replace with systemd-networkd
- dnf remove NetworkManager
  Note: this removes heaps of stuff. It worked for me. Be brave!
- edit and setup /etc/systemd/network/20-wired.network
  
  [Match] Name=eth0 [Network] Address=192.168.0.19/24 Gateway=192.168.0.254 DNS=192.168.0.254
- systemctl enable systemd-networkd
- systemctl start systemd-networkd
Soundblaster Katana no mic
- amixer -c2 set 'PCM Capture Source',0 'External Mic'
Nvidia Graphics
- Downgraded hardware in latest desktop build from GT 1060 6GB -> GT 1030 2GB
- GT 1030 2GB uses a maximum of 30 watts (no more PSU power cables)
- Performance dictates using Nvidia propietary drivers. Massive speed difference.
Akonadi
- dnf remove kf5-akonadi-server

MacOS

StackOverFlow - How to install Guest addition in Mac OS as guest and Windows machine as host