Ubuntu Touch persistent configurations
This is a description of how to create persistent configurations in Ubuntu Touch. This resolves the issue where advanced users lose their custom Linux configurations after an "OTA" (Over The Air update). Using a "remount-hack-remount" cycle to achieve changes to the distribution's Read Only area has the drawback that those changes will be lost every time there is an OTA.
Essentially the process of configuring many components of the operating system's distribution (which is largely held on Read Only storage) relies on overlaying the relevant folders/files. Overlaying means to leave the original folder/files where they are and replace all run-time references with pointers to user writeable folders/files.
In Ubuntu Touch user writeable configuration information is saved, by convention, in particular part of the file system: /userdata/. A key feature of /userdata/ is that it is not affected by an OTA.
In general user modified configurations which are part of the distribution's Read Only data can be copied to the userdata area. Really all that needs to happen is that the original configuration information from the Read Only area must to be copied in to the /userdata/ writeable area. Changes are then made to the /userdata/ copy.
The only remaining challenge is to make the operating system use the /userdata/ version. In Ubuntu Touch this is performed by the linux "mount" command with the "bind" option when the phone boots. Essentially a script (/init) executes a series of bind mount commands and sets up the file system layout prior to most stuff starting up.
Bind mount operation
Remount part of the file hierarchy somewhere else. The call is:
mount --bind olddir newdir
or by using this fstab entry:
/olddir /newdir none bind
After this call the same contents are accessible in two places.
It is important to understand that "bind" does not to create any second-class or special node in the kernel VFS. The "bind" is just an‐
other operation to attach a filesystem. There is nowhere stored information that the filesystem has been attached by "bind" operation. The
olddir and newdir are independent and the olddir may be umounted.
One can also remount a single file (on a single file). It's also possible to use the bind mount to create a mountpoint from a regular di‐
rectory,
for example:
mount --bind foo foo
The bind mount call attaches only (part of) a single filesystem,
not possible submounts. The entire file hierarchy including submounts is
attached a second place by using:
mount --rbind olddir newdir
Note that the filesystem mount options maintained by kernel will remain the same as those on the original mount point. The userspace
mount options (e.g. _netdev) will not be copied by mount(8) and it's necessary explicitly specify the options on mount command line.
mount(8) since v2.27 allows to change the mount options by passing the relevant options along with --bind. For example:
mount -o bind,ro foo foo
This feature is not supported by the Linux kernel; it is implemented in userspace by an additional mount(2) remounting system call. This
solution is not atomic.
The alternative (classic) way to create a read-only bind mount is to use the remount operation,
for example:
mount --bind olddir newdir
mount -o remount,bind,ro olddir newdir
Note that a read-only bind will create a read-only mountpoint (VFS entry),
but the original filesystem superblock will still be writable,
meaning that the olddir will be writable,
but the newdir will be read-only.
It's also possible to change nosuid,
nodev,
noexec,
noatime,
nodiratime and relatime VFS entry flags by "remount,bind" operation. The an‐
other (for example filesystem specific flags) are silently ignored. It's impossible to change mount options recursively (for example with
-o rbind,ro).
mount(8) since v2.31 ignores the bind flag from /etc/fstab on remount operation (if "-o remount" specified on command line). This is nec‐
essary to fully control mount options on remount by command line. In the previous versions the bind flag has been always applied and it
was impossible to re-define mount options without interaction with the bind semantic. This mount(8) behavior does not affect situations
when "remount,bind" is specified in the /etc/fstab file.
The init script will bind mount a list of directories contained in the file:
/etc/system-image/writable-paths
which is essentially a list of all system data that can be modified or customised by a user.
# 1st column: Mount point # 2nd column: Path relative to root of persistent storage (or auto) # 3rd column: type => persistent|synced|temporary # 4th column: action => none|transition (transition requires persistent) # 5th column: mount flags /android/data android-data persistent none none /etc/NetworkManager/system-connections auto persistent none none /home user-data persistent transition none /media auto temporary none defaults /opt/click.ubuntu.com auto persistent transition none /tmp none temporary none defaults /etc/cups auto persistent none none /var/spool/cups auto persistent none none /var/cache/cups auto temporary none defaults /var/crash auto persistent none none /var/lib/AccountsService/users auto persistent none none /var/lib/aethercast auto persistent none none /var/lib/biometryd-meizu-fp-reader auto persistent none none /var/lib/dbus auto persistent none none /var/lib/extrausers auto persistent transition none /var/lib/logrotate auto persistent none none /var/lib/NetworkManager auto persistent none none /var/lib/ofono auto persistent none none /var/lib/openvpn/chroot/tmp auto temporary none defaults /var/lib/PackageKit auto persistent none none /var/lib/bluetooth auto persistent none none /var/lib/lightdm auto persistent none none /var/lib/lightdm-data auto persistent none none /var/lib/sudo auto temporary none defaults,mode=0700 /var/lib/system-image auto persistent none none /var/lib/systemd auto synced none none /var/lib/upower auto persistent none none /var/lib/usermetrics auto persistent none none /var/lib/ubuntu-location-service auto persistent none none /var/log auto persistent transition none # ufw /etc/default/ufw auto persistent transition none /etc/ufw auto persistent transition none /lib/ufw/user6.rules auto persistent transition none /lib/ufw/user.rules auto persistent transition none # apparmor cache is pregenerated in the image builds /etc/apparmor.d/cache auto persistent transition none # needed by click-apparmor - use transition since some core apps are # pre-installed on the image /var/cache/apparmor auto synced none none /var/lib/apparmor auto synced none none # for a writable dconf db used by customization /custom/etc/dconf auto persistent none none # ssh /etc/ssh auto persistent transition none /etc/init/ssh.override auto persistent transition none # sudoers.d /etc/sudoers.d auto persistent transition none # used for various writable files (timezone, localtime, ...) /etc/writable auto synced none none # ureadahead /var/lib/ureadahead auto persistent transition none # apport /var/lib/apport auto persistent transition none # allow us to disable apport as it slows down image /etc/default/apport auto persistent none none # needed for rfkill persistance /var/lib/rfkill auto persistent transition none # needed for urfkill persistance /var/lib/urfkill auto persistent transition none # needed for usb tethering /var/lib/misc auto persistent transition none # needed to persist ntp enabled/disabled /etc/network/if-up.d auto persistent transition none # snappy /snap auto persistent transition none /var/lib/snapd auto persistent transition none # nfcd /var/lib/nfcd auto persistent transition none # Waydroid /var/lib/waydroid auto persistent none none
The first thing to note here is that I have added the lines about the sudoers.d directory. The effect of that is to ensure that the directory in /userdata/systemdata/etc/sudoers.d "overlays" the real /etc/sudoers.d.
Before attempting to change the bind list, don't forget to make the system-image writable as per the SDcard HowTo page. Remember to reset things back to RO afterwards.
Experimental evidence shows that an OTA does not appear to modify the list of directories needing bind mounting.
Clearly making changes to the bind mount list requires a remount-hack-remount cycle as outlined in the SDCARD Howto page since the list lives on the Read Only area of storage (making it something of an exception to the rule).
Note: As pointed out by Tobiyo Kuujikai, while this may remain persistent for the DEV channel, it may well not remain so for the Release Candidate and Stable channels. Further testing with these channels shows the suspicion is indeed correct. The data in the replicated folder(s) remain intact over OTAs, however the /etc/system-image/writeable-paths list file is overwritten by the RC and Stable channel's more complete update.
Update: (1 Dec 2021) With release candidate version 2 the persistent data in /etc/system-image/writable-paths was *not* overwritten. Nevertheless the modifications/additions to the writable-paths were saved in to /userdata/system-data/etc/system-images/writable-path-customization-to-be-added just in case.
Note: (3 Dec 2021) Tobiyo Kuujikai: "Ah I see what you noticed, yes it is not always overwritten. As ota updates are delta updates When that delta is too large to be determined it will trigger what we call a "full ota update" and be overwritten however"
At this point the OTA process is something of a russian roulette as far as customizations to /etc/system-image/writable-paths is concerned. While it's not hard to recover it's hardly ideal.