Using overlayFS for read-only SD cards

Development
1

In order to make Zynthian more “Live Friendly” and tolerant of unexpected power down, I am looking into what is required to bring overlayFS into Zynthian.

OverlayFS provides a shadow filesystem so that the root filesystem can be mounted as read-only. When a reboot or normal shutdown is performed, a sync operation occurs and the root file system is mounted as read-write, and the modified files are copied to the SD card and then the filesystem is returned to read-only status before final shutdown.

https://www.domoticz.com/wiki/Setting_up_overlayFS_on_Raspberry_Pi

I started by following the instructions above. I found that I needed to add /root to the list of directories and filesystem that are used as overlayFS.

I forked the domoticz repo on github: GitHub - hansrune/domoticz-contrib: Contribution / added files for Domoticz home automation and I am making the changes for Zynthian specific. This will simlify the instructions below as I make improvements.

The first time the sync operations are run, it takes a really long time (minutes) to shutdown/sync the filesystem, so patience is required.

These are the overlayFS directories:

  • /var
  • /etc
  • /root
  • /home

The following is mounted RO, but is not an overlayFS:

  • /boot

Start with a fresh image
https://os.zynthian.org/2019-05-17-zynthianos-stretch-lite-1.0.0.zip

Full Installer - one script to do the whole install
(Working as of 2019-05-18T04:00:00Z)

>cd /tmp
>wget https://github.com/smiths73v3/overlayFS/raw/master/scripts/full_install.sh
>chmod a+rx full_install.sh
>./full_install.sh
>reboot

Step by step instructions (Working as of 2019-05-18T04:00:00Z)

Disable Swap

>dphys-swapfile swapoff
>dphys-swapfile uninstall
>systemctl disable dphys-swapfile

Install packages

>apt-get install fuse lsof

Get the fetch utility and fetch required files:

>cd /tmp
>wget https://github.com/smiths73v3/overlayFS/raw/master/scripts/get_files.sh
>chmod a+rx get_files.sh
>./get_files.sh

Run the Installer

>./install_parts.sh

Enable the service

>systemctl daemon-reload
>systemctl enable syncoverlayfs.service

Change the boot commandline

>sed -i.bak -e “s/$/ noswap fastboot ro/” /boot/cmdline.txt

Update fstab

>./fixup_fstab.sh

Prepare the overlays directories
This can take a LONG time to complete, at least 20 minutes on a fast SD card.

>./movefs.sh

REBOOT, and enjoy

>reboot

Extra Credit, verify

>mount -v “overlay”
overlay on /root type overlay (rw,relatime,lowerdir=/root_org,upperdir=/root_rw/upper,workdir=/root_rw/work)
overlay on /home type overlay (rw,relatime,lowerdir=/home_org,upperdir=/home_rw/upper,workdir=/home_rw/work)
overlay on /var type overlay (rw,relatime,lowerdir=/var_org,upperdir=/var_rw/upper,workdir=/var_rw/work)
overlay on /etc type overlay (rw,relatime,lowerdir=/etc_org,upperdir=/etc_rw/upper,workdir=/etc_rw/work)

Check for Read Only

> mount -v | grep “^/”
/dev/mmcblk0p2 on / type ext4 (ro,noatime,data=ordered)
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

The / and /boot filesystems should be ro for read-only. To make changes to /boot, or make changes from the webconf, use the following command to make /boot and / read-write

>rootrw

To return to read-only mode:

>rootro

To check that ram-swap is disabled:

>free -m | grep Swap
Swap: 0 0 0

4 Likes
2

It would be advantageous to save changes made by the user more frequently than just formal shutdown. A musical instrument user (player) does not necessarily find it intuitive to close down the instrument ‘properly’ to ensure sound design changes made during a session persist. I understood that Zynthian was designed to minimise disk writes to allow repowering with reduced risk of corruption. I welcome any enhancement that improve resilience. Let’s ensure we cover off the user requirements and avoid conflicting design whilst adding to this great project.

3 Likes
3

Good point @riban,

For background on the issue, see the following post:
Avoiding zynthian power down failure

Unexpected power down can cause SD card corruption. This will result in a non-functioning Zythian

Assumptions:

  1. If an SD card is mounted read-only, there will be no SD card corruption on unexpected power-down.
  2. The user must be able to make changes to the system which cannot be achieved with the SD card in read-only mode.

Requirements:

  1. The user must be able to perform a sync operation to force any changes to be written to the SD card.
  2. The system must be able to perform an automatic sync operation on normal shutdown.
  3. The user should be able to select a read-only “live mode” from the UI, and specify this as the default at boot. Any automatic sync would be disabled in this mode.
  4. If the user selects MIDI or Audio recording, the filesystem will be remounted read-write avoid issues with running out of space.
  5. The user should be able to select a read-write “designer mode” from the UI, and specify this as the default at boot. In this mode, all write operations occur immediately.
  6. The status bar must contain some indication of the “read-only/read-write” state of the SD card, any pending writes that need to be synced.
1 Like
4

In my investigation, I found that the /boot partition is VFAT, and that is not able to be used by overlayFS. This means we need a different solution to deal with writes to /boot.

I expect any changes to /boot to be performed only by the webconf, and that it would be able to make a call to the rootrw utility to remount /boot as read-write, and then return to read-only after the writes have completed.

I did not find a hook I could use to accomplish this task, and could use a hint if anyone has an idea of where I might make that change.

5

It may be advantageous to give the user the option to sync even in the read-only mode, e.g.

  • Default boot set to read-only
  • User makes changes that they want to save
  • Changes indicated in status
  • Menu option to sync changes
  • Option to save changes during manual shutdown process

Other synthesisers require user to change to an edit mode before changes can be made. I find this irksome at times but if it can be simple to access then it may be an acceptable workflow. It may prove more popular than having to keep changing mode based on context, performance / design. This may be especially true during rehearsal sessions where a mix of performance and design may occur.

6

Hmm, looks like I missed the “Normal Mode” description, which would perform like you describe.

  1. Live Mode: Read-Only, changes saved on request only. Safest for SD cards
  2. Normal Mode: Read-Only, changes saved on request, and at shutdown. Mostly safe for SD cards.
  3. Designer Mode: Read-write, changes saved immediately. Not safe for SD cards, unexpected power down may cause corruption.

I am suspicious that some of the problems caused by unexpected power down are related to the swapfile. The proposed changes disable use of the swapfile.

7

Ah! I thought swapfile was already disabled. Flash memory has limited write cycles and a flash memory based system like the RPi should avoid filesystem writes where ever possible. A swapfile is a bad idea. It will cause excessive wear on the flash and, as part of the wider filesystem increases the risk of filesystem corruption. A swap partition may avoid that latter risk but we should, if possible, illuminate swap. This may constrain the synthesis and processing capabilities because we only have 1GB RAM. (I remember thinking 16KB was luxury!)

1 Like
8

I have it working, and the instructions above are up to date. I’ve forked the original Domoticz scripts to customize for a Zynthian install.

4 Likes
9

After a bit of use, I have a few more optimizations. I’m going to add /etc to the list of overlays so that webconf and wifi enable/disable work as expected. I think I might also make some improvements with the logfiles, possibly use ramlog? I’m not happy with how long it takes to shut down a system that should have had no changes.

2 Likes
11

Many thanks @smiths73v3 for this nice guide!

I am using this on Aruk-RC3 and hope to get away those annoying defect filesystems I have often exactly when using Zynthian with my band (hello Mr Murphy!).

Yes, that’s a little bit annoying, too. But would it be much better to disable logging by default or from webconf?

Regards, Holger

12

Hi @smiths73v3!

Normally you can poweroff your zynthian unpluging the cable. I do that several times a day and i didn’t have a corrupted SD in years!! Simply be careful of not doing it while it’s recording audio/MIDI to the SD-card or updating the software, etc.

Regards,

1 Like
13

Thanks @jofemodo,

Yeah, It should work just fine most of the time.

For a device I want to use Live, it’s got to be as reliable as possible. This is a known issue with raspberry pi devices. Folks including myself have seen evidence of the unexpected power off or crash corrupting an SD card. Before I take this Live, I want to make sure my power off after “sound check” does not kill my Zynth for the show.

1 Like
14

If no write operations while powering off, no SD corruption. So, simply avoid this write operations:

  • Internal Audio/MIDI recording to SD
  • Software Updates
  • Modifying configuration from Webconf / Admin Menu
  • Saving Snapshots (very very short write!)

Of course, having a fast SD card will help to reduce the write time, so reducing the time your SD is exposed to corruption.

Anyway, if you really want to be safe on stage, i would recommend to use a battery pack. It’s cheap and quite reliable. Also, some battery packs can be connected to power while working, so would act as a kind of SAI/UPS.

Regards,

1 Like