an alkaline blog

Unattended Upgrades and Docker Image Updates on Rasberry Pi OS

fen (historic) -

In This Blog– Some extended documentation on the what and why behind how I've setup the auto updates noted in my github repository, but if you know what you're doing and just want some known-good config for executing docker updates you can go there directly: https://github.com/crayzeigh/pihole-unifi

Documented in this previous blog entry, I configured my RasberryPi to run PiHole for some tracking/ad protection and my Unifi Controller for managing my local network using docker-compose to ease and automate configuration. That's been working beautifully, but updating the images was a manual chore. So was updating any local raspian packages for that matter. I'd tossed auto-updates in the "get to it later" pile, but that meant a lot of getting annoyed at outdated package notifications and then having to find the time that I could interrupt DNS services to the whole house for a few minutes while I got updates completed.

The better answer was to get some automation setup and schedule it for dead times like 3am on a Sunday. If I'm up at that point, I should probably go to bed anyway.

Note: Most of the work here has been completed months ago at time of writing, so I seem to have lost most of the process despite my detailed outline including things like, "the problem with x" or "fix for y"... anway, this process is likely to be incomplete, but I'll try to include known-good configs at least in case someone runs into the same problems.

Requirements

My main goal is for this little server to behave primarily as an appliance, plug it in and forget about it, it should keep working for years without manual intervention. Since Things Happen, though, I also want to be notified when changes occur, just in case. So that leaves us with the following requirements:

  • Check for system updates periodically
  • Install as necessary
  • Reboot as necessary
  • Check for updates to docker images periodically
  • Download and replace as necessary
  • Only do disruptive tasks during quiet hours
  • Notify me on successful (changes) and unsucessful (errors) updates
  • Don't notify me if there are no updates

Email 

Regardless of any other update processes, I'll need someway for the raspberry pi to send emails to me. I could operate an SMTP server locally, but there is a ton to consider here both for security and to avoid bouncebacks and this is antithetical to the entire idea of set and forget. Instead, I configured mailx to use my Gmail account and Google's SMTP servers meaning I just need some configuration entries to get this working and don't need to worry about modern email complications like SPF records.

Before you begin any installation, you'll need to have an app password from Gmail to authenticate to their servers without having to use the standard authentication pages. For information on setting up app passwords, see Google's support documentation.

My outline here says ".mailrc problems" but not what they actually were. As always, I have greatly overestimated my ability to remember something in the future. I'm going to guess I had some challenges getting mailx to work correctly and that I had to do something very specific for the config files. I also want to note that my setup has identical .mailrc files under my user and the root user's home directories presumably to deal with the cron jobs running as root and to test my script, but I don't see why it couldn't be added to /etc/mail.rc as a global setting instead. That is the course I'd recommend, but I have not tested this and have had ".mailrc problems" so... if it doesn't work, in /etc/mail.rc than I'd add it directly to both your home directory and in the root home directory (/root/)as /.mailrc.

Installation & Configuration

  1. Setup an app password for Gmail as documented in the linked support docs
  2. Install mailx through the package manager:
    sudo apt install bsd-mailx
  3. After installing, append /etc/mail.rc with the following configuration making changes for your_username, YourAppPassword and your_email as appropriate:
    set smtp-use-starttls
set ssl-verify=ignore
set smtp=smtp://smtp.gmail.com:587
set smtp-auth=login
set smtp-auth-user=your_username@gmail.com
set smtp-auth-password=YourAppPassword
set from="your_email@gmail.com"

Unattended Updates

Unnattended upgrades is its own package to be installed and configured rather than some OS feature. I made a note here about fixing this for raspberry pi, and doing some internet sleuthing, I'm betting this was the fact that by default, the unattended upgrades configs are assuming a straight Debian build rather than the Raspbian/Raspberry Pi OS. I'll include copies of my config files as a point of comparison.

Also of note, as found elsewhere on the internet, The Raspberry Pi Foundation doesn't use a separate security updates channel from the rest of their package updates. This means keeping your Pi secure, means subscribing to all updates. This fact is why I wanted to ensure I was getting mail notifications.

Installation & Configuration

  1. Install unattended-upgrades from the package manager
    sudo apt install unattended-upgrades

  2. Edit the 20auto-upgrades configuration file to ensure unattended upgrades are running:
    sudo vi /etc/apt/apt.conf.d/20auto-upgrades

    It should look similar to noting that "Unattended-Upgrade" is set to "1":

    APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradable-Packages "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::Verbose "1";
APT::Periodic::AutocleanInterval "7";

  3. Edit the 50unattended-upgrades config to ensure the proper packages are being downloaded and adjust any options you'd like personally:
    sudo vi /etc/apt/apt.conf.d/50unattended-upgrades

    Be sure the following lines are included and uncommented (I don't recommend deleting yours to replace with this, there's a lot of good information contained in the files comments, but I want to save some space here with the minimal text required):

    Unattended-Upgrade::Origins-Pattern {
    "origin=${distro_id},codename=${distro_codename},label=${distro_id}"
    "origin=Raspberry Pi Foundation,codename=${distro_codename},label=Raspberry Pi Foundation"
};

Unattended-Upgrade::Mail "youremail+pi@domain.com";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:00";

  4. Optional: Make any additional changes to 50unattended-upgrades as needed or desired. If you check the included file you'll note options for prohibiting updates on certain packages (if you need a specific version for a dependency for instance) or if you want to only recieve emails on errors.

  5. Recommended: test your config and email by running manually with debug options: sudo unattended-upgrades -d or as a dry run: sudo unattended-upgrades --dry-run

Once you're all set with the above and your debug or dry-run works, you should be all set to walk way and ensure that you'll get an email any time packages are updated on your system. I have my reboot time set for 03:00 to ensure restarts only happen when no one needs the connection regardless of when unattended updates run or complete. Setting the time also guarantees that even if something goes wrong and updates stick or run into normal awake hours, reboots won't happen until 03:00 the next day.

Automating Docker Image Updates

At this point, we've got email working and the rest of our packages upgrading, but with our main packages of concern running inside docker containers, they won't be affected by any of that. But that's OK, we chose containers on purpose because updates could be done in place with minimal disruption. Since all of the relevant permanent files are stored outside the containers, we can easily download updated images and replace containers like nothing ever happened. The binaries will all be new, but persistent data will be the same.

Manually the process is something like:

  1. Pull down the latest images with docker-compose -f /path/to/docker-compose.yml pull
  2. Bring up the new/changed containers in detached mode with docker-compose -f /path/to/docker-compose.yml up -d
  3. Clean-up any unused images that are still hanging around after the process with a docker prune -af

That should be easy enough to script, but I also want to recieve emails any time a container is rebuilt especially since this is managing my wifi and DNS at home. That could easily be done by offloading the output to a file and emailing that, but I want to minimize disk writes on the SD card storage, so instead of creating and destroying files every time this is run, I want to just keep the text in RAM until it's sent off. So I ended up with the resulting script:

#!/bin/bash

email="you@domain.com"

body="Start: $(date)
Checking for new images...
$(/bin/docker-compose -f /path/to/docker-compose.yml pull --no-parallel 2>&1)

Updating changed containers...
$(/bin/docker-compose -f /path/to/docker-compose.yml up -d 2>&1)

Cleaning up...
$(/bin/docker system prune -af 2>&1)

End $(date)
"

# Only receive emails if an image changed
if [[ ${body} == *"Recreating"* ]]; then
  echo "${body}" | mail -s "Docker Updates Completed" ${email}
fi
https://github.com/crayzeigh/pihole-unifi/blob/main/docker-updates.sh

So I'm still running all the same basic commands, but doing them as a way to generate the multi-line string for the body variable. Storing it all in that variable, and then creating an email as long as somewhere it contains the word "Recreating" (which is what it'll output when it rebuilds a container due to updates).

In hindsight, I probably should also check for errors and mail those, too? But at this point it's good enough for my needs and it's not exactly production infrastructure. If you come up with a tweak for this I'd be greatful for the PR to add that functionality.

In the meantime, feel free to use what I've got:

Installation

  1. Download the script as written:
    wget https://raw.githubusercontent.com/crayzeigh/pihole-unifi/main/docker-updates.sh
  2. Edit the script and replace /path/to/docker-compose.yml with the full path to your docker-compose file.
    vi ./docker-updates.sh
  3. Move the script into the appropriate cron directory for periodic execution:
    sudo cp ./docker-updates.sh /etc/cron.daily
  4. Edit the ownership and permissions to allow execution by the root user during automated execution:
    sudo chown root: /etc/cron.daily/docker-updates.sh
sudo chmod 755 /etc/cron.daily/docker-updates.sh

This is where it's important to note that this script gets executed by the root user and not you. If you were having issues with the mail setup with the config in /etc/mail.rc you may want to add the account settings to /root/mail.rc and see if that fixes that issue.

Conclusion

With all the above you should be up and in "forget about it" mode as far as updates are concerned. You'll get all the latest image updates for Pi-Hole, the Unifi controller, and the Nginx build as well as the latest package updates for what you've got installed directly on your Raspi. You'll want to at least periodically check on the docker images remianing updated since I believe they're all community maintained, but it's nothing that I've had to worry about over the past year.

If you find any issues or something that could be a little better, I'm happy to recieve suggestions on the GitHub Repo.