Hosting a signed APT repository

Distributing code for Debian based distributions and derivatives through a PPA can be a little difficult. The following guide will break down the steps and try to explain what is going on. At a high level, you will need a GPG Keypair, somewhere to store the PPA, a machine to do the building and some deb packages to host!

GPG Keyset

For the sake of repeatability I have scripted this out.

#!/bin/bash
set -e

REAL_NAME=$1
EMAIL=$2
PASS_PHRASE=$3

cat > ppa-key <<EOF
     %echo Generating a basic OpenPGP key
     Key-Type: 1
     Key-Length: 4096
     Subkey-Type: 1
     Subkey-Length: 4096
     Name-Real: $REAL_NAME
     Name-Email: $EMAIL
     Expire-Date: 0
     Passphrase: $PASS_PHRASE
     # Do a commit here, so that we can later print "done" 🙂
     %commit
     %echo done
EOF

gpg --batch --generate-key ppa-key
rm -rf ppa-key
echo "$PASS_PHRASE" | gpg --batch --quiet --yes --passphrase-fd 0 --pinentry-mode loopback --export-secret-keys --armor info@albeego.com > ppa-private-key.asc
gpg --export --armor info@albeego.com > KEY.gpg

This script will setup a batch file for creating they key, run the key generation, remove the batch file then export the private key as ppa-private-key.asc and the public key as KEY.gpg. You will need to store these 3 items securely in backup somewhere. If you loose them, you will no longer be able to update your PPA without regenerating the keys, your consumers will see some strongly worded warnings about the validity of your PPA in this case.

The PPA

To build the PPA we will be using apt-ftparchive. This tool will generate the folder structure, cache and files describing the repository structure for apt to consume.

apt-ftparchive.conf

This configuration file is used to specify the structure of your PPA, where are things stored, what compressions to use, supported version and what architectures are available.

Dir {
	ArchiveDir "./debian";
	CacheDir "./cache";
};
Default {
	Packages::Compress ". gzip bzip2";
	Sources::Compress ". gzip";
	Contents::Compress ". gzip";
};
TreeDefault {
	BinCacheDB "packages-$(SECTION)-$(ARCH).db";
	Directory "pool/$(SECTION)";
	Packages "$(DIST)/$(SECTION)/binary-$(ARCH)/Packages";
	SrcDirectory "pool/$(SECTION)";
	Contents "$(DIST)/Contents-$(ARCH)";
};
Tree "dists/bionic" {
	Sections "main";
	Architectures "amd64 armhf arm64";
};
Tree "dists/focal" {
	Sections "main";
	Architectures "amd64 armhf arm64";
};

This configuration will support ubuntu 18.04 and 20.04 for 64 bit x86 systems and raspberry PIs including the new 4 series.

You will need to create the following folders to support the configuration

  1. debian/dists/bionic/main/binary-amd64
  2. debian/dists/bionic/main/binary-arm64
  3. debian/dists/bionic/main/binary-armhf
  4. debian/pool/main
  5. cache
mkdir -p debian/dists/bionic/main/binary-amd64
mkdir -p debian/dists/bionic/main/binary-arm64
mkdir -p debian/dists/bionic/main/binary-armhf
mkdir -p debian/pool/main
mkdir cache

Your .debs need to be copied in to the debian/pool/main directory.

NB: If you are updating the PPA, make sure you include all the previously uploaded .debs too or they will not be indexed

You can now generate the indexes using the following command:

apt-ftparchive generate apt-ftparchive.conf

There will be some files missing in the resulting structure, you will need to add these. They are the Release files. These files correspond to the supported distributions, each one will require a configuration file. In our case bionic.conf and focal.conf

bionic.conf

APT::FTPArchive::Release::Codename "bionic";
APT::FTPArchive::Release::Origin "My repository";
APT::FTPArchive::Release::Components "main";
APT::FTPArchive::Release::Label "Packages hosted by me!!!";
APT::FTPArchive::Release::Architectures "amd64 arm64 armhf";
APT::FTPArchive::Release::Suite "bionic";

focal.conf

APT::FTPArchive::Release::Codename "focal";
APT::FTPArchive::Release::Origin "My repository";
APT::FTPArchive::Release::Components "main";
APT::FTPArchive::Release::Label "Packages hosted by me";
APT::FTPArchive::Release::Architectures "amd64 arm64 armhf";
APT::FTPArchive::Release::Suite "focal";

These configuration files are important, without them, consumers will not find packages in your archive as there will be no indexes for their architecture or distribution

You can now generate the Release files

apt-ftparchive -c bionic.conf release debian/dists/bionic >>debian/dists/bionic/Release
apt-ftparchive -c focal.conf release debian/dists/focal >>debian/dists/focal/Release

The release files will now need signatures attached to attest to the validity and your ownership of these .debs

echo "$PASS_PHRASE" | gpg -u "${PRIVATE_KEY_EMAIL}" --batch --quiet --yes --passphrase-fd 0 --pinentry-mode loopback -abs -o - debian/dists/bionic/Release >debian/dists/bionic/Release.gpg
echo "$PASS_PHRASE" | gpg -u "${PRIVATE_KEY_EMAIL}" --batch --quiet --yes --passphrase-fd 0 --pinentry-mode loopback --clearsign -o - debian/dists/bionic/Release >debian/dists/bionic/InRelease
echo "$PASS_PHRASE" | gpg -u "${PRIVATE_KEY_EMAIL}" --batch --quiet --yes --passphrase-fd 0 --pinentry-mode loopback -abs -o - debian/dists/focal/Release >debian/dists/focal/Release.gpg
echo "$PASS_PHRASE" | gpg -u "${PRIVATE_KEY_EMAIL}" --batch --quiet --yes --passphrase-fd 0 --pinentry-mode loopback --clearsign -o - debian/dists/focal/Release >debian/dists/focal/InRelease

You will notice above that the commands are expecting PASS_PHRASE and PRIVATE_KEY_EMAIL variables to be available in your shell. I use this as part of a script which will be included for your convenience at the end of the article

You now have the Release.gpg files which are detached signatures and the InRelease files which are the Release contents with the signature wrapping the message (attached) at the correct points in the file structure.

Simply upload your debian directory to your target hosting system. I personally used https://www.ovh.co.uk Object Storage, it’s cheap and will support some gigantic .debs if you need them. You could also use github pages as long as none of your .debs are 500Mb or larger and your entire PPA is within their repository size limit

<my_repository>.list

This is the final item to load in to your hosting platform, the .list file call it something sensible for your PPA, led-sys.list would do for me! Its contents should be as follows:

deb http://your-hosting-url bionic main
deb http://your-hosting-url focal main

Consuming your PPA

curl -s --compressed http://your-hosting-url/KEY.gpg | sudo apt-key add -
sudo curl -s --compressed -o /etc/apt/sources.list.d/<my_repository>.list "http://your-hosting-url/<my_repository>.list"
sudo apt update

Make sure you change the URL of the PPA and the name of the .list file to match, you will then be able to apt install your packages from your Signed APT repository

A Full Script for Managing a PPA in an OVH Object Storage container

#!/bin/bash
set -e

STORAGE_CONTAINER_URL=$1
PRIVATE_KEY=$2
PRIVATE_KEY_EMAIL=$3
PASS_PHRASE=$4
PUBLIC_KEY=$5
PROJECT_ID=$6
SWIFT_USERNAME=$7
SWIFT_PASSWORD=$8
REGION=$9
CONTAINER_NAME=${10}
LIST_FILE_NAME=${11}

download_files() {
  swift --os-auth-url https://auth.cloud.ovh.net/v3 --auth-version 3 \
    --os-project-id "$PROJECT_ID" \
    --os-username "$SWIFT_USERNAME" \
    --os-password "$SWIFT_PASSWORD" \
    --os-region-name "$REGION" \
    download "$CONTAINER_NAME" \
    --prefix debian/pool/main/
}

upload() {
  swift --os-auth-url https://auth.cloud.ovh.net/v3 --auth-version 3 \
    --os-project-id "$PROJECT_ID" \
    --os-username "$SWIFT_USERNAME" \
    --os-password "$SWIFT_PASSWORD" \
    --os-region-name "$REGION" \
    upload "$CONTAINER_NAME" "$1"
}

write_key_to_file() {

  KEY="${3//-----BEGIN PGP $1 KEY BLOCK-----/}"
  KEY="${KEY//-----END PGP $1 KEY BLOCK-----/}"

  echo "-----BEGIN PGP $1 KEY BLOCK-----" >"$2"
  printf "%s\n" "$KEY" >>"$2"
  echo "-----END PGP $1 KEY BLOCK-----" >>"$2"
}

write_private_key_to_file() {
  write_key_to_file "PRIVATE" private.key "$PRIVATE_KEY"
}

write_public_key_to_file() {
  write_key_to_file "PUBLIC" KEY.gpg "$PUBLIC_KEY"
}

rm $LIST_FILE_NAME || true

write_private_key_to_file
gpg --import private.key
rm private.key

mkdir -p debian/dists/bionic/main/binary-amd64
mkdir -p debian/pool/main
cp -r *.deb debian/pool/main
download_files
mkdir cache
apt-ftparchive generate apt-ftparchive.conf
apt-ftparchive -c bionic.conf release debian/dists/bionic >>debian/dists/bionic/Release
echo "$PASS_PHRASE" | gpg -u "${PRIVATE_KEY_EMAIL}" --batch --quiet --yes --passphrase-fd 0 --pinentry-mode loopback -abs -o - debian/dists/bionic/Release >debian/dists/bionic/Release.gpg
echo "$PASS_PHRASE" | gpg -u "${PRIVATE_KEY_EMAIL}" --batch --quiet --yes --passphrase-fd 0 --pinentry-mode loopback --clearsign -o - debian/dists/bionic/Release >debian/dists/bionic/InRelease
upload debian
upload cache

wget "$STORAGE_CONTAINER_URL"/$LIST_FILE_NAME || echo "deb $STORAGE_CONTAINER_URL bionic main" >$LIST_FILE_NAME
upload $LIST_FILE_NAME

wget "$STORAGE_CONTAINER_URL"/KEY.gpg || write_public_key_to_file
upload KEY.gpg

rm KEY.gpg
rm debian
rm cache

The above script is ready to go as part of a build pipeline, it will synchronise the object storage to the local machine, copy any .debs to the pool directory and rebuild the indexes, upload everything and tidy up after itself. This is using the OpenStack swift client, any OpenStack compatible Object Storage container will work. Just change the URI for the authorisations in the swift commands.

The whole process is available as a GitHub action here: https://github.com/albeego/apt-repository-action

One thought on “Hosting a signed APT repository

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: