i’ve had one hell of a time figuring out how to replace the apt_key module in Ansible for automated key and repository management on Debian Bookworm. if you have to grab a key in one format, and the repository is signed with the key in the same format, no problem. but what about when you need to convert a key in one format to another, i.e. you have need to —dearmor it first? Ansible doesn’t provide a straightforward way to move through that workflow with their apt module, and apt_key is deprecated. so how do you deal with this?

if you’re like me and hate reading someone’s backstory for a recipe, you can just jump straight to the solution here. otherwise, enjoy the journey, as i had some troubles and i’m going to tell you all about them.

the problem

once upon a time, you could use the apt_key module in Ansible to quickly grab a signing key and load a repository in the apt sources list. this has changed as recent versions of Debian, and thus Debian-flavored Linux distros, have deprecated the underlying apt key service.

before this occurred, you could just

- name: Add an Apt signing key, uses whichever key is at the URL
  ansible.builtin.apt_key:
    url: https://download.docker.com/linux/debian/gpg

and Ansible would handle the retrieval and delivery of the key under the hood.

however, the apt system has been evolving, and the /etc/apt/trusted.gpg.d directory has been shunted for the new /etc/apt/keyrings. i don’t know the why behind this change, as i haven’t looked into it, but i’m sure there are innumerable other changes linked to this so it likely wasn’t arbitrary.

and if it was arbitrary, well kudos to whoever argued for it because keyrings makes more sense than trusted.gpg.d from an aesthetic point of view.

anyway, the apt_key module general still works to some degree but you’re riddled with warnings when you make use of it, so it’s just better to migrate away from it. as mentioned in the introduction, if the repository uses the signing key in the same format as it is retrieved, then the migration is a simple set of two tasks instead of one:

- name: Retrieve Apt signing key
  ansible.builtin.get_url:
    url: https://download.docker.com/linux/debian/gpg
    dest: /etc/apt/keyrings/docker.asc
    mode: a+r

- name: Add Apt reposotory
  ansible.builtin.apt_repository:
    repo: deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable

in the first task, Ansible retrieves the public key, stores it in the /etc/apt/keyrings directory as docker.asc, and sets the permissions for the file. in the second task, Ansible uses the key to verify the repository and add it to /etc/apt/sources.list.d. so far, so good, right?

this is fine, except there are often cases where you need to retrieve a key in one format, and the repository is signed by the key in a different format. the above two step solution does not work in cases like this, as there are no calls to GPG being made to convert the key to an appropriate form. this is the problem i ran into when i tried to set up keys for VS Code via Ansible:

- name: Retrieve Apt signing key
  ansible.builtin.get_url:
    url: https://packages.microsoft.com/keys/microsoft.asc
    dest: /etc/apt/keyrings/packages.microsoft.gpg #this does not convert the file from .asc to .gpg

- name: Add apt repository
  ansible.builtin.apt_repository:
    repo: deb [arch=amd64,arm64,armhf signed-by=/etc/apt/keyrings/packages.microsoft.gpg] https://packages.microsoft.com/repos/code stable main
    # this throws an error during plaoybook execution

an intervening step needs to be taken between retrieving the key and verifying the repository. the script to install VS Code via command line for Debian looks like this:

wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > packages.microsoft.gpg
sudo install -D -o root -g root -m 644 packages.microsoft.gpg /etc/apt/keyrings/packages.microsoft.gpg

of note is the gpg --dearmor > packages.microsoft.gpg portion. the downloaded file is piped to the gpg command, so only the output is set. seems like all you need to do is run this in a shell and it should be fine, right?

right?!

my temporary solution

wrong. but i’ll get to that in a moment. i didn’t wait to figure things out; i wanted to get these tasks and playbooks up and running, so my temporary solution was hacky at best:

- name: Retrieve Apt signing key # noqa: command-instead-of-module
  ansible.builtin.command:
    cmd: >
      wget -qO- https://packages.microsoft.com/keys/
      microsoft.asc | gpg --dearmor > packages.microsoft.gpg

- name: Move Apt signing key # noqa: command-instead-of-shell no-changed-when
  ansible.builtin.command:
    cmd: sudo install -D -o root -g root -m 644 packages.microsoft.gpg /etc/apt/keyrings/packages.microsoft.gpg

- name: Add apt repository
  ansible.builtin.apt_repository:
    repo: deb [arch=amd64,arm64,armhf signed-by=/etc/apt/keyrings/packages.microsoft.gpg] https://packages.microsoft.com/repos/code stable main

hacky. as. fuck. don’t do this. this is counter to the whole philosophy of idempotency in Ansible. i mean, do what you gotta do—I did—but this was only ever supposed to be temporary while i figured out the other shit.

my problem with the problem

back to the problem at hand. the most frustrating part of this whole process was dealing with a whole lot of incomplete information in every resource i looked up to help me bridge this divide between the two tasks. there are lots of blogs that all say same the same fucking thing with copy/paste jobs from the Ansible docs; lots of suggestions on help sites that kind of answer the issues that arise, but mostly don’t; and neither the Ansible nor GPG docs were helpful at all.

at least for me, of all the resources i perused, it took quite a bit of hobbling ideas together before things clicked and i figured out what works. the biggest part of the problem is that nowhere i looked did i find the appropriate way to script the GPG command so it would work properly. the errors i received from Ansible during playbook execution weren’t helpful either. the only thing i gathered was there was a problem with the way the command was ordered, and as i had zero reference on how to properly write it—seriously, the docs themselves are not specific and the man pages don’t even mention dearmor—it was a major leap of logic on my part before i finally figured out how the script was supposed to work.

finally, i came across this post where someone plainly laid out what --dearmor was looking for with regard to options, something which was largely lacking in other places i looked. sure, examples were abundant, but they were all redundant and not helpful. here was something i could sink my teeth into though:

gpg --dearmor -o output.gpg input.asc

this still wasn’t what i needed, but it gave me a way in. all i needed to do was move things around a bit before Ansible was happy with what it was fed and executed the repository add with the correct signing key and format.

the actual solution

so, the result that i have found (mostly) works is this:

- name: Retrieve gpg key for VS Code
  ansible.builtin.get_url:
    url: https://packages.microsoft.com/keys/microsoft.asc
    dest: /etc/apt/keyrings-temp/microsoft.asc
    mode: a+r

- name: Covert .asc key for VS Code # noqa: command-instead-of-shell no-changed-when
  ansible.builtin.shell:
    cmd: gpg -o /etc/apt/keyrings/packages.microsoft.gpg --dearmor /etc/apt/keyrings-temp/microsoft.asc
    # this is what tripped me up the most: first output, then input

- name: Add apt repository for VS Code
  ansible.builtin.apt_repository:
    repo: deb [arch=amd64,arm64,armhf signed-by=/etc/apt/keyrings/packages.microsoft.gpg] https://packages.microsoft.com/repos/code stable main

Ansible was very specific about wanting the output set before the input in the gpg command, and stringent on where --dearmor was to be placed. took me a few tries to figure that out.

in my workflow, if a key isn’t used in the format it’s retrieved in, i have it saved to a temp directory, converted and copied to /etc/apt/keyrings, then the downloaded key is removed. i’ve found this addresses issues where apt is unsure what key to use to sign the repository despite being told explicitly what key to use.

this workflow isn’t bulletproof. i’m currently running into issues with key management when attempting to add Warp Terminal to the sources list. in its current state, if playbook execution errors out at this point, i can remove the keys and source list, run the playbook again, and it works fine, but this is not ideal. so this solution still requires some tweaks before it’s solid. in most cases, however, i’m finding this is working great. i’ll follow this up with another post once i figure out a better way to execute this process.

if you’ve been trying to find a way around the apt_key module deprecation for Ansible, i hope this helps. and if you have a better way to do this, when dealing with key conversions, i would love to hear it. shoot me skeet on Bluesky!

shameless promotion

this article was borne out of this project:

this is the ephemeralrogue.bookworm Ansible collection, which automates the local provisioning of Debian virtual machines. i wrote about it here.

I’m also working on these babies:

the Design Compendium is a project in collaboration with LVNACY, who designs all the graphics on this blog. i’ve written about this here.

the Hyperion Archive is a Docker Compose-powered instance of AnythingLLM backed by Ollama. not sure where i’m taking this, but at the moment, it’s a quick way to spin up agents to work with and process docs.

the magi is a local three-node elasticsearch instance run via Docker Compose. future updates include migrating the service to minikube for local kubernetes deployment. this will be the first piece in a modular local development system.