stable URL: /l/refresh-tokens

Handling secrets

How to create, rotate and destroy secrets

This page has an internal companion page which contains additional information.

Problem

You want to

add a new secret to the secrets store
remove an obsolete secret from the secrets store
rotate a time-limited secret because its expiration date is looming

Checking prerequisites

Steps:

Make sure you have a current deployment-all checkout, and VAULT_ADDR set up as described in the README.md file.
Make sure you have access to a working cki-tools installation, either via
- installation into you user account
- installation via direnv
- a Podman container with the cki-tools image launched by something like
```
podman run --interactive --tty --rm \
  --env VAULT_ADDR \
 --volume ~/.vault-token:/cki/.vault-token \
 --volume .:/data --workdir /data \
  quay.io/cki//cki-tools:production
```
- a toolbx container based on the cki-tools image created via something like
```
toolbox create --image quay.io/cki/cki-tools:production
```
Log into HashiCorp Vault via
```
cki_secrets_login --oidc
```

Managing pre-existing secrets

The following steps use the secrets module of the credentials helper from cki-tools.

Adding pre-existing secrets to the secrets store

Steps:

Secrets can be added interactively; for a new secret SECRET_NAME, the following command will open a text editor that allows you to enter the secret data:
```
cki_edit_secret SECRET_NAME
```
Quit the editor with a nonzero exit code to abort the operation, e.g. :cq in vi.

This will create the secret value in HashiCorp Vault, and add a reference to it together with some default meta data in the secrets.yml file:
```
SECRET_NAME:
  backend: hv
  meta:
    active: true
    created_at: '2025-01-22T10:54:04+00:00'
    deployed: true
```

Additional meta data can be added either via editing secrets.yml directly, or by calling cki_edit_secret via something like

cki_edit_secret SECRET_NAME#meta-data-key meta-data-value

This results in the following secrets.yml entry:

SECRET_NAME:
  backend: hv
  meta:
    active: true
    created_at: '2025-01-22T10:54:04+00:00'
    deployed: true
    meta-data-key: meta-data-value

After adding the secret, commit the changes to the secrets.yml file and submit the MR.

Removing secrets from the secrets store

Steps:

Completely remove any reference to the secret from the secrets.yml file:

# remove all meta data from the secrets.yml file
cki_edit_secret SECRET_NAME# ''
# remove the reference to the secret from the secrets.yml file
cki_edit_secret SECRET_NAME: ''

Commit the changes to the secrets.yml file and submit the MR.

Do not proceed with the next step until the MR has been merged!
This step is optional, and can be done in bulk after multiple secrets have been removed from the secrets.yml file.

Remove the secret from HashiCorp Vault with the vault executable via
```
vault kv metadata delete apps/cki/SECRET_NAME
```

Handling managed secrets

The following steps use the manager module of the credentials helper from cki-tools.

Creating managed secrets

In contrast to the pre-existing secrets workflow above, managed secrets are created via the credential-manager, and require the meta data to exist in the secrets.yml file before the secret can be created.

Steps:

Check the credential-manager documentation to make sure the token type supports the create operation,
From the credential-manager documentation, find the section about the token type you want to create, and create all meta data fields marked as required. As an example, for an SSH key, the meta data might look like
```
SERVER_SSH_KEY:
  comment: server-name
  key_size: 4096
  token_type: ssh_private_key
```

Use the credential manager to create the actual SSH key and submit it to HashiCorp Vault:

python3 -m cki_tools.credentials.manager create --token-name SERVER_SSH_KEY

Inspect the generated secret meta data in the secrets.yml file, and the actual secret in HashiCorp Vault via

$ cki_secret SERVER_SSH_KEY# --json
{
  "active": true,
  "comment": "server-name",
  "created_at": "2025-01-22T10:54:04+00:00",
  "deployed": true,
  "key_size": 4096,
  "token_type": "ssh_private_key"
}
$ cki_secret SERVER_SSH_KEY: --json
{
  "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----\n....",
  "public_key": "ssh-rsa AAAAB3NzaC1yc2E...."
}

Additional meta data can be added either via editing secrets.yml directly, or by calling cki_edit_secret as described above.
Commit the changes to the secrets.yml file and submit the MR.

Removing managed secrets

Steps:

Check the credential-manager documentation to make sure the token type supports the destroy operation,
Destroy the secret on the external service (if applicable) and mark it as inactive via
```
python3 -m cki_tools.credentials.manager destroy --token-name SERVER_SSH_KEY
```
This flips the active field to false in the secrets.yml file:
```
$ cki_secret SERVER_SSH_KEY#active
false
```
The secret can be removed from HashiCorp Vault as described above.

Rotating managed secrets

To be able to rotate secrets without downtime, multiple version of the secret need to exist at the same time, and must be configured correctly on both the service providing the secret and on the service consuming the secret.

In general, the service providing the secret needs to be configured to provide the deployed version of the secret, while the service consuming the secret needs to be configured to accept all the active versions of the secret.

Examples:

`token_type`	provider: `secret[deployed]`	consumer: `secret[active]`
`dogtag_certificate`	UMB `connect()`	handled automatically
`gitlab_project_token`	Bearer token	approval rule user ids
`ssh_private_key`	`ssh-add -`	`~/.ssh/authorized_keys`

Steps:

Check the credential-manager documentation to make sure the token type supports either the rotation operation or both the create and destroy operations.
Prepare a suitable second version of a secret via
```
python3 -m cki_tools.credentials.manager prepare --token-name SECRET_NAME [--force]
```
Use --force if the secret is not yet close to expiry, but it still needs to be rotated.

If necessary, this will create a new active secret version in HashiCorp Vault, and add a reference to it together with the appropriate default meta data in the secrets.yml file.

If a new secret version was created, commit the changes to the secrets.yml file and submit the MR.

Do not proceed with the next step until the MR has been merged and deployed successfully!
Switch to the new secret version via
```
python3 -m cki_tools.credentials.manager switch --token-name SECRET_NAME
```
This will mark the new secret version as deployed: true, and mark any other versions as deployed: false.

Commit the changes to the secrets.yml file and submit the MR.

Do not proceed with the next step until the MR has been merged and deployed successfully!
Clean up any superfluous secret versions via
```
python3 -m cki_tools.credentials.manager clean --token-name SECRET_NAME
```
This will destroy the superfluous versions on the external service (if applicable) and mark them as inactive via active: false.

If secret versions were destroyed, commit the changes to the secrets.yml file and submit the MR.

Do not proceed with the next step until the MR has been merged and deployed successfully!
Optionally, purge any inactive secret versions via
```
python3 -m cki_tools.credentials.manager purge --token-name SECRET_NAME
```
Again, commit the changes to the secrets.yml file and submit the MR.

Additionally, the secret can be removed from HashiCorp Vault as described above.