Why you should git-commit secrets

Intro

At this point most people have bought into the fact that storing secrets in source code is a bad idea. So what if I told you that you can store secrets in git? In this post I will show you a tool that can help you manage secrets and authorize users access to secrets through the git PR workflow.

Why is storing credentials in git a bad idea?

Before we dive deeper, let’s briefly examine the assumption that storing secrets in source code is bad.

Reason 1: Code is meant to be shared

Source code is often shared with other colleagues since we want to re-use our code, and as we share our code-creations we sometimes inadventently share secrets.

Reason 2: In git, it’s difficult to change the past

Once a secret has been commited to git but then removed it is still there in the git history. There are Saas tools and less friendly enteprises that scan public repositories for high-entropy strings like passwords that have been edited out from source code. Think about that! :)

Reason 3: Secrets aren’t treated as secrets

If developeres aren’t encouraged to think about what is sensitive and is not sensitive, secrets will be shared in slack, printed in logs etc.

How to `git commit` your secrets the right way

So storing secrets in git is a bad idea, and there exists a myriad of tools that offer to store, manage, and rotate secrets. Today we will look at the following toolstack:

Sops for seamless encryption/decryption
Age as encryption/decryption backend that uses public and private keys
Git/Github for approving new users access to secrets

Why sops over “insert tool”

Because it is very easy to use and fits nicely into a git repo.

Sops & Age

Secrets Operations (Sops) is a tool that transparently decrypts and encrypts secrets stored as files on disk.
Age is a simple, modern and secure file encryption tool

Install `age` and `sops`

Sops can be installed like this

# Download the binary
curl -LO https://github.com/getsops/sops/releases/download/v3.9.0/sops-v3.9.0.linux.amd64

# Move the binary in to your PATH
mv sops-v3.9.0.linux.amd64 /usr/local/bin/sops

# Make the binary executable
chmod +x /usr/local/bin/sops

To install age, do this


pacman -S age # For arch


apt install age # for debian/ubuntu

Generate an age key

mkdir -p ~/.config/sops/age/
age-keygen -o  ~/.config/sops/age/key.txt

Note down the age public key that was printed to the terminal. Mine looked like this

# created: 2024-09-29T21:55:13+02:00
# public key: age18g93tkrhudw7flaf6uzr664knn7xnec7p2kh7nyy8x3kud3zmv2qm32cna

This information will be used to encrypt secrets.

Setup `.sops.yaml`

Start by pulling a repo with secrets that need encryption. You can try this repo if you want to code along:

git clone https://github.com/reinthal/swapi.git && cd swapi

First, make your secrets folder.

mkdir -p secrets

Then create a .sops.yaml file at the root of the repo and add your public key

keys:
  - &bob PUBLICKEYGOESHERE
creation_rules:
  - path_regex: secrets/[^/]+\.(yaml|json|env|ini)$
    key_groups:
      - age:
          - *bob

⚠ Reminder:

Do not forget to add your public key to the configuration file!

Now copy the env.example into your clipboard

and put that info into a sops encrypted secrets file

sops secrets/shhh.env

Sops will throw you into your default editor as defined by the EDITOR environment variable. Sops will automatically have populated secrets/shhh.env like this

keys:
# This is an example file.
hello=Welcome to SOPS! Edit this file as you please!
example_key=example_value
example_multiline=foo\nbar\nbaz

Now you can move your sensitive information into this file. For example, add the snowflake credentials from env.exampel to secret/shhh.env. Exit sops and study the file:

cat secrets/shhh.env

DESTINATION__SNOWFLAKE__CREDENTIALS__DATABASE=ENC[AES256_GCM,data:4Dh5H5LHAiFbpMiEa4sWDyu5uFM=,iv:N6O91RvYZtNL4g3lTySh3RzNJdQ9ueuZ27i3VOUatIc=,tag:m6DORkT+xzEhEnQtcJSTeg==,type:str]
DESTINATION__SNOWFLAKE__CREDENTIALS__HOST=ENC[AES256_GCM,data:PTxBAFYlcMwaCbny4hFFTsUEHzecMElw1nKH+KM=,iv:aiOuoClh65xnla75BzAPKsyC0J2Z+QDXe1VcWPki6is=,tag:Rnm7mBhmm8UZDBL0tDDbZQ==,type:str]
DESTINATION__SNOWFLAKE__CREDENTIALS__USERNAME=ENC[AES256_GCM,data:pDAI2MZG6KFgpQ==,iv:levW/HiWDYFOyZtmX3YU8D5EchCgYTlQ2xCTdJIm4mM=,tag:ZS5yYMNrxcRBJYD3lkMLiA==,type:str]
DESTINATION__SNOWFLAKE__CREDENTIALS__PASSWORD=ENC[AES256_GCM,data:Z/eWyttr32y/2B0iNqfQRsWsrQ==,iv:XisGuNzQDYIJVNGwlx7lWtRl8RvStNjfu/qllX54/aA=,tag:jGkNl2moSxDz10GP4AtV9Q==,type:str]
DESTINATION__SNOWFLAKE__CREDENTIALS__ROLE=ENC[AES256_GCM,data:N+zI+6hSmktk7xDizAbz,iv:ySR6KRDqMIoTaCKGFD9xuWJidyZBpDSAmJscIr98M28=,tag:j3cQVYdtassaO94iRUV4+A==,type:str]
DESTINATION__SNOWFLAKE__CREDENTIALS__WAREHOUSE=ENC[AES256_GCM,data:FX+dR4oymeNqneFmcG6nInNwTl0=,iv:iY/lipKXO2Gh3acjA5nPLTvVY46TgKUvFCozYxwCEZg=,tag:L2fbdVb4e3rK4NB7pEcj1A==,type:str]
PROGRESS=ENC[AES256_GCM,data:MxJGBky84RM8,iv:wmtTRD6tPiIlWKqdG/q6Q/tTeZcZa/bLKLXqHQLqNYE=,tag:ODFGLbgxgSbxFg3akDg89g==,type:str]
sops_age__list_0__map_enc=-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBK3ltdFFlMHk3L3pqcUsv\nNzgySDdvOHZlM3FWNGNlaEMxMTVyMFlSOEJ3ClJGeEVoL3pYYVlLbkhXOWpDQnZD\nMy9yWUFoVm42VXkwZzgzTGpKUDEyZlkKLS0tIG9ZdWgvcWdmdTU0Qk9ucDlMT3kw\nSGxjelVDdG81Z0ZFODFNWGRCTEY5aTAKJ/QOdBdn1fV5msIuoXAoIAWSBv6MFmLn\nkLMZl7fcVWoXhEmhf46atRVmo1MDyU4Qd8bFq/tr+/9cEBBBJMrJJA==\n-----END AGE ENCRYPTED FILE-----\n
sops_age__list_0__map_recipient=age18g93tkrhudw7flaf6uzr664knn7xnec7p2kh7nyy8x3kud3zmv2qm32cna
sops_lastmodified=2024-09-29T20:13:05Z
sops_mac=ENC[AES256_GCM,data:QZi2HeXPKH9szNL+Z1V+p6neg3Wc84hdGfP8g7iOX9+eno55JT3UFcGAALY9ixhLV+dulN1KQzDcdeI9Ne34fy74S+JceThWoh4RMZXOhBKTHOFnYHi+POoOlpcEOnI9sRN3aY2wz1tmMngnApZgemZl1auzLSg/9LrhDds4Xok=,iv:veWzIKvg+ota00bPWtDD0ZnGUffJEpmSciLRt03pImw=,tag:2nICufxwKRgGou93G/sQvQ==,type:str]
sops_unencrypted_suffix=_unencrypted
sops_version=3.9.0

Secrets are now encrypted and can only be decrypted by each public keys respective private key as defined in .sops.yaml.

Git commit that secrets/shhh.env and create a pr!

git add .sops.yaml secrets/shhh.env && git checkout -b feat/add-sops && git commit -m 'add sops' && git push origin HEAD

Add a new user

So Alice joins the coding tea-party and wants to start working with secrets/shhh.env. Then, she would do something like this

Alice checks out the repo and installs sops and age
She’ll add her public key and create a PR

Schematic view of Alice Requesting Authorization decrypting secrets using git and sops.

Bob would pick up Alice’s PR and study the diff

 keys:
   - &bob age18g93tkrhudw7flaf6uzr664knn7xnec7p2kh7nyy8x3kud3zmv2qm32cna
+  - &alicce age123123123laf6uzr664knn7xnec123123125y8x3kud3zmv2qm32cna
 creation_rules:
   - path_regex: secrets/[^/]+\.(yaml|json|env|ini)$
     key_groups:
       - age:
           - *bob
+          - *alice

Bob’s like “LGTM”!
Bob runs the following command in the terminal

kog@test-sops:~/swapi$ sops updatekeys secrets/shhh.env
2024/09/29 23:09:51 Syncing keys for file /home/kog/swapi/secrets/shhh.env
The following changes will be made to the file's groups:
Group 1
    age18g93tkrhudw7flaf6uzr664knn7xnec7p2kh7nyy8x3kud3zmv2qm32cna
+++ age1rktgpvuzkq47s89sncwf94d8gpqe9y59gm6gjsla2m48eqrt7y2sg4w0cx
Is this okay? (y/n):y
2024/09/29 23:10:04 File /home/bob/swapi/secrets/shhh.env synced with new keys

which effectively re-encrypts secrets/shhh.env and adds Alice’s public key to the list of keys that can decrypt the secrets. Bob Pushes the new secrets/shhh.env and Approves the PR.

The PR is merged and Alice can now decrypt the secrets.

Schematic view of Bob authorizing Alice’s public key for decryption of secret.s

Discussion

Here are some benefits of using Sops:

Using sops we can leverage the existing workflow of git to manage users access to the secrets within the project.
Just like with documentation, it will be very convenient to keep secrets close to the source code for developers.
Adding and deleting secrets is done simply by opening the secrets file sops secrets/shhh.env
Git creates an audit trail for when developers access was added or removed to the authorized list of public keys.
Keeps things simple for developers.

If course, Sops does not provide automatic key rotation. So that’s a drawback. Neither does it integrate out-of-the-box with Microsoft Entra or other identity providers. But we just want to get developing. The path of least resistance is clearly Sops.

Sops is currently in Cloud Native Sandbox project which means that it is early stage. It integrates with other identity management software keyvaults from all the major cloud vendors.

Summary

We have looked at the problem of adding secrets to git version control system. We solved this issue using Sops.