Using AWS SSM Parameter Store With Git SSH Keys

and employing them securely

At Archer, we have been moving credentials into AWS Systems Manager (SSM) Parameter Store and AWS Secrets Manager. One of the more interesting credentials is an SSH key that is used to clone a GitHub repository into an environment that has IAM roles available (E.g., AWS Lambda, Fargate, EC2).

We’d like to treat this SSH private key as a secret that is stored securely in SSM Parameter Store, with access controlled by AWS IAM, and only retrieve it briefly when it is needed to be used. We don’t even want to store it on disk when it is used, no matter how temporarily.

After a number of design and test iterations with Buddy, here is one of the approaches we ended up with. This is one I like for how clean it is, but may not be what ends up going into the final code.

This solution assumes that you are using bash to run your Git commands, but could be converted to other languages if needed.

Using The Solution

Here is the bash function that retrieves the SSH private key from SSM Parameter Store, adds it to a temporary(!) ssh-agent process, and runs the desired git subcommand using the same temporary ssh-agent process:

git-with-ssm-key()
{
  ssm_key="$1"; shift
  ssh-agent bash -o pipefail -c '
    if aws ssm get-parameter \
         --with-decryption \
         --name "'$ssm_key'" \
         --output text \
         --query Parameter.Value |
       ssh-add -q -
    then
      git "$@"
    else
      echo >&2 "ERROR: Failed to get or add key: '$ssm_key'"
      exit 1
    fi
  ' bash "$@"
}

Here is a sample of how the above bash function might be used to clone a repository using a Git SSH private key stored in SSM Parameter Store under the key “/githubkeys/gitreader”:

git-with-ssm-key /githubsshkeys/gitreader clone git@github.com:alestic/myprivaterepo.git

Other git subcommands can be run the same way. The SSH private key is only kept in memory and only during the execution of the git command.

How It Works

The main trick here is that ssh-agent can be run specifying a single command as an argument. That command in this case is a bash process that turns around and runs multiple commands.

It first gets the SSH private key from SSM Parameter Store, and adds the key to the ssh-agent process by passing it on stdin. Then it runs the requested git command, with the ssh-agent verifying identity to GitHub using the SSH private key.

When the git command has completed, the parent ssh-agent also disappears, cleaning up after itself.

Note: The current syntax doesn’t work with arguments that include spaces and other strange characters that might need quoting or escaping. I’d love to fix this, but note that this is only needed for commands that interact with the remote GitHub service.

Setting Up SSM Parameter Store

Now let’s go back and talk about how we might set up the AWS SSM Parameter Store and GitHub so that the above can access a repository.

Create a new SSH key with no passphrase (as it will be used by automated processes). This does go to disk, so do it somewhere safe.

keyname="gitreader" # Or something meaningful to you
ssh-keygen -t rsa -N "" -b 4096 -C "$keyname" -f "$keyname.pem"

Upload the SSH private key to SSM Parameter Store:

ssm_key="/githubsshkeys/$keyname" # Your choice
description="SSH private key for reading Git" # Your choice

aws ssm put-parameter \
        --name "$ssm_key" \
        --type SecureString \
        --description "$description" \
        --value "$(cat $keyname.pem)"

Note: The above uses the default AWS SSM key in your account, but you can specify another with the --key-id option.

Once the SSH private key is safely in SSM Parameter Store, shred/wipe the copy on the local disk using something like (effectiveness may vary depending on file system type and underlying hardware):

shred -u "$keyname.pem" # or wipe, or your favorite data destroyer

Setting Up GitHub User

The SSH public key can be used to provide access with different Git repository hosting providers, but GitHub is currently the most popular.

Create a new GitHub user for automated use:

https://github.com/

Copy the SSH public key that we just created

cat "$keyname.pem.pub"

Add the new SSH key to the GitHub user, pasting in the SSH public key value:

https://github.com/settings/ssh/new

Do not upload the SSH private key to GitHub. Besides, you’ve already shredded it.

Setting Up GitHub Repo Access

How you perform this step depends on how you have set up GitHub.

If you want the new user to have read-only access (and not push access), then you probably want to use a GitHub organization to own the repository, add the new user to a team that has read-only access to the repository.

Here’s more information about giving teams different levels of access in a GitHub organization:

https://help.github.com/articles/about-teams/

Alternatively, you can add the new GitHub user as a collaborator on a repository, but that will allow anybody with access to the SSH private key (which is now located in SSM Parameter Store) to push changes to that repository, instead of enforcing read-only.

Once GitHub is set up, you can go back and use the git-with-ssm-key command that was shown at the start of this article. For example:

git-with-ssm-key "$ssm_key" clone git@github.com:MYORG/MYREPO.git

If you have given your GitHub user write access to a repo, you can also use the push and related git subcommands.

Cleanup

Once you are done with testing this setup, you can clean up after yourself.

Remove the SSM Parameter Store key/value.

aws ssm delete-parameter \
  --name "$ssm_key"

If you created a GitHub user and no longer need it, you may delete it carefully. WARNING! Make sure you sign back in to the temporary GitHub user first! Do not delete your main GitHub user!

https://github.com/settings/admin

When the GitHub user is deleted, GitHub will take care of removing that user from team membership and repository collaborator lists.

GitHub vs. AWS CodeCommit

For now, we are using GitHub at our company, which is why we need to go through all of the above rigamarole.

If we were using AWS CodeCommit, this entire process would be easier, because we could just give the code permission to read the Git repository in CodeCommit using the IAM role in Lambda/Fargate/EC2.

Original article and comments: https://alestic.com/2018/12/aws-ssm-parameter-store-git-key/