Secret Storage with Vault

There are times we want our extensions to use configuration data not stored with the code itself. One way to do this is by storing this information in the Q2 Database and accessing it with self.wedge_address_configs in your extension itself. However, this may not be practical if the data is too large or if additional secrecy is needed. For this we turn to a secure key/value store called Vault.

Q2 uses Vault for all kinds of secure data storage, including the credentials the SDK uses to connect to HQ in production and staging. In fact, in production even the SDK developers cannot see the credentials the SDK uses to log into HQ.

Note

More details on Vault itself can be found in Hashicorp’s Vault Docs.

To access data inside Vault, whether it is HQ Credentials or an https certificate, you will use the SDK’s builtin Vault module. This object takes care of authentication and communication to the appropriately configured Vault instance for the environment the SDK is running in. Caliper gets a valid Vault Token when launched in our datacenter, and rotates the token periodically, so you can spend your time writing your extension rather than worry about these low level details.

Typically interaction with Vault takes a running Vault instance, login credentials, token authentication, and a few other pieces. For ease of use during development, the Q2SDK_VAULT_LOCAL_DIR environment variable is set to the ‘vaultdir’ directory in your home folder and the SDK will treat that directory as if it is an active Vault. You can use our Vault helpers to put secrets into the appropriate file system paths.

Here’s an example of pulling data:

Note

These examples assume you have the VAULT_SCOPED_READ variable set to True in your settings file. Prefer using self.vault within the context of an extension.

### python from extension
from q2_sdk.core.vault import StorageLevel

async def default(self):
    # Read secret stored specifically for this stack (tied to the database this
    # is running from)
    # This is the most common. Distinct secrets for each environment this extension
    # is deployed into
    stack_secret = self.vault.read('mysecretkey')

    # Read secret stored at the level of an institution (tied to the
    # settings.COMPANY variable)
    # This might be useful if you have a multitenant extension deployed across
    # several environments that all need the same secret (such as an API key)
    company_secret = self.vault.read('mysecretkey', level=StorageLevel.Institution)

### python raw (outside the context of an extension)
from q2_sdk.core import vault
client = vault.get_client()
response = client.read('mysecretkey')

In a real datacenter environment, the Vault service will be running and serving these secrets. While testing, you can simulate this by using the filesystem with our CLI helper:

# CLI
$ q2 vault create_dirs
$ q2 vault add --filename "mysecretkey" --key "key" --value "value"
$ q2 run

With this, if you hit that route, response will be a dictionary like this {'data': {'key': 'value'}}

Let’s look at using the get_certificate helper. Since the certificate is written to the filesystem in a secure directory, this will require a NOMAD_SECRETS_DIR environment variable to be set to ‘nomadsecrets’ in your home directory:

### python from extension

async def default(self):
    response = self.vault.get_certificate('mycert', self.hq_credentials, as_obj=True)
    cert = response.cert
    key = response.key


### python raw (outside the context of an extension)
from q2_sdk.core import vault
client = vault.get_client()
response = client.get_certificate('mycert', self.hq_credentials)

# CLI
$ q2 vault add_cert --name mycert --cert {cert-filepath} --key {key-filepath}
$ q2 run

Things Vault might be useful for:

  • Certificates

  • Encryption Keys

  • Api Keys

Things Vault might not be suitable for:

  • User data

  • Session Tokens

  • Anything dynamic

Rule of Thumb: If you need to store more than a handful of Vault keys, Vault is probably not the right solution. For example, if you need encrypted data stored for every end user, encrypt it and store the data in the database, then put the encryption key in Vault.

Uploading to Vault

External Developers

Note

To upload to vault:

  • You must have self service access (which you can receive either from your company’s q2developer admin or by making a support ticket on q2developer.com)

  • You need to enable Two-factor Authentication. You can setup Two-factor Authentication here: https://www.q2developer.com/settings/security

First you will login and click the “Self Service” option on the left-hand menu. Under “Self Service” you will chose the “Vault Upload” menu option. There you can choose the name and the upload type and provide your value. Currently, this will only upload to vault and you will have to request a support person to move the vault upload to the proper location for your extension to access.

Internal Developers

To upload to vault please see the guide on confluence here: https://confluence.q2ebanking.com/display/SREArchive/Vault+-+Updating+Secrets