Creating Databricks secrets via the Databricks CLI
by Nathan Purvis
Background
Databricks secrets are key-value pairs used to store confidential information, such as connection credentials, so that we don’t need to use and store raw values within our Databricks notebooks. Secrets are stored within secret scopes - containers held within an encrypted database hosted, owned and managed by Databricks themself.
Secrets must adhere to the following criteria:
Individual secret names must be unique (within that scope)
Can’t be longer than 128 characters
Only use alphanumeric characters, dashed, underscores, @ symbols and periods
Finally, secret names are case-insensitive so there’s no need to worry about correct capitalisation during creation or subsequent referencing.
Creating a scope
As this blog is covering secret creation and not command line interface (CLI) setup, we will assume that the Databricks CLI is already installed and configured using a personal access token (PAT).
In order to first create a scope, we use the following command in our command line tool:
databricks secrets create-scope --scope <scope name>
Like so:
Now that we have our scope namespace set up, we can then add individual secrets to it with the following command template:
databricks secrets write --scope <scope name> --key <secret name> --string-value <secret value>
Like so - here we are creating a ‘Username’ secret with an underlying value of ‘Nathan’, along with a ‘Password’ with value ‘Supersafepassword123’:
In order to first of all check our scope (or list of scopes) has been created, we can execute:
databricks secrets list-scopes
To then move down a level and query a list of individual secrets within a scope, we can run the following command:
databricks secrets list –scope <scope name>
As shown below, we then see the ‘Username’ and ‘Password’ secrets stored earlier, along with their ‘Last updated’ UNIX timestamp (UTC):
We can also amend and delete secrets (as well as entire scopes), with commands such as:
databricks secrets delete –scope <scope name> –key <secret name>
Like we see here, with the removal of the ‘Username’ secret from the ‘BlogDemo’ scope:
Using our Databricks secrets
Now that we have our secrets stored within a scope, we can then reference them within our Databricks notebooks and they will remain hidden during output and within any logs. This is demonstrated below if we try to print the secrets, which we can call by using ‘dbutils.secrets.get()’:
To use a real example, here is a simple notebook I created to ingest data from Snowflake. As part of the connection details, I pass in the username and password variables from my ‘SnowflakeCreds’ secret scope in order to prevent them appearing in the notebook:
So there you have it! A very quick and (hopefully) simple guide to getting up and running with Databricks secrets via the CLI. As usual, please feel free to provide any feedback, criticisms or suggestions you may have for future content! There are more resources available to cover managing Databricks secrets and bulk adding multiple at a time, including the following links:
https://docs.databricks.com/en/security/secrets/secrets.html
https://docs.databricks.com/en/security/secrets/secret-scopes.html