How to Secure Sensitive Data in Configuration

In the blog post “How to Configure .Net Core Environments With Practical Examples”, we shared the .NET Core Environment Configuration with the appsettings.json file.

As you can see, the appsettings.json file helps us to store the configuration, the setting, etc. However, there is a big concern that all information on this file can be accessed by anyone who has the source code or the server that hosted the app. For this reason, it’s not a good place to store sensitive data.

But what is sensitive data? Sensitive data is anything that can be potentially exploited by a third party. For example, API keys, connection strings, tokens, username, emails, passwords, password hashes, potentially some URLs, etc. In this article, I would like to share with you how to secure sensitive data in local and production environments.

How to secure sensitive data in the development environment (Locally)

In the development environment, sensitive data should not be stored in configuration files, much less in the source code itself. But it should be easy to be updated.

You can choose either to store your configuration in the environment variables or as a user secret by using the Secret Manager. Neither of these options is good enough for production since they store your data as plain texts instead of encrypting the data. However, you can use them to avoid conflicts while working with other developers on a project. By doing this, you don’t have to worry if secrets and environment variables are committed by mistakes.

Environment variables.

According to Wikipedia, an environment variable is a dynamic-named value that can affect the way running processes will behave on a computer.

For Windows in CMD: you can use the “set” command to set an environment variable with the syntax like this:

 set [variable name]=[value]

 You can check the full options for set commands.

Example:  

set ConnectionStrings_ProjectConnectionString=server=\\DEV-SERVER; database=DatabaseName; User Id=myaccount; Password=mypassword;

And then, you can use “set” without any arguments to list all environment variables.

For  PowerShell: 

[System.Environment]::SetEnvironmentVariable('[variable name]','[value]',[System.EnvironmentVariableTarget]::User)

Example: The same command with the above CMD is: 

[System.Environment]::SetEnvironmentVariable('ConnectionStrings_ProjectConnectionString','server=\\DEV-SERVER; database=DatabaseName; User Id=myaccount; Password=mypassword;',[System.EnvironmentVariableTarget]::User)

You can check your variable with the following:

[System.Environment]::GetEnvironmentVariable('[variable name]','user')

If you need to read the environment variable programmatically, please refer to “How to Configure .Net Core Environments With Practical Examples”. It also includes a tutorial to manage environment variables with the Windows GUI.

Secret Manager

.NET Core provides a secrets manager tool to store sensitive data during development. This tool allows you to store your sensitive data locally on your machine, outside the project tree.

To start using secrets, you need to enable them. Go to the project directory and enter the command:

dotnet user-secrets init

As a result, the command creates an entry in the .csproj file:

Now to create or modify a secret, you can use the dotnet user-secrets set command as follows:

	dotnet user-secrets set "ConnectionStrings_ProjectConnectionString" "server=\\DEV-SERVER; database=DatabaseName; User Id=myaccount; Password=mypassword;"

And then use the below “list” command to check if you’ve successfully added it:

	dotnet user-secrets list

Here is the result:

To access the user secret in the .NET Core project, you will need to add some additional codes for ConfigurationBuilder as below:

To remove a secret, you can use “dotnet user-secrets remove [secret key]”, and use “dotnet user-secrets clear” to remove all.

But if you are using Visual Studio, you can easily manage the user secret by GUI. Just right-click on the project and choose “Manage User Secrets”.

The secrets.json file will be opened with all your secret settings. Now, you can edit and save it just like the “appsettings.json” file.

This secrets.json file is not super secure, and the keys are not encrypted. However, it provides an easy way to avoid storing secrets in your project config files and having to remember to add them to the source control ignore list.

Using the Azure Key Vault to secure sensitive data in the production environment

Azure Key Vault is a cloud service for securely storing and accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, or cryptographic keys. You can change the secret directly from the Azure portal, and no need to redeploy your application.

Setting up Azure key vault

To set up a key vault on the Azure portal, you need to follow the following steps:

1. From the Azure portal menu or the Home page, select Create a resource.

2. In the search box, enter Key Vault.

3. From the results list, choose Key Vault.

4. On the Key Vault section, choose to Create.

5. On the Create key vault section, provide the following information.

• Subscription: choose a subscription.
• Resource group: choose to create a new one and enter a resource group name.
• Key vault name: a unique name is required.
• Region: choose your region.
• Pricing tier: check the pricing for the Azure key vault.

Note: Leave the other options to their defaults.

6. Click on “Review + Create” to finish your setting.

Now, let’s add your secrets. You can refer to this Azure Quickstart for the details of how to create an Azure key vault and add a secret with the Azure portal.

Connecting the .NET Core app with Azure key vault

To connect your .NET Core app with the Azure key vault, you will need to do some further steps:

• First, you need to install some NuGet packages as below:
  • Microsoft.Azure.KeyVault
  • Microsoft.Azure.Services.AppAuthentication
  • Microsoft.Extensions.Configuration.AzureKeyVault
• Next, you need to add some configuration to the Program.cs file.

The “https://contoso-vault2.vault.azure.net/” is the “DNS Name” on your Azure key vault overview page.

If your application is hosted on Azure. You can easily manage access to the Azure key vault on the Azure portal under “Access policies”.

If you don’t use Azure, you can follow this Microsoft document to config the IP firewall or certificate to connect the Azure key vault from outside of the Azure network.

Once the app is connected to the Azure key vault, you can update the Azure Key Vault without updating and redeploying the application. 

However, you need to restart the app to refresh the Key Vault cache since the application caches the secret from the key vault on the application startup. Though, it is not a good idea. 

Fortunately, we can specify a ReloadInterval to reload the secrets from the Key Vault whenever the Reload Interval duration is over. Just update the config for the above Azure key vault like this:

Using AWS Systems Manager Parameter Store

Like Azure Key Vault, AWS Systems Manager Parameter Store (let's call it AWS Parameter Store) provides secure, hierarchical storage for configuration data management and secret management. Data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes can be stored as parameter values. And values can be stored as plain text or encrypted data.

Setting up AWS Parameter Store

To set up AWS Parameter Store, you need to take the following steps:

1. Go to AWS Management Console > find the “System Manager” service.

2. On the “Systems Manager” screen, choose the “Parameter Store” under “Shared Resources.”

3. Click on the “Create Parameter” button.

4. On the Create Parameter Store screen, enter the name, parameter type, and  KMS info > click on the “Create Parameter” button.

5. Now, add your secrets to AWS Parameter Store. If you use the AWS console, refer to this Create a Systems Manager parameter (console). If you use Windows PowerShell, use the command below:

Write-SSMParameter `
    -Name "parameter-name" `
    -Value "parameter-value" `
    -Type "SecureString"  `
    -KeyId "a KMS CMK ID, a KMS CMK ARN, an alias name, or an alias ARN" `
    --tags "Key=tag-key,Value=tag-value" 

Connecting the .NET Core app with AWS Parameter Store

First, make sure that you install Amazon.Extensions.Configuration.SystemsManager NuGet package, which is a configuration provider for the .NET Core configuration system to connect your .net core app with AWS Parameter Store.

Next, like the Azure Key Vault, you need to update your app configuration to use the AWS Parameter Store provider. Take a look at the below picture to see how to update your Program.cs file:

public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration((context, config) =>
                {
                    if (context.HostingEnvironment.IsProduction())
                    {
                        var azureServiceTokenProvider = new AzureServiceTokenProvider();


                        var keyVaultClient = new KeyVaultClient(
                            new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));




                        var keyVaultOptions = new AzureKeyVaultConfigurationOptions
                        {
                            Client = keyVaultClient,
                            Manager = new DefaultKeyVaultSecretManager(),
                            ReloadInterval = TimeSpan.FromMinutes(5) // Reload in a minute
                        };


                        config.AddAzureKeyVault(keyVaultOptions);
                    }


                })
                .ConfigureAppConfiguration(builder =>
                {
                    builder.AddSystemsManager("/myapplication");
                })
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                });

Here, we add all of the parameters from the Parameter Store starting with “/myapplication” we create on the setup AWS Parameter Store step.

Like the Azure Key Vault, we also can set the reload interval time, certificate, caching, etc. You can check the details in the .NET Core configuration provider for AWS Systems Manager.

Creating custom configuration providers

Besides the existing configuration providers, and 3rd providers like Azure Key Vault, AWS  Parameter Store, etc., we can build custom configuration providers. 

Many software architectures recommend storing configuration sources that reside in separate stores to decouple the configurations from the application. Then, send an HTTP request or query the database to load those configurations. This matter, custom configuration providers will help us to do it easily. But how to create a custom provider? Check this out!

To create a custom provider, we need to implement the IConfigurationSource and IConfigurationProvider interfaces. 

Create configuration source

Preparing the required information to build a configuration provider is the main responsibility of a configuration source. A configuration source needs to implement the IConfigurationSource interface. This interface requires us to implement the IConfigurationProvider Build (IConfigurationBuilder builder) method.

Below is an example of an implementation of a configuration source. This MyCustomConfigurationSource class has two properties, including ConnectionString and Query. They are required to determine the SQL Server database connection and the key-value pairs retrieval query.

using Microsoft.Extensions.Configuration;


namespace WebApplication.Configurations{


    public class MyCustomConfigurationOptions    {


        public string ConnectionString { get; set; }


        public string Query { get; set; }
    }
    public class MyCustomConfigurationSource : IConfigurationSource    {


        public string ConnectionString { get; private set; }


        public string Query { get; private set; }


        public MyCustomConfigurationSource(MyCustomConfigurationOptions options)
        {
            ConnectionString = options.ConnectionString;


            Query = options.Query;
        }
        public IConfigurationProvider Build(IConfigurationBuilder builder)        {


            return new MyCustomConfigurationProvider(this);
        }
    }
}

Create configuration provider

Loading the configuration data from a configuration source is the main responsibility of a configuration provider. A configuration provider must implement the IConfigurationProvider interface. However, the fact is the configuration provider class can inherit from the ConfigurationProvider base class, which has implemented all methods in the IConfigurationProvider interface. Simply by doing this, we can make use of available codes and skip some overlapped steps in implementing many commonly used methods, such as a method to set a value for a key, a method to find a value with a given key, etc.

The simple custom provider to load the setting on the SQL Server database will be like this:

using Microsoft.Extensions.Configuration;
using System.Data.SqlClient;
namespace WebApplication.Configurations{


    public class MyCustomConfigurationProvider : ConfigurationProvider{


        private MyCustomConfigurationSource Source { get; }


        public MyCustomConfigurationProvider(MyCustomConfigurationSource source){


            Source = source;


        }


        public override void Load(){


            using (var connection = new SqlConnection(Source.ConnectionString)){


                connection.Open();


                using (var command = new SqlCommand(Source.Query, connection)){


                    using (var reader = command.ExecuteReader()){


                        while (reader.Read()){


                            Set(reader.GetString(0), reader.GetString(1));

                        }

                    }

                }

                connection.Close();
            }

        }

    }

The constructor of the MyCustomConfigurationProvider class takes an instance of MyCustomConfigurationSource as the parameter. We override the Load() method with our custom implementation from lines 17 to 31. In this method, we query the database using the native .NET SQL Server client from the System.Data.SqlClient NuGet package. We can further add some codes to retry the database connection, handle exceptions, and decrypt the encrypted configuration data if needed.

Add the custom configuration provider to the .NET core configuration system.

Once the configuration source and the configuration provider are ready to use, we can add them to the .NET core configuration system. Commonly, we use an extension method to add the configuration source to the configuration builder.

using Microsoft.Extensions.Configuration;
using System;


namespace WebApplication.Configurations
{
    public static class ConfigurationExtensions
    {
        public static IConfigurationBuilder AddMyCustomConfiguration(this IConfigurationBuilder configuration, Action<MyCustomConfigurationOptions> options)
        {
            if (options == null)
                throw new ArgumentNullException(nameof(options));


            var configurationOptions = new MyCustomConfigurationOptions();


            options(configurationOptions);
            configuration.Add(new MyCustomConfigurationSource(configurationOptions));


            return configuration;
        }
    }
}

And then use it.

public Startup(IWebHostEnvironment environment)
        {
            var builder = new ConfigurationBuilder();
            builder.SetBasePath(environment.ContentRootPath)
            .AddJsonFile("appsettings.json", false, true)
            .AddMyCustomConfiguration(options =>
            {
                options.ConnectionString = Environment.GetEnvironmentVariable("DEV_DB_CONNECTION_STRING");
                options.Query = "SELECT [Key],[Value] FROM [dbo].[Settings]";
            })
            .AddEnvironmentVariables();


            if (environment.IsDevelopment())
            {
                builder.AddUserSecrets<Startup>();
            }


            Configuration = builder.Build();
        }

I stored the SQL Server connection string in the environment variable “DEV_DB_CONNECTION_STRING”. Here is what the database for the Settings table looks like:

One thing to notice here, the setting on the custom provider will override the appsettings.json file if it duplicates because we call the ‘AddMyCustomConfiguration’ method after ‘AddJsonFile("appsettings.json")’.  This rule is the same for the ‘AddEnvironmentVariables’. It’s how the .NET core loads the configuration in case we have multiple configuration providers. The last provider will override all loaded providers. Thus, we must pay attention to the priorities of the providers.

Final thoughts,

As you can see, there are several options for securing sensitive configuration data, each with different levels of security and ease of use. No matter which option you ultimately select, follow these basic guidelines:

• never commit secrets or sensitive data to a code repository, even a private repository
• do not store secrets or sensitive data in source code
• use an encryption mechanism for your data if you can.

References

• Rick Anderson, Kirk Larkin, Daniel Roth, and Scott Addie, Safe storage of app secrets in development in ASP.NET Core, www.docs.microsoft.com, 2020. 
• Environment variable, www.en.wikipedia.org, 2020. 
• set (environment variable), www.docs.microsoft.com, 2017.
• Quickstart: Set and retrieve a secret from Azure Key Vault using the Azure portal, www.docs.microsoft.com, 2019.
• Authenticate to Azure Key Vault, www.docs.microsoft.com, 2020.
• Rick Anderson and Kirk Larkin, Custom configuration provider, www.docs.microsoft.com, 2020.
• AWS Systems Manager Parameter Store, www.docs.aws.amazon.com.