How to Create and Manage Secrets in Kubernetes

Kubernetes Secrets are a built-in resource type that's used to store sensitive data. This blog teaches you how to work with Secrets in Kubernetes.

Kubernetes can do many things, but we usually refer to it as a “container orchestrator.” Orchestrating containers means starting and restarting them when needed, ensuring their configuration matches the declared state, and autoscaling them. But Kubernetes can do much more than that. It can also, for example, manage networking and storage for your pods. Another thing it can do is manage Secrets. In this post, you’ll learn all about working with Secrets in Kubernetes.

What Are Kubernetes Secrets, Anyway?

Let’s start with the basics: what Kubernetes Secrets actually are. If you already have some experience with Kubernetes, you’ve probably heard about pods, namespaces, or deployment. These are examples of a few built-in Kubernetes resources. And Kubernetes Secrets are just another built-in resource type which, as you can guess by the name, are meant to be used to store sensitive data. Because they’re one of Kubernetes's resources, it means that you can work with them just like with any other Kubernetes resource. This means that you can create, list, edit, and delete them, which we’ll do later in this post. But first, a bit more theory.

Secret Types

A “secret” or “sensitive data” could mean many things. Passwords, tokens, certificates, SSH keys, just to name a few. For this reason, there are eight different types of Kubernetes Secrets:

• opaque

• tls

• service-account-token

• dockercfg

• dockerconfigjson

• basic-auth

• ssh-auth

• (bootstrap) token 

But don’t worry. On a daily basis, you’ll only need to remember the two most common types (the first two from the list above). Nevertheless, let’s explore what all of these types are meant to be used for.

Opaque

This one is by far the most common Secret type. It’s basically a general type of Secret that you can use for anything. 

TLS

As the name suggests, this Secret type is meant to store TLS certificates (together with their private keys). This type of Secret is commonly used to encrypt the traffic in your cluster, but you can also use them inside your pod for anything you may need.

service-account-token

This is a special type of Secret. It was initially used for mounting service account credentials to your pods that could be used to access Kubernetes’s API from inside the pod. However, since Kubernetes 1.22, it actually shouldn’t be used that way anymore. Instead, there’s now a new TokenRequest API available for that.

dockercfg and dockerconfigjson

Both of these types are used to store credentials for accessing a Docker container registry. You may need those if you’re using a custom-built Kubernetes cluster with some private internal registry. For managed Kubernetes solutions and their respective Docker registries in the cloud, you usually don’t need to worry about them, since Docker registry authentication is handled differently by your cloud provider.

basic-auth

This type of Secret, as the name suggests, is meant to be used specifically for storing HTTP basic authentication credentials (usernames and passwords). You don’t have to use it, and you could simply use a standard opaque Secret and pass a username and password there. This type is only for convenience and clarity.

ssh-auth

Similar to the one above, this type of Secret is also self-explanatory. It’s for storing SSH credentials (ssh-privatekey), and just like basic-auth one, it’s not mandatory to use it. You also could use opaque for the same purpose, but again, it’s just cleaner to use this one.

(Bootstrap) Token

This is a special type of Secret that you’d only use if you build your own Kubernetes cluster from scratch. It’s used for bootstrapping nodes. So you’ll never have to deal with it if you’re using managed Kubernetes solutions.

Okay, let’s create some Secrets. Just like any other Kubernetes resource, you can manage Secrets either directly using the kubectl command or with a YAML definition file. Let’s start with the first option.

The bare minimum to create a Secret using kubectl is to execute kubectlcreate secret generic followed by a Secret name, just like this:

As you can see, your Secret has been created. However, there’s nothing Secret about it since it doesn’t actually contain any sensitive data. It’s just an empty Secret. So, a bit more realistic example of a simple Secret creation is as follows:

The above command will create a Kubernetes Secret with two keys inside username and password. That’s more useful, right? The --from-literal parameter that we provided is meant to be used to directly specify the Secret data in the command in the plain text, just like we did above. And already, here’s a first tip for you. As you can imagine, this way of creating Secrets is not really secure, because not only do you need to pass the Secret value in plain text, but it’ll also be saved in your shell history. Therefore, you should only use this method for testing or any other not-so-secret use cases. Let’s try a little bit more of a secure approach, shall we?

Instead of specifying secret values directly in the command line, we can instruct the kubectl command to load the values from a file. Let’s say we have a simple file called credentials.txt with the following content:

username=admin

password=super_password

We can then modify the previously used command, and instead of using --from-literal, we can specify --from-file followed by a file path. For example:

The outcome will be the same as the previous command, but as you can imagine, it’s far more secure, since no one looking at your screen will be able to see the actual sensitive data. On top of that, you can further secure the sensitive data with standard Linux file permissions (for example, by restricting read permissions of the file to only a specified group or users).

Now you know how to create basic Secrets with the kubectl create command, but in most day-to-day situations, you’ll probably want to handle them using YAML files, just like the rest of your Kubernetes resources. The structure of that file will be the same as for any other Kubernetes resource. The most important part is to set the kind value to Secret, and the data section will depend on the type of Secret you want to create. In the case of the most common type, opaque, the file will look like this:

And again, just like any other Kubernetes YAML file, you can apply it using the kubectl apply command with the -f parameter followed by the file name:

There’s one thing to note about that YAML file, though. Did you notice that we used the stringData field there? This is to allow us to specify credentials in plain text. If you’d try to use a more common data field, you’d get an error similar to this:

This is because Kubernetes only allows already-base64-encoded Secret values when using the data field. So, in order to comply with that, you’d need to first encode your desired values using base64 binary, like this:

and only then you can use those encoded values together with the data field in the YAML file:

As you can probably guess, this method is a little bit more secure because, again, if someone was looking at your screen while you had the YAML file open, they wouldn’t be able to see the sensitive data in plain text. However, keep in mind that base64 is only a very basic encoding mechanism. It’s not encryption, which means anyone who gets the actual file can very easily decode the values back to plain text.

Working With Kubernetes Secrets

Okay, we created our first Kubernetes Secrets. Now what? Well, we can either consume them in a pod or perform any typical Kubernetes operations such as list, edit, or delete.

Listing Kubernetes Secrets

For starters, let’s see how to list our Secrets, in order to validate that they actually exist. You can do that with the kubectl get secrets command:

Also, keep in mind that Secrets are namespaced, so the command above will only give you Secrets created in the default namespace. If you want to see all Secrets from any namespace, you need to add the -A option to the above command.

Viewing Kuberentes Secret Values

I can imagine that you’d often need to not only see the list of your Secrets, but also their values. If you already have some experience with Kubernetes, you’d probably assume that it’s as easy as executing kubectl describe secret [secret_name], but you’d be only partially right. Let’s take a look:

As you can see, the kubectl describe command indeed gives you detailed information about Secrets, just like it does for other Kubernetes resources. You can see your Secret keys. However, it won’t show you the values of the Secret. Kubernetes is smart enough to avoid showing Secret values by default, because… they should be secret, right? But what if you actually need to see the value, for example, when troubleshooting authentication errors in your pods? You need to ask specifically for the data field of the Secret by passing the -o jsonpath='{.data}' option to the kubectl get secret command. See the example below:

This will return the Secret's values, but there’s a catch again. You can see the values, but not in plain text. They’re supposed to be Secret, remember? That’s another protection mechanism that Kubernetes uses to avoid leaking credentials. But fear no more. Fortunately (and unfortunately at the same time), this is only base64 encoding, which means it’s very easy to get the plain text values. You only need to use the same base64 binary that we used to encode the values before. This time you need to pass the –decode parameter to it, just like this:

Editing Kubernetes Secrets

Okay, so now you know how to see the value of a Secret, but let’s say you spotted a typo in the value of your Secret. How do you fix that? Well, that’s easier than getting their values. Simply execute the kubectl edit secret [secret_name] command or update the value in your YAML definition and reapply it with kubectl apply:

The main difference between those two methods is that with kubectl edit, you’ll have to pass already-base64-encoded values, while with the YAML file, you can just edit their plain text values (if you used the stringData method shown earlier).

Immutable Secrets

Speaking of editing Secrets, there’s something you should know. If you want to prevent someone from editing the value of your Secret, you can mark it as immutable. This will make it read-only. Kubernetes won’t allow you to edit the value of an immutable Secret.

In order to create an immutable Secret, you only need to add immutable: true to your Secret YAML definition file:

We can now apply such a file:

Then if you try to edit that Secret, for example, with kubectl edit secret myfourthsecret, you’ll see an error message similar to this one:

Using Kubernetes Secrets

Alright, we already created five Secrets, so it’s time to make some use of them. There are two main ways of loading a Secret into a pod, either as environment variables or as files.

The first option is a little bit more common, because it’s a little easier to use. This is because most applications can load Secrets from environment variables natively, so you usually don’t need to change anything in the application code to consume them.

On the contrary, the second option, in some cases, may require code changes to the application. However, Secrets mounted as volumes have one important advantage over those loaded as environment variables. If you change the value of a Secret while a pod consuming it is already running, the environment variable Secret won’t get updated automatically (you’ll have to restart the pod), while the volume-mounted one will. With that being said, let’s see how to use both methods.

Secret as an Environment Variable

We’ll start with mounting a Secret as an environment variable. To do that, you need to add the envFrom: parameter to the container section of your YAML definition. The envFrom: parameter offers a few different options for loading environment variables, and for our use case, we need to specify secretRef: and then provide a Secret name, like this:

Just keep in mind that name in the above snippet refers to the name of the Secret object, not the Secret keys. The Secret object name is the one that you see when you execute kubectl get secrets.

So, to create a simple pod that uses one of our previously created Secrets, we need a YAML file similar to this:

We can then use kubectl apply -f as usual to create a pod:

Now, let’s check if the Secret data was indeed included in the pod. For that, we can use kubectl exec to execute the env command in the pod, which will list all environment variables available in the pod:

And as you can see, we do indeed have our username and password variables with the values that we specified when creating the Secret. You can then use them in your application just like any other environment variables. Let’s try the second method and see how we can load the same Secret as a volume.

Secret as a Volume

This requires a few more lines to be added to our YAML file. For readability, I’ll create a different pod for that. We need to add two things there. First, a volume definition to the pod spec section and then volumeMounts to the container definition. In the volume definition, we need to define which Secret to load, and in volumeMounts definition, we define how it should be mounted to the pod. Let’s see an example:

Above, I specify that I want to mount a Kubernetes Secret object called mysecondsecret as a volume for my pod, and in the container definition, I specify that I want that volume to be mounted in the container at /opt/secret.

This method will create a file in the /opt/secret directory for each key from our Kubernetes Secret with its value as the content of the file. Let’s create that pod and validate if that’s what happens:

Voilà! That’s how you load Secrets as files to Kubernetes pods.

Kubernetes Secrets Management

We’ve covered quite a lot of information about using Kubernetes Secrets. You now know how to create them in different ways, what the different types of Secrets are, how to edit them, and how to use them in your pods. However, this is only half of the story. In the real world, no one wants to create Secrets manually. These days, everything is managed by CI/CD pipelines. But how would you create Secrets using CI/CD? You’d need to define your Secrets somewhere, somehow, so the CI/CD pipeline can read their definition and apply it to your cluster. And here’s the problem.

We’ve seen that the actual values of your Secrets, those that should be sensitive and not easily seen, are defined either in plain text or in very easy-to-decode base64. This means that creating Kubernetes Secret YAML Definition files and storing them anywhere is posing a risk that those files can be read by anyone who has access to wherever you’ve put them. Or, even worse, imagine that someone commits those files by mistake to a public code repository. If you pay attention and ensure that the files are properly protected, it could be somewhat secure. But there are better ways to do it. That’s where Kubernetes Secret Management solutions come into place. These are tools that try to solve that exact problem. If you want to explore them, take a look, for example, at ESO or Sealed Secrets.

Summary

We covered quite a bit today, but I hope you enjoyed the ride. Secrets are essential for any modern solution. But remember that we only showed you the basics of what they actually are and how to use them. Also, keep in mind that putting your password, tokens, or any other sensitive information into Kubernetes Secret objects doesn’t automatically make them “secret.” There’s more to consider in order to make them actually secure.

About StackState 

Designed to help engineers of all skill levels who build and support Kubernetes-based applications, StackState provides the most effective solution available for Kubernetes troubleshooting. Our unique approach to SaaS observability helps teams quickly detect and remediate issues, so they can ensure optimal system performance and reliability for their customers. With StackState’s comprehensive observability data, the most complete dependency map available, out-of-the-box applied knowledge and step-by-step troubleshooting guidance, any engineer can remediate issues accurately and with less toil.  

As a company dedicated to helping teams succeed with Kubernetes, we want to provide useful information in as many related areas as we can. We hope this tutorial proves helpful for your team. When it’s time to set your focus on Kubernetes troubleshooting, sign up for a free trial to see how simple it can be.