The Top 5 Secrets Management Mistakes and How to Avoid Them

Unknown publicly exposed secrets are almost inevitable for most orgs. Secrets could be anywhere — in your application servers, containerized apps, infrastructure as code (IaC), business-critical applications, and CI/CD toolchain. Name a place, and there’s probably a secret or two hiding there.

Managing the availability and integrity of secrets — passwords, API keys, tokens, etc. — is a responsibility with little room for failure. With credentials in hand, even the most novice bad actor can inflict serious damage. In a recent study, Unit 42 researchers periodically scanned and analyzed unsecured Kubernetes clusters on the internet. Among thousands of containers, images, and deployments, they found more than 80,000 exposed secrets.

Here are the top secrets management mistakes we see and how to avoid them. Continue reading to learn how to find secrets, where to securely store them, and more vital tips to secure your cloud environment.

1. Hard Coding Secrets

Developers are always looking for a way to save time. But one area where it can come back to bite you is hard coding or embedding credentials like SSH and API keys in code. Typically meant as a temporary fix, hard-coded secrets are especially risky as agile development expands to include third-party developers and code exchanges on cloud-based repos like GitHub.

How risky? According to OWASP, when hard-coded passwords are in use, it's almost certain that malicious users will gain access through the account in question.

Anyone with access to a project can view secrets that have been hard coded, making them not so secret. This can escalate rather quickly. Harvesting tools make it possible to grab access keys, escalate privileges and do all kinds of damage, such as moving into cloud databases with unrestricted access. From there, they can pick and choose from a buffet of IP, payment data, personally identifiable information (PII), and more.

If you do decide to use hard-coded secrets for local development, make sure you include the variable files in your .gitignore file, so they aren’t committed to your repository. For production, use a secrets store. You can find these in hardware, virtual appliance, software, or even SaaS versions. We’ll discuss the three most popular secret stores a little later.

2. Using One-Dimensional Scanning Tools

Everyone knows the “finding a needle in a haystack” metaphor. Secret scanning can feel a bit like this — trying to identify 15 real secrets out of 15,000 possible ones. Using the right techniques can help detect and pare down thousands of possible secrets to a precise, manageable group. Yet, many tools disparately scan using only one of the three techniques listed below, missing many secrets or adding a bunch of unwanted noise from false positives.

Regular Expressions

Secrets using regular expressions are those that scanners most commonly identify. These are specific patterns related to one type of credential — think AWS access keys, Slack tokens, and so on. But regular expressions are often too broad or inaccurately defined to show up correctly in a scan. For example, what you actually have in code might not match the denoted regular expression for a Twilio API key.

Keywords

Secrets using keywords include phrases that have a high likelihood of being a secret, such as password, pwd, api*_key. These are relatively easy to catch, but can still ring up false positives or duplicates.

High Entropy Patterns

Secrets using high entropy patterns have longer strings of highly random character combinations. You may think to lean heavily on higher entropy scores, but not all secrets are made of highly random data or strong passwords. Because user-generated passwords are usually fairly simple, they are more likely to fail a scan based on entropy alone.

How can you increase true positives and reduce false positives? With many proprietary and open source scanning tools to choose from, the most effective employ scanning for a combination of high entropy patterns and keywords. Combining detectors — in other words, searching for keywords or regular expressions with high entropy patterns — can help lower the ratio.

3. Scanning for Secrets Too Late in the Development Lifecycle

Traditionally, cloud security and compliance checks happen in runtime. But with new IaC security tooling, it’s becoming more common to scan for issues before runtime via continuous integration (CI) pipelines. But a more proactive approach — checking code for security issues before it even makes its way to a version control system (VCS) — can help remediate issues before they turn into headaches for all teams involved.

This goes for hard-coded secrets, as well.

Although secrets scanning can be done in runtime, you can also take it one step further by scanning in code configuration even before code gets committed to shared repositories. A tool like Checkov scans IaC templates for strings that follow patterns of a variety of secrets and flags them for review. It can track down identified secrets using the three methods mentioned above: regular expressions, keyword matching, and entropy-based scanning.

Encouraging your individual developers to integrate secrets scanning into their quality assurance workflows prior to integrating code into shared repositories is the way to go. Then, on top of that, be sure to add guardrails at each step of the way — in your VCS, CI pipeline, and runtime environment as well.

4. Storing Secrets in a Public Place

If you do a quick Google search, you’ll find a bunch of comparisons between Hashicorp Vault, AWS Parameter Store, and AWS Secrets Manager. While Secrets Manager was designed to encrypt confidential information and offers a built-in password generator through the use of AWS CLI, Parameter Store expands the use case to application configuration variables like URLs, DB hostnames, custom settings, product keys, and more. Vault provides its secrets storage service via a user interface, CLI or HTTP API.

Which one you choose is up to you, but there is one place you do not want your secrets hanging out, and that is anywhere that is open to the public, such as your VCS or in a registry. And especially not in your CI/CD configuration files, like a GitHub Action YAML or CodeBuild buildspec.yml file.

Remember, any secret added directly into one of these file types is visible in the VCS or will be revealed in any resulting logs from the build. Be sure to use environment variables or secrets managers, and inject and obfuscate the secrets only when used.

5. No Process for Key Rotation and Revocation

Secrets management tools like those mentioned above are useful in that they can automate the creation and storage of secrets. But another important aspect they cover is revocation and rotation, especially important in the case of a leak or compromise.

It’s a best practice to rotate your keys periodically. Most organizations rotate on a monthly or quarterly time frame, but more specifically, the National Institute of Standards and Technology (NIST) recommends rotating keys before approximately 232 encryptions have been performed. To estimate the number of encryptions, take the sum of the following metrics:

• The vault.barrier.put telemetry metric.
• The vault.token.creation metric where the token_type label is batch.
• The merkle.flushDirty.num_pages metric.
• The write-ahead logging (WAL) index.

Secrets Management Tips

By avoiding all of these mistakes, you’re on the right path to keeping your secrets safe. But, if you do find your secrets exposed anywhere online, quickly follow these steps to remediate:

1. First, take a deep breath.
2. Disable and revoke the key ASAP.
3. Ascertain which services have been accessed by the keys and start relevant infosec/data leakage processes.
4. Review logs for nefarious activities, such as VMs you didn’t spin up.
5. Clean your git history or log history, where possible.
6. Create a new key and distribute it to the various services that need it.
7. Monitor your services for any that break because you haven’t updated the key for that service.

Follow these secrets management best practices. It’s a relatively easy way to keep your credentials airtight, maintain trust within your teams, and improve your security culture.

Interested in learning more about secrets scanning? Discover the 6 key features of any complete secrets scanning solution with the Secrets Security Checklist.

This post was originally published on The New Stack.