By Eduard Schwarzkopf in AWS — Aug 7, 2023

I Exposed AWS Access Keys, On Purpose: Here's What I Learned and How I Boosted Incident Response

Introduction

Who doesn't know that, you just write a quick fix for your code? After a few hours, the function is finally finished.

Proud to have finally done it. Your Brain, toast. Your commit, fast. Pushing your credentials without realising, even faster! Let's find out what happens next.

It happens like that every day. However, I wanted to experience this myself. What is happening on the AWS side, and what steps can I add to increase our security posture here. So join me on my journey of making the Access Keys public so we can both learn from it.

Setup

The setup for this experiment is simple:

A public repository on GitHub
A clumsy user (me)
Access key from that user
SNS
SES
Step Functions
EventBridge

So the first thing I did was to create a repository on GitHub. Nothing special. It contains a README.md and .env file. That should be enough. Usually, the .env file shouldn't be committed in any repository, but clumsy me wanted to finally finish the work. So I forgot to add this to the .gitignore. Whoopsie!

Luckily, for me, I'm using a specific user for this case alone. Since I knew beforehand that I will expose those keys "by accident", I was prepared. After all, exposing any keys is always dangerous. Don't try this at home!

The dummy user is not allowed to do anything, nada, niente. Not only implicitly, but explicitly! AWS has already created a role for this purpose AWSDenyAll. So I'm using this policy for that clumsy user.

Now that that's done, quickly created a SNS Topic and put my email in it as a subscriber.

Now I have to get the corresponding event. The magic word is, therefore: EventBridge Rule. For this, I followed this tutorial. After I set up everything, I did the first test. A little Test and the result: One support ticket, one email from AWS, and no SNS notification. Interesting. Like the good first-class programmer I am, let's not change anything and run it again. Still no change. Very interesting.

A small adventure

A few exposed keys and just as many support tickets later, I figured it out. Let me explain it to you. You see, the event is triggered by the Support Center. More precisely, as soon as an issue is created. See the following screenshot:

As you can see the event is called AWS_RISK_IAM_QUARANTINE and NOT AWS_RISK_CREDENTIALS_EXPOSED. Glad nobody told me this before! But that is not all. Under the EventBridge rule AWS_RISK_CREDENTIALS_EXPOSED is not available in the dropdown either! Isn't that great?

Fortunately, the solution here is relatively simple, do it yourself:

Here is the JSON

{
  "source": ["aws.health"],
  "detail-type": ["AWS Health Event"],
  "detail": {
    "service": ["RISK"],
    "eventTypeCategory": ["issue"],
    "eventTypeCode": ["AWS_ACCESS_KEY_EXPOSED", "AWS_RISK_IAM_QUARANTINE"]
  }
}

Yep, there is still more to do. The region plays a decisive role here as well because the Support Center is a global service, but it directly means that the events arrive in us-east-1. Global usually means us-east-1. Remember that! So the EventBridge Rule as well as the SNS Topic and whatever you need to automate in this case must be created in us-east-1!

Some AWS Health events are not Region-specific. Events that aren't specific to a Region are called global events. These include events sent for AWS Identity and Access Management (IAM). To receive global events, you must create a rule for the US East (N. Virginia) Region.

Source

So again short and sweet:

the EventBridge Rule to receive a global event must be in us-east-1.
the SNS Topic must be in us-east-1.
the event pattern must be as described above

Now this is all setup, time to make a whoopsie-daisy and expose our access key.

The Exposure

As described before, the key is made public in the .env file "by accident". To do this, I simply enter the Access Key and Secret Access Key and use the magic words git commit -m "exposium!" && git push origin main.

After the push, some processes are initiated at GitHub and AWS. This happens on every commit and this allows AWS to react in time if they find access keys in a repository. You can read here about GitHubs secret scanner. In the default case, 3 things happen:

a support ticket is opened on AWS Support
you get an email with the info that access keys have been exposed
the user gets the policy AWSCompromisedKeyQuarantineV2 attached.

The email describes some steps you can do in this case. Here is an excerpt:

And here is the event that gets created:

{
   "version":"0",
   "id":"<id>",
   "detail-type":"AWS Health Event",
   "source":"aws.health",
   "account":"<account_id>",
   "time":"2023-07-10T09:30:04Z",
   "region":"us-east-1",
   "resources":[
      "AKIARCDX5PTSMNPTIQOR"
   ],
   "detail":{
      "eventTypeCode":"AWS_RISK_IAM_QUARANTINE",
      "communicationId":"<communicationId>",
      "eventScopeCode":"ACCOUNT_SPECIFIC",
      "eventTypeCategory":"issue",
      "affectedEntities":[
         {
            "entityValue":"AKIARCDX5PTSMNPTIQOR"
         }
      ],
      "eventMetadata":{
         "accountId":"<account_id>",
         "publicKey":"AKIARCDX5PTSMNPTIQOR",
         "userName":"ExposedKeysUser",
         "exposedUrl":"https://github.com/<github_user>/expose-aws-keys-experiment/blob/<commit>/.env"
      },
      "eventArn":"arn:aws:health:us-east-1::event/RISK/AWS_RISK_IAM_QUARANTINE/AWS_RISK_IAM_QUARANTINE-oBGgvLlBIX",
      "service":"RISK",
      "eventDescription":[
         {
            "latestDescription":"Your AWS Account may be compromised! We have opened a Support Case with more details. Please visit the AWS Support Center https://aws.amazon.com/support to review the case we've opened for you and take action immediately.",
            "language":"en_US"
         }
      ],
      "lastUpdatedTime":"Tue, 10 Jul 2023 09:30:04 GMT",
      "startTime":"Tue, 10 Jul 2023 09:30:04 GMT",
      "eventRegion":"us-east-1",
      "endTime":"Tue, 25 Jul 2023 09:30:04 GMT",
      "statusCode":"open"
   }
}

The email from AWS describes what is best to do, but those are manual steps. Can you imagine? Manual steps...

No, we are what we are, because we automate all the things!

Now let's take a look at what we can derive from this.

From Experiment to Improvement

Now is the question what to do with all the information? AWS is providing great steps in the e-mail/issue that you can do in the case of compromised access keys, but as I've said before, I want this to be automated. So I did a little bit of thinking and came up with the following steps I want to include:

Deactivate the access key
Send useful notifications to the internal security team

Here is the diagram of what should happen:

This is a minimal approach and more like a proof of concept, rather than a mature action plan. Anyway, I've created a simple step function that uses the previous notification and consists of the following steps:

{
  "Comment": "A step function to send an email when an AWS access key is exposed.",
  "StartAt": "DeactivateAccessKey",
  "States": {
    "DeactivateAccessKey": {
      "Type": "Task",
      "Parameters": {
        "UserName.$": "$.detail.eventMetadata.userName",
        "AccessKeyId.$": "$.detail.eventMetadata.publicKey",
        "Status": "Inactive"
      },
      "Resource": "arn:aws:states:::aws-sdk:iam:updateAccessKey",
      "Next": "SendEmail",
      "ResultPath": "$.AccessKeyStep.status"
    },
    "SendEmail": {
      "Type": "Task",
      "Parameters": {
        "Content": {
          "Simple": {
            "Body": {
              "Text": {
                "Data.$": "States.Format('Dear Security Team,\r\n\r\nWe wish to bring to your immediate attention an incident of AWS access key exposure detected by automated \"GitHub secret scan\". Here is the incident details:\r\n\r\nEvent ID: {}\r\nIAM User: {}\r\nAccess Key ID: {}\r\nExposed Key Status: [Status will be filled here]\r\nEvent Timestamp: {}\r\nInitial Detection Source: AWS Health Dashboard\r\nAutomated Action Taken: [Automated Action will be filled here]\r\n\r\nPlease acknowledge receipt of this notification and keep us updated on your progress with the investigation and remediation. Remember, security is our collective responsibility.', $.detail.eventArn, $.detail.eventMetadata.userName, $.resources[0], $.time)"
              }
            },
            "Subject": {
              "Data": "Security Alert: AWS Access Key Exposure"
            }
          }
        },
        "Destination": {
          "ToAddresses": [
            "security@example.com"
          ]
        },
        "FromEmailAddress": "alert@example.com"
      },
      "Resource": "arn:aws:states:::aws-sdk:sesv2:sendEmail",
      "ResultPath": null,
      "End": true
    }
  }
}

This can be easily extended with more steps, checks, and whatnot. For example, you could also add them GetAccessKeyLastUsed to have an idea of where to look, when an incident occurs. Be careful! This data might not be the most recent one. Make sure to check CloudTrail for other activities. Use the filter to search for your exposed access key: https://<region>.console.aws.amazon.com/cloudtrail/home?region=<region>#/events?AccessKeyId=<access_key_id>.

If you feel fancy, you can add this link to your notification. Just to give you another example of what you can do.

Lessons Learned and Best Practices

I've learned a lot from this little experiment. Especially how to avoid exposing my keys over and over again thanks to EventBridge replay functionality. Please be smarter than me and use this instead. (AWS Support, I'm sorry)

Regarding access keys, I've learned the following:

Don't use them
Avoid them
That's it

Simple right, but you are probably now like:

"But I need access keys because my resources are outside of AWS". In that case, just use AWS Identity and Access Management Roles Anywhere. Problem solved.

If you still insist on using access keys, make sure to do at least the following:

Create a dedicated user for each application
Give that user the least privileges
Don't hardcode your access keys into your project
rotate them
remove them

By removing them, I mean especially those that are no longer used!

Consider moving your application where your access keys are. Yes, into the cloud! A far better solution to consider is to use roles instead of access keys. Here is an excellent article No more AWS keys for you from Nicole Yip.

Still, this doesn't answer the question of what do to in the case of exposed keys in detail. After some digging around, I found the response steps from NIST Special Publication 800-61r2 Computer Security Incident Handling Guide (what a long name):

Preparation
Detection
Containment
Eradication
Recovery
Lessons learned

What does this mean in detail? AWS kindly provides an example Incident Response Playbook what those steps mean in detail, which I proudly stole from them:

[PREPARATION] Perform an Asset Inventory

[PREPARATION] Implement a training plan to identify and respond to exposed IAM credentials

[PREPARATION] Implement a communication strategy for incident response

[DETECTION] Identify Root Account Access (authorized and not)

[DETECTION] Identify New or unrecognized IAM users

[DETECTION] Identify Unrecognized or unauthorized resources (e.g., EC2, Lambda)

[DETECTION] Identify and find exposed secrets

[DETECTION] Identify Unusual billing increases

[DETECTION] Respond to notification from AWS or a third party that my AWS resources or account might be compromised

[DETECTION] Identify any potentially unauthorized IAM user credentials

[PREPARATION] Identify Escalation Procedures

[DETECTION AND ANALYSIS] Review CloudTrail Logs

[DETECTION AND ANALYSIS] Review VPC Flow Logs

[DETECTION AND ANALYSIS] Review Endpoint / Host Based Logs

[CONTAINMENT] Perform appropriate containment actions

[ERADICATION] Review the findings from Review CloudTrail event history for activity by the compromised access key

[ERADICATION] Review the Avoiding unexpected charges

[RECOVERY] Perform appropriate recovery actions

[PREPARATION] Perform a Prowler IAM Scan

[PREPARATION] Enable MFA

[PREPARATION] Verify your account information

[PREPARATION] Use AWS Git projects to scan for evidence of unauthorized use

[PREPARATION] Avoid using the root user for day-to-day operations

[PREPARATION] Evaluate your Overall Security Posture

From this guideline, you can create a detailed playbook which fits your own needs. Make sure to have it ready when you need it and give the whole guideline a read!

Sidenote: For all my fellow germans, there is also a guideline provided by the BSI (Thanks to Manuel for the info!). This might be more useful, because of compliance reasons.

Conclusion

That wraps it up. Nothing beats hands-on experience. I've learned a lot by just exposing some access keys over and over again. Don't worry, I've also learned to use EventBridge the right way, so I won't end up with 15.000 support tickets, so should you!

My takeaways from this are as follows:

Avoid using access keys where you can. Use roles instead!
Prepare for access key exposure!
Make sure to lay out your action plan
Test your plan yourself before somebody forces you to do it

While I've still got some research to do, and more measures to implement, I hope this exploration of the world of public access key exposure has been as enlightening for you as it was for me. If you are as prone to face-palm moments as I am, I'd like to think that you'll sleep a bit easier tonight.

My next steps are now to bring my new knowledge into a playbook that follows the best practices from AWS and NIST.

Stay tuned and good night.

This article was originally written for evoila