✦Crazy Wisdom

Crazy Wisdom

Episode #448: From Prompt Injection to Reverse Shells: Navigating AI's Dark Alleyways with Naman Mishra

Episode #448 · March 31, 2025 · 47 min

MP3 · Apple Podcasts · Spotify

About this episode

In this episode of Crazy Wisdom, I, Stewart Alsop, sit down with Naman Mishra, CTO of Repello AI, to unpack the real-world security risks behind deploying large language models. We talk about layered vulnerabilities—from the model, infrastructure, and application layers—to attack vectors like prompt injection, indirect prompt injection through agents, and even how a simple email summarizer could be exploited to trigger a reverse shell. Naman shares stories like the accidental leak of a Windows activation key via an LLM and explains why red teaming isn’t just a checkbox, but a continuous mindset. If you want to learn more about his work, check out Repello's website at repello.ai.
Check out this GPT we trained on the conversation!

Timestamps
00:00 - Stewart Alsop introduces Naman Mishra, CTO of Repel AI. They frame the episode around AI security, contrasting prompt injection risks with traditional cybersecurity in ML apps.
05:00 - Naman explains the layered security model: model, infrastructure, and application layers. He distinguishes safety (bias, hallucination) from security (unauthorized access, data leaks).
10:00 - Focus on the application layer, especially in finance, healthcare, and legal. Naman shares how ChatGPT leaked a Windows activation key and stresses data minimization and security-by-design.
15:00 - They discuss red teaming, how Repel AI simulates attacks, and Anthropic’s HackerOne challenge. Naman shares how adversarial testing strengthens LLM guardrails.
20:00 - Conversation shifts to AI agents and autonomy. Naman explains indirect prompt injection via email or calendar, leading to real exploits like reverse shells—all triggered by summarizing an email.
25:00 - Stewart compares the Internet to a castle without doors. Naman explains the cat-and-mouse game of security—attackers need one flaw; defenders must lock every door. LLM insecurity lowers the barrier for attackers.
30:00 - They explore input/output filtering, role-based access control, and clean fine-tuning. Naman admits most guardrails can be broken and only block low-hanging fruit.
35:00 - They cover denial-of-wallet attacks—LLMs exploited to run up massive token costs. Naman critiques DeepSeek’s weak alignment and state bias, noting training data risks.
40:00 - Naman breaks down India’s AI scene: Bangalore as a hub, US-India GTM, and the debate between sovereignty vs. pragmatism. He leans toward India building foundational models.
45:00 - Closing thoughts on India’s AI future. Naman mentions Sarvam AI, Krutrim, and Paris Chopra’s Loss Funk. He urges devs to red team before shipping—"close the doors before enemies walk in."

Key Insights
  1. AI security requires a layered approach. Naman emphasizes that GenAI applications have vulnerabilities across three primary layers: the model layer, infrastructure layer, and application layer. It's not enough to patch up just one—true security-by-design means thinking holistically about how these layers interact and where they can be exploited.
  2. Prompt injection is more dangerous than it sounds. Direct prompt injection is already risky, but indirect prompt injection—where an attacker hides malicious instructions in content that the model will process later, like an email or webpage—poses an even more insidious threat. Naman compares it to smuggling weapons past the castle gates by hiding them in the food.
  3. Red teaming should be continuous, not a one-off. One of the critical mistakes teams make is treating red teaming like a compliance checkbox. Naman argues that red teaming should be embedded into the development lifecycle, constantly testing edge cases and probing for failure modes, especially as models evolve or interact with new data sources.
  4. LLMs can unintentionally leak sensitive data. In one real-world case, a language model fine-tuned on internal documentation ended up leaking a Windows activation key when asked a completely unrelated question. This illustrates how even seemingly benign outputs can compromise system integrity when training data isn’t properly scoped or sanitized.
  5. Denial-of-wallet is an emerging threat vector. Unlike traditional denial-of-service attacks, LLMs are vulnerable to economic attacks where a bad actor can force the system to perform expensive computations, draining API credits or infrastructure budgets. This kind of vulnerability is particularly dangerous in scalable GenAI deployments with limited cost monitoring.
  6. Agents amplify security risks. While autonomous agents offer exciting capabilities, they also open the door to complex, compounded vulnerabilities. When agents start reading web content or calling tools on their own, indirect prompt injection can escalate into real-world consequences—like issuing financial transactions or triggering scripts—without human review.
  7. The Indian AI ecosystem needs to balance speed with sovereignty. Naman reflects on the Indian and global context, warning against simply importing models and infrastructure from abroad without understanding the security implications. There’s a need for sovereign control over critical layers of AI systems—not just for innovation’s sake, but for national resilience in an increasingly AI-mediated world.
Episode transcript
Stewart Alsop III00:00

Welcome to the Crazy Wisdom Podcast. This podcast is for you. If you have an insane drive to find the truth of things, it's not the good answers that we seek, but the good questions. I interview a range of different guests from many different fields, all with the intention to uncover the simple truths that are hidden in plain sight. Most people don't want to go there. I go there, my guests go there, and you benefit. Please let me know if you enjoy these episodes and as always, subscribe on itunes, Spotify, or wherever you listen to the podcasts.

Stewart Alsop III00:37

Welcome to the Crazy Wisdom Podcast. I've got Naman Mishra here and he is the CTO of Repel AI. Very interesting. It's so funny because I've just noticed that nobody talks about security. It's such an important thing. And I'm so glad that you're. You're here to. To kind of demystify what is going on with security. And welcome to the show.

Naman Mishra00:59

Thank you. Thank you, sir. Thank you so much for having me.

Stewart Alsop III01:02

So there's. It feels like there's two separate worlds of security within LLMs. There's one, the basic just the usage of LLMs, prompt injection, getting them to do things that they're not trained to do, and all these different things. So far, it doesn't seem like there's been a huge amount of. Maybe I just haven't heard of them, but people kind of getting screwed over by these types of things. Like, there's a lot in the beginning, but you don't hear that too much about prompt injection. I do know that Claude sort of did some things to help help their prompt injection. But then there's the other part of security, which seems like you guys might be doing with Repello, which is like, now you're building this machine learning app. There are all these sort of traditional cybersecurity things that have to do with computation that have been problems for. And it is also a mysterious world for me.

Your first full transcript is free. After that, an email opens every transcript in the index — a list of readers we can write to, not a guest book.

Subscribe