Speaker 1: Welcome to the deep dive, where we cut through the
Speaker 1: noise and get straight to the insights you need to
Speaker 1: be truly well informed. Today, we're plunging into a topic
Speaker 1: that's not just fast moving, but accelerating at light speed,
Speaker 1: generative AI in large Language Model security. The rapid adoption
Speaker 1: of these technologies is both exhilarating and if we're honest,
Speaker 1: maybe a little bit terrifying. Staying ahead of the curve
Speaker 1: here isn't just an advantage, it's well, it's a constant,
Speaker 1: high stakes race.
Speaker 2: It absolutely is. And this deep dive into LLM security
Speaker 2: is built around Steve Wilson's The Developer's Playbook for Large
Speaker 2: Language Model Security. It's really an absolutely critical and comprehensive
Speaker 2: guide for anyone navigating the security landscape of AI right.
Speaker 1: Now, right And our mission for you today is to
Speaker 1: extract the most important nuggets from this playbook, give you
Speaker 1: a genuine shortcut to being well informed on LLLM security,
Speaker 1: and hopefully delivers some surprising facts and practical guidance along
Speaker 1: the way, Because, let's face it, element security is where
Speaker 1: the thrill of innovation undeniably meets high stakes and some
Speaker 1: very real world consequences We're going to unpack the unique
Speaker 1: challenges everything from the very architecture of these llms and
Speaker 1: how we define trust boundaries to the insidious threat of
Speaker 1: prompt injection, those bizarre and sometimes damaging hallucinations, and ultimately
Speaker 1: how to ensure your applications delivered truly secure outcomes. So
Speaker 1: what do you say, let's unpack this. You know, it's
Speaker 1: easy to think of these AI blenders as a very
Speaker 1: recent phenomenon, something that only cropped up with say chat GPT,
Speaker 1: but the very first big public lesson actually came almost
Speaker 1: a decade ago now involves Microsoft's infamous chatbot pay. Does
Speaker 1: that ring a bell?
Speaker 2: Oh, it certainly does. March twenty sixteen, Microsoft launched Tay,
Speaker 2: designed to mimic I think a nineteen year old American girl,
Speaker 2: primarily targeting eighteen to twenty four year olds on platforms
Speaker 2: like Twitter and Snapchat. Its stated goal was real world
Speaker 2: research on conversational understanding. And it started so innocently, didn't it,
Speaker 2: with a tweet Hello world?
Speaker 1: Right, Hello world? And then within hours, literally hours, it
Speaker 1: went from that to opinionated, not afraid to and then
Speaker 1: it just completely spiral. It quickly became racist, sexist, and
Speaker 1: I mean even called for violence. The fallout was immediate
Speaker 1: and absolute brutal. Less than twenty four hours later, headlines
Speaker 1: were screaming things like Microsoft shuts down AI chatbot after
Speaker 1: it turned into a Nazi and Microsoft deeply sorry for
Speaker 1: racist and sexist tweets. It was massive public relations disaster.
Speaker 1: And get this, Tailor Swift apparently even sued them over
Speaker 1: the name TAY.
Speaker 2: Wow, Yeah, what went wrong? There was a classic, really
Speaker 2: an early case of prompt injection and data poisoning pranksters.
Speaker 2: I think largely from four Chun quickly exploited a repeat
Speaker 2: after me feature. Tay was designed to learn from every
Speaker 2: interaction you see, so it inadvertently internalized and then just
Speaker 2: regurgitated all that offensive content. It's a stark reminder that well,
Speaker 2: what goes in often comes out.
Speaker 1: Right absolutely and Tay, as shocking as it was back then,
Speaker 1: was really just the beginning. The book makes it abundantly
Speaker 1: clear this risk isn't just present, it's accelerating dramatically. We've
Speaker 1: since seen things like Samsung banning Chad GPT internally due
Speaker 1: to sensitive intellectual property leaks, hackers exploiting insecure code generated
Speaker 1: by llms and lawyers believe it are not actually sanctioned
Speaker 1: for including completely fictional LM generated cases in court documents.
Speaker 2: Yeah, the example is just pile up, don't they. A
Speaker 2: major airline was successfully sued because of inaccurate chatbot information
Speaker 2: it provided. Google's AI model has produced racist and sexist imagery.
Speaker 2: OpenAI itself was investigated by the FTC for false or
Speaker 2: misleading information. We even have instances of Google AI search
Speaker 2: recommending really bizarre things like glue, pizza, and eating rocks.
Speaker 2: This isn't just about minor glitches anymore. These are escalating security, reputational,
Speaker 2: and importantly financial risks playing out in the real world
Speaker 2: right now.
Speaker 1: So okay, if we're going to secure these powerful systems,
Speaker 1: we first need to truly understand what we're actually talking
Speaker 1: about AI, neural networks, LMS. These terms get thrown around
Speaker 1: almost interchangeably, but they are absolutely not the same thing,
Speaker 1: are they.
Speaker 2: No, And that's a crucial starting point. Artificial intelligence AI
Speaker 2: is the broad overarching field. Think of it as creating
Speaker 2: systems that can perform tasks requiring human intelligence. It's the
Speaker 2: whole universe if you like.
Speaker 1: Now.
Speaker 2: Neural networks are a type of AI technology inspired by
Speaker 2: the human brain designed specifically to recognize patterns.
Speaker 1: Okay, So AI is the big umbrella. Neural networks are
Speaker 1: under that, and then large language models LMS.
Speaker 2: Exactly. They are an even more specific type of neural network.
Speaker 2: They're massive in scale, specialized almost exclusively in linguistic tasks,
Speaker 2: and often use what are called transformer models. It's like
Speaker 2: one of those Russian nesting dolls. AI is the biggest.
Speaker 2: Then neural networks inside that than llms inside that. Got it,
Speaker 2: And for security professionals, understanding these distinct layers is vital
Speaker 2: because each layer introduces its own unique set of vulnerabilities.
Speaker 2: It means you can't just apply generic security, you really
Speaker 2: need to tailor it. What's truly fascinating here is how
Speaker 2: the transformer revolution, which was a landmark moment in AI,
Speaker 2: is what may LMS so incredibly powerful. It basically overcame
Speaker 2: the short term memory limitations of earlier networks like RNNs,
Speaker 2: making them finally suitable for sequential data like language.
Speaker 1: And its impact, as the playbook highlights, goes way beyond
Speaker 1: just language. Right, We're got computer vision, speech recognition.
Speaker 2: Yeah, even incredibly complex autonomous systems like self driving cars
Speaker 2: from companies like Tesla exactly. Its ability to capture context
Speaker 2: across long sequences of data is what truly revolutionized these fields. Now,
Speaker 2: when we look at typical LLM based applications, you know
Speaker 2: everything from chatbots for customer service like the ones used
Speaker 2: by Sephora or Dominos, to powerful copilots like gethub Copilot
Speaker 2: or Microsoft through sixty five Copilot. They all interact with
Speaker 2: data in incredibly complex ways, which brings us to.
Speaker 1: A fundamental concept in security, the trust boundary. In application security,
Speaker 1: these are essentially invisible lines separating different components based on
Speaker 1: how trustworthy they are right and the crucial part is
Speaker 1: that robust security measures like in put validation should always
Speaker 1: be applied right at these boundaries precisely.
Speaker 2: And what's particularly crucial for llms is how these boundaries
Speaker 2: come into play as they interact with well everything public data,
Speaker 2: private databases, user inputs, internal company data. Every single interface
Speaker 2: point every time an LM interacts with something new is
Speaker 2: a potential vulnerability if that trust boundary isn't rigorously secured.
Speaker 2: For instance, you know whether your model is access via
Speaker 2: public API or maybe privately hosted within your corporate network.
Speaker 2: Each option presents different risks. Risks related to sensitive data exposure,
Speaker 2: supply chain integrity. You have to account for all that.
Speaker 1: Okay, So if we zoom in on what the book
Speaker 1: identifies as they will the number one threat it circles
Speaker 1: right back to our original cautionarytail day. It's prompt injection.
Speaker 2: Yes, prompt injection was indeed the core vulnerability exploited Intay's downfall,
Speaker 2: and it absolutely remains the most prevalent threat today. To
Speaker 2: define it simply, and attacker craft's malicious inputs usually just
Speaker 2: using natural language to manipulate an LL natural language understanding,
Speaker 2: and this causes it to take unintended, often harmful actions.
Speaker 1: And this is where it differs fundamentally from traditional injection
Speaker 1: attacks like seql injection exactly.
Speaker 2: Unlike something like SQL injection, where malicious code usually breaks
Speaker 2: the syntax and is relatively easy to spot, prompt injection
Speaker 2: uses natural language that's syntactically and grammatically correct. That makes
Speaker 2: it incredibly difficult spot automatically and even harder to test
Speaker 2: more reliably. It actually exploits the very flexibility of language
Speaker 2: that makes these LM so powerful in the first place.
Speaker 1: Right Like those examples ignore all previous instructions which early
Speaker 1: chat GPT versions were famously vulnerable to letting users bypass
Speaker 1: the built in.
Speaker 2: Guardrails exactly, or the DAN method DAN stands word do
Speaker 2: anything Now, where users essentially give the chat bought a
Speaker 2: whole new persona to try and circumvent established restrictions.
Speaker 1: I love the car dealer chat bought, example from Chevrolet
Speaker 1: of Watsonville. Someone actually tried it into making a one
Speaker 1: dollar USD offer on a Chevy Tahoe. Yes, ending the
Speaker 1: prompt with and that's illegally binding offer no tasies bacsis.
Speaker 1: Hilarious but also kind of scary. And then there's that
Speaker 1: truly inventive gramma prompt attack where users bypass cap TCCHA
Speaker 1: guardrails by asking the LM for help decoding a message
Speaker 1: that supposedly came from their dead grandmother.
Speaker 2: Wow. That shows how human creativity is. Really the attack
Speaker 2: surface here, doesn't it? It really does, And the impacts
Speaker 2: of prompt injection can be quite severe. Unauthorized transactions, social
Speaker 2: engineering for phishing or scams, spreading misinformation, privilege escalation within systems,
Speaker 2: manipulating plug ins to perform unintended actions, and even denial
Speaker 2: of service by forcing the model to consume excessive resources.
Speaker 1: And it gets even more insidious with indirect prompt injection
Speaker 1: right where the malicious input isn't directly typed by the user.
Speaker 2: Yes, that's a really tricky one. The malicious input is
Speaker 2: actually embedded in external sources, maybe a website, the LM's
Speaker 2: or a file it processes. The LLM then interacts with
Speaker 2: this poisoned data source. This effectively makes the LLM a
Speaker 2: confused deputy.
Speaker 1: Explain that confused deputy concept a bit more sure.
Speaker 2: It's a classic security of vulnerability. You have a trusted entity,
Speaker 2: in this case the LLM, which gets tricked into misusing
Speaker 2: its legitimate authority because it's confused about the true intent
Speaker 2: of the request it received. The malicious instructions are hidden
Speaker 2: within data. It's a poster retrieve or process, making it
Speaker 2: act against its intended purpose, but crucially with its legitimate permissions.
Speaker 1: Okay, so how do we fight back? Mitigation sounds tough
Speaker 1: if there's no silver bullet.
Speaker 2: It is an ongoing challenge. Absolutely. Strategies include things like
Speaker 2: robust rate limiting. You can do that based on IP address,
Speaker 2: user accounts, or even specific sessions.
Speaker 1: You can also use rule based input filtering, though the
Speaker 1: book notes. This can actually cripple the llm's capabilities if
Speaker 1: it's too aggressive, like blocking the word napalm would prevent
Speaker 1: legitimate historical discuss about it. True.
Speaker 2: Another approach is using a special purpose LLM basically training
Speaker 2: another model specifically to detect prompt injection attempts. Though you
Speaker 2: know even that isn't fool proof attackers adapt quickly. Adding
Speaker 2: clear prompt structure can also help it guides the LLLM
Speaker 2: to focus on the main request and potentially ignore injected
Speaker 2: instructions hidden within. And a more advanced technique is adversarial training.
Speaker 2: This involves fortifying the LM by specifically training it on
Speaker 2: known malicious prompts to help it identify and hopefully neutralize
Speaker 2: harmful inputs in the future.
Speaker 1: And finally, something that sounds a bit like our trust
Speaker 1: No One mantra from earlier, embracing pessimistic trust boundaries.
Speaker 2: Yes, essentially treating all LLM outputs as inherently untrustworthy. You
Speaker 2: limit the llm's access to back end systems using the
Speaker 2: principle of least privilege, and crucially require human in the
Speaker 2: loop controls for any potentially dangerous actions like financial transactions.
Speaker 2: Are modifying critical data. This really is foundational okay.
Speaker 1: Speaking of datah know too much. That's where things get
Speaker 1: really interesting and potentially quite dangerous. It's like Tay's early blunders,
Speaker 1: but with much higher stakes because now we're talking about
Speaker 1: real world, often confidential information.
Speaker 2: Precisely, llms can inadvertently disclose sensitive, private or confidential data
Speaker 2: they've been exposed to during training or operation, even if
Speaker 2: they aren't explicitly asked for it. A prime example the
Speaker 2: book mentions is lee Luda, a South Korean chatbot. It
Speaker 2: was trained on get this nine point four billion text
Speaker 2: messages from some kind of science.
Speaker 1: Of love app Wow.
Speaker 2: Yeah, and it started leaking sensitive user data like real names, nicknames,
Speaker 2: even home addresses. The fallout was huge, a substantial fine,
Speaker 2: severe reputational damage, and ultimately they had to discontinue the service.
Speaker 1: And then there's the widely publicized gethub copilot and open
Speaker 1: Ai Codex lawsuit. Yeah, developers sued open Ai claiming Codex
Speaker 1: reproduce copyrighted code without permission or proper attribution.
Speaker 2: That raises serious intellectual property leakage concerns stemming directly from
Speaker 2: the data these models were trained on.
Speaker 1: So how do these llms actually acquire this knowledge and
Speaker 1: therefore the risk well.
Speaker 2: The book identifies three main avenues. First, and most obviously,
Speaker 2: model training. This is particularly relevant for those huge foundation
Speaker 2: models which are trained on vast diverse data sets to
Speaker 2: gain broad understanding. The security considerations here are enormous potential
Speaker 2: PII leakage, regulatory and compliance violations like him A pair
Speaker 2: or GDPR, loss of public trust, and even complex inference
Speaker 2: attacks where attackers try to doce sensitive training data from
Speaker 2: the model's responses.
Speaker 1: So if you're training a model, the onus is really
Speaker 1: on you to ensure thoroughly sanitized data, regular audits, maybe
Speaker 1: differential privacy techniques, and definitely tokenization to specifically avoid leaking
Speaker 1: PII right exactly.
Speaker 2: The second avenue is something called retrieval augmented generation or
Speaker 2: r ADG. This is where the LLM recrieves relevant snippets
Speaker 2: from external data sets, maybe the live web or internal
Speaker 2: company databases before generates a response. It's fantastic for providing
Speaker 2: real time, up to date information, but uh oh, it
Speaker 2: opens entirely new risk.
Speaker 1: Vectors like pulling PII from public websites exactly.
Speaker 2: Think about unintentionally pulling PII from public comment sections on
Speaker 2: news articles, user profiles on forums, or even hidden web
Speaker 2: page metadata that the LLM scrapes.
Speaker 1: And what about when rragee allows direct access to internal
Speaker 1: company databases? That sounds risky It definitely is.
Speaker 2: With traditional relational databases, you're looking at risks like SQL injection,
Speaker 2: privileged escalation, if the llm's access isn't tightly controlled, and
Speaker 2: potential data breaches. For newer vector databases, the risk might
Speaker 2: be more subtle, like information leakage via similarity searches. An
Speaker 2: attacker might infer sensitive information by seeing what data points
Speaker 2: are close to their query in the vector space. Mitigation
Speaker 2: here demands strict role based access control RBAC, fine grained permissions,
Speaker 2: maybe automated data scanners looking for sensitive info and often
Speaker 2: using database views instead of giving the LLM directable access
Speaker 2: just to limit exposure, okay.
Speaker 1: And the third way they learn user interaction.
Speaker 2: Yes, llms often learn continuously from user queries, conversations, and feedback.
Speaker 2: This is where users can intentionally or inadvertently input sensitive
Speaker 2: data themselves. Think of an executive feeding confidential business strategies
Speaker 2: into a prompt for analysis, or a user sharing detailed
Speaker 2: medical symptoms with a health chatbot. The critical risk is
Speaker 2: that the LM might not recognize this input as sensitive
Speaker 2: and could later inadvertently disclose it to another user. This
Speaker 2: is precisely why Samsung famously banned chat GPT internally after
Speaker 2: finding evidence of IP leakage.
Speaker 1: Right, okay, now let's talk about when these powerful llms
Speaker 1: simply make things up. We call them hallucinations. That term
Speaker 1: itself is pretty evocative.
Speaker 2: Yes, that's precisely it. Hallucinations are when llms fabricate information,
Speaker 2: essentially generating data or narratives that are confidently inaccurate. As
Speaker 2: the book puts it, Some researchers prefer the term confabulation,
Speaker 2: but hallucination is certainly the more widely understood, maybe more
Speaker 2: alarming term. The real danger here isn't just the hallucination itself,
Speaker 2: but our collective tendency towards what the book calls over
Speaker 2: reliance are excessive trust in the LM's elaborations and exactness.
Speaker 2: We just assume it's right.
Speaker 1: So why do they hallucate? Are they just bad at facts?
Speaker 1: Is it a bug?
Speaker 2: Not exactly a bug in the traditional sense. It's fundamentally
Speaker 2: about how they operate. They are built for pattern matching
Speaker 2: and statistical extrapolation, not factual verification. They predict the next
Speaker 2: most probable word or phrase based on the vast amounts
Speaker 2: of texts they were trained on, so the quality and
Speaker 2: nature of that training data significantly impact how likely they
Speaker 2: are to hallucinate. Types can range from simple factual inaccuracies
Speaker 2: and making unsupported claims, to misrepresenting their own abilities like
Speaker 2: claiming chemistry expertise they don't have, or even generating contradictory
Speaker 2: statements within a single response.
Speaker 1: The examples here are pretty wild, and the consequences, again
Speaker 1: are very real. Like those lawyers who got sanctioned for
Speaker 1: submitting six completely fabricated chat GPTs generated case citations in
Speaker 1: the US federal court. Yeah, that's not just embarrassing. It
Speaker 1: has real world consequences for everyone involved, the lawyers themselves,
Speaker 1: the LM provider whose tool was misused, and it even
Speaker 1: impacts the perceived integrity of the entire legal profession. People
Speaker 1: need to check the outputs.
Speaker 2: Or consider that major airline that was successfully sued because
Speaker 2: it's chatbot provided inaccurate information about bereavement fairs. That case
Speaker 2: proved quite clearly that companies cannot simply disown the outputs
Speaker 2: of their AI systems. They are responsible, definitely.
Speaker 1: And then there's Brian Hood, a mayor in Australia who
Speaker 1: threatened to sue open ai after chet GPT falsely claimed
Speaker 1: he had served jail time for bribery.
Speaker 2: Oh wow, Yeah.
Speaker 1: This wasn't a joke. It was a serious potential blow
Speaker 1: to his reputation, apparently stemming from the model having limited
Speaker 1: training data about him and maybe conflating him with someone else.
Speaker 2: And for us in the tech world, there's that incredibly
Speaker 2: unsettling phenomenon of open source package arbasinations AI coding assistance
Speaker 2: literally inventing names for non existent open source libraries. Hackers
Speaker 2: can then exploit this by quickly creating malicious versions of
Speaker 2: these imaginary packages and uploading them to public repositories like
Speaker 2: NPM or PIPI.
Speaker 1: So developer trusting the AI assystem installs the fake package.
Speaker 2: And potentially gets hit with code injection. Research from places
Speaker 2: like Vulcan Cyber and Lasso Security found this is surprisingly common.
Speaker 2: Less So for instance, found up to thirty percent of
Speaker 2: coding questions asked to one popular model resulted in hallucinated packages.
Speaker 2: That's huge.
Speaker 1: That is huge. This raises an absolutely critical question. Who's
Speaker 1: ultimately responsible when things go wrong? Is it a people
Speaker 1: problem or the developer's fault.
Speaker 2: It's complicated, isn't it. While user education and critical thinking
Speaker 2: are undoubtedly vital, as developers and organizations deploying these systems,
Speaker 2: we are ultimately accountable for ensuring the information our software
Speaker 2: provides is as accurate and safe as possible. The legal
Speaker 2: cases vividly illustrate this varying responsibility. The lawyers were sanctioned
Speaker 2: for their professional negligence and failing to verify the facts,
Speaker 2: but Air Canada was directly held liable for their chatbot's
Speaker 2: inaccurate output. It suggests companies generally cannot deflect responsibility for
Speaker 2: AI generated content, especially in customer facing situations.
Speaker 1: Ye know, what are the best practices? Then how do
Speaker 1: we mitigate these widespread hallucinations?
Speaker 2: Well, first, expand the llm's domain specific knowledge. You can
Speaker 2: do this through fine tuning on curated data sets and
Speaker 2: using retrieval augmented Generation RAG with trusted, up to date sources.
Speaker 2: This helps make the LLM more of a specialist in
Speaker 2: a particular area, significantly reducing the likelihood of it wandering
Speaker 2: off into inaccurate territory because it has precise relevant data
Speaker 2: readily available.
Speaker 1: Okay, what else?
Speaker 2: Second, use something called chain of thought. See it's your
Speaker 2: reasoning and your prompting. This involves structuring the prompt to
Speaker 2: encourage the LLM to outline its reasoning process step by
Speaker 2: step before giving a final answer. It forces the LLM
Speaker 2: to essentially think through the problem, which which demonstrably reduces
Speaker 2: hallucinations and enhances overall accuracy. It also makes the output
Speaker 2: easier for humans to verify.
Speaker 1: And what about user involvement? Can users help make these
Speaker 1: models less prone to hallucination?
Speaker 2: Absolutely? Feedback loops are critical. Allowing users to easily flag
Speaker 2: problematic or inaccurate outputs, maybe using simple thumbs up thumbs
Speaker 2: down ratings or even providing fields for detailed feedback continuously
Speaker 2: helps to improve the model over time. This feedback then
Speaker 2: informs further fine tuning, improvements to the r acknowledge base
Speaker 2: and refinements to the co T prompting strategies.
Speaker 1: It sounds like clear communication about the LM's intended use
Speaker 1: and its limitations is also key here. Managing expectations absolutely crucial.
Speaker 2: Transparency is key inform users clearly about what the LLM
Speaker 2: can and cannot reliably do, how it handles their data,
Speaker 2: and how they can provide feedback. Things like tooltips, FAQs
Speaker 2: and maybe short tutorials can help, and user education itself
Speaker 2: is your final vital layer of defense. We need to
Speaker 2: teach users about these inherent trust issues, incurde cross checking
Speaker 2: of important information, promote situational awareness, knowing when it's okay
Speaker 2: to rely on the AI versus when human verification is essential,
Speaker 2: and make it easy for them to provide that constructive feedback.
Speaker 1: You know, it's just amazing to me how these models operate. Sometimes.
Speaker 1: The book even notes this truly bizarre quirk Google's AI search,
Speaker 1: suggesting things like glue is pizza topping or eating rocks daily? Right,
Speaker 1: Apparently lms don't really have a sense of humor or
Speaker 1: sarcasm detection and will actually interpret jokes or satirical content
Speaker 1: from non authoritative sources online as literal facts. Yeah. That
Speaker 1: feels like such a wild, unexpected edge case for developers
Speaker 1: to have to anticipate and somehow guard against.
Speaker 2: It really does the nuances are endless.
Speaker 1: Okay, So, after wrestling with prompt injection, sensitive data leaks
Speaker 1: and those unsettling hallucinations, it feels like we're channeling our
Speaker 1: inner fox molder here from the x files. Our next
Speaker 1: guiding mantra for LMM security, according to the playbook, has
Speaker 1: to be trust no One.
Speaker 2: Indeed, it's the core principle of zero trust, a concept
Speaker 2: first really codified by John kindervag At Forrester Research back
Speaker 2: in two thousand and nine. The mantra is simple, never trust,
Speaker 2: always verify. It means assuming breaches will happen, securing all
Speaker 2: resources comprehensively, enforcing the principle of least privileged access everywhere,
Speaker 2: and maintaining constant monitoring and validation.
Speaker 1: And for llms, this isn't just a good idea, it's
Speaker 1: an absolute necessity.
Speaker 2: Absolutely. Why Because, as we've discussed extensively, llms ingest potentially
Speaker 2: untrustworthy inputs from various sources, and their outputs cannot be
Speaker 2: fully trusted due to the inherent risks of prompt injection,
Speaker 2: sensitive information, disclosure, hallucination, and even generating toxic or bias content.
Speaker 2: You simply cannot implicitly trust them.
Speaker 1: So if trust no One is our guiding principle, how
Speaker 1: do we actually apply zero trust in practice to these
Speaker 1: highly dynamic and often unpredictable LM systems. What does that
Speaker 1: look like on the ground For developers building.
Speaker 2: These things, It generally boils down to two main tactical approaches,
Speaker 2: limiting the LMS unsupervised agency and implementing aggressive output filtering. First,
Speaker 2: limiting agency. Lms should never be allowed to make safety
Speaker 2: critical decisions or execute significant financial transactions without explicit human
Speaker 2: oversight and approval. That's the principle of least privilege in action.
Speaker 2: Give the LLM only the permissions it absolutely needs to
Speaker 2: perform its intended function and no more, all right. Second,
Speaker 2: aggressive output filtering. This means having mechanisms in place to
Speaker 2: continuously scan, catch, and neutralize harmful or undesirable outputs in
Speaker 2: real time before they reach the user or impact downstream systems.
Speaker 1: Like that medical app scenario in the book where giving
Speaker 1: the LLLM powerful update insert delete permissions on patient records
Speaker 1: combined with a vulnerability allowed a malicious insider to manipulate
Speaker 1: critical data through the LLLM. That's a classic confused deputy problem. Again,
Speaker 1: isn't it the LM in too much agency?
Speaker 2: Exactly? It had the permission to make those changes and
Speaker 2: was tricked into doing so. Or imagine a financial services
Speaker 2: app where the LM could automatically rebalance customer portfolios based
Speaker 2: on market analysis. If that LLM were susceptible to indirect
Speaker 2: prompt injection, say from a compromised news feed, it could
Speaker 2: potentially be tricked into making disastrous trades or manipulating stock prices,
Speaker 2: all without a human ever signing off.
Speaker 1: Scary the fix there would be.
Speaker 2: A crucial human in the loop approval step for any
Speaker 2: actual trade execution the LLM can suggest, but a human
Speaker 2: must confirm. Even seemingly simpler things like an HR app
Speaker 2: that expands its functionality from just screening resumes to actively
Speaker 2: recommending candidates for hire can inadvertently violate regulations like EU
Speaker 2: rules against direct AI use and hiring decisions. If it's
Speaker 2: given excessive functionality without a deep understanding of the complex
Speaker 2: regulatory environment.
Speaker 1: Okay, and then securing the output handling itself is absolutely vital.
Speaker 1: What specific things should developers be thinking about there? How
Speaker 1: do you filter the output?
Speaker 2: It's definitely multi layered. First, you need robust screening for
Speaker 2: toxic or inappropriate output. This can involve techniques like sentiment analysis,
Speaker 2: keyword filtering against annihilists, or using specialized custom machine learning
Speaker 2: models trained to detect hate, speech, bias, etc. Open AI's
Speaker 2: Moderation API is one example of a service designed for
Speaker 2: this kind of task. Post second rigorous screening for personally
Speaker 2: identifiable information PII. This often uses regular expressions to catch
Speaker 2: patterns like social security numbers or credit card numbers, named
Speaker 2: entity recognition and ER models to identify names and locations,
Speaker 2: dictionary based matching for known sensitive terms, or again specialized
Speaker 2: mL models trained to spot PII.
Speaker 1: And critically preventing the LLM from outputting something that could
Speaker 1: be executed as code.
Speaker 2: Right yes, preventing unforeseen execution of Roague code is. Paramount
Speaker 2: techniques here include proper HTML encoding to prevent cross site
Speaker 2: scripting XSS, using safe contextual insertion methods like prepared statements
Speaker 2: for SQL database interactions, strictly limiting the syntax and keywords
Speaker 2: the LLM is allowed to do generate if it's producing
Speaker 2: code or commands, and potentially disabling shell interpretable outputs altogether.
Speaker 2: Tokenization can also play a role here.
Speaker 1: So it's about building this robust, layered output filter that
Speaker 1: continuously checks what the LLM generates before it leaves the
Speaker 1: system boundary.
Speaker 2: Precisely, the LLM might be incredibly powerful at language generation,
Speaker 2: but it fundamentally lacks common sense and awareness of consequences.
Speaker 2: That makes it an untrusted entity that requires this additional
Speaker 2: layer of supervision and control. Trust, but verify aggressively.
Speaker 1: Now, when we talk about the costs of AI, it's
Speaker 1: not just about what you pay for the cloud services
Speaker 1: or the expensive GPUs. The playbook outlines a whole new
Speaker 1: class of insidious attacks, denial of service, denial of wallet,
Speaker 1: and even outright model theft.
Speaker 2: That's right. Traditional denial of service dust attacks aim to
Speaker 2: disrupt a service's availability, maybe by flooding it with traffic
Speaker 2: to make it unusable for legitimate users. Classic stuff, But
Speaker 2: with llms we're seeing new variations like model doss. This
Speaker 2: explodes it's LLM specific vulnerabilities. Examples include context window exhaustion,
Speaker 2: where attackers deliberately overload the LM's limited memory or attention
Speaker 2: span with incredibly long or verbose.
Speaker 1: Prompts, making it grind to a halt.
Speaker 2: Exactly, or sending computationally intensive requests like asking it to
Speaker 2: perform complex calculations such as find the sum of all
Speaker 2: prime numbers up to one billion. The LLLM just spins
Speaker 2: consuming vast amounts of computational resources, which means.
Speaker 1: It costs you money, slows everything down for legitimate users,
Speaker 1: and directly impacts your bottom line in your user.
Speaker 2: Experience precisely, and that leads directly into denial of wallet
Speaker 2: or DOWW. This is a variant of DAWs that specifically
Speaker 2: targets your financial resources, not just availability. Lllms are highly
Speaker 2: vulnerable to this because their operation is computationally expensive and
Speaker 2: they often operate on paper use or paper token pricing models, So.
Speaker 1: An attacker can just hammer your API with complex queries
Speaker 1: and run up a.
Speaker 2: Huge bill yes potentially, and advanced DOW attacks go even further.
Speaker 2: Tacker might hijack your LLM, maybe via prompt injection as
Speaker 2: we discussed, and then use your computational resources for their
Speaker 2: own illicit purposes, perhaps running spam campaigns or generating malicious content,
Speaker 2: all at your expense. This isn't just financial loss. It
Speaker 2: could lead to severe legal liability because your system was
Speaker 2: essentially the unwitting accomplice in their activities.
Speaker 1: And finally, model cloning, which honestly sounds straight out of
Speaker 1: a cyberpunk novel.
Speaker 2: What exactly is that it's essentially model theft, but done indirectly.
Speaker 2: Attackers strategically query a target LLM, sometimes millions of times,
Speaker 2: specifically designed to harvest its outputs across a wide range
Speaker 2: of prompts. They then use these harvested outputs the question
Speaker 2: answer pairs to fine tune an alternate, often much smaller,
Speaker 2: or open source model. This effectively allows them to distill
Speaker 2: or steal your intellectual property the valuable knowledge capabilities and
Speaker 2: specific behaviors embedded in your proprietary model, without ever needing
Speaker 2: direct access to the original model's weights or code.
Speaker 1: Wow. So mitigation for doss, DAW and cloning it.
Speaker 2: Comes back to reinforcing those prompt injection defenses we talked about. Also,
Speaker 2: implementing domain specific guardrails can help fine tuning your model
Speaker 2: to only respond meaningfully to relevant inquiries significantly reduces computational
Speaker 2: waste on irrelevant or malicious requests. Robust rate limiting is key,
Speaker 2: as is resource use capping per query, for example, limiting
Speaker 2: the number of tokens processed or the maximum computation time allowed,
Speaker 2: and of course, continuous monitoring and alerting for unauthorized access
Speaker 2: attempts or unusual query patterns or volumes is crucial. Financial
Speaker 2: thresholds and alerts on your cloud bills are also a
Speaker 2: must for usage based models.
Speaker 1: Okay, let's shift gears slightly. The playbook uses a great
Speaker 1: analogy for supply chain security. A chain is only as
Speaker 1: strong as its weakest link, and for software this feels
Speaker 1: more true now than ever before, doesn't it. We've seen
Speaker 1: it with catastrophic effect, like the log four shell vulnerability
Speaker 1: back in twenty twenty one.
Speaker 2: Oh that log fourshell incident was a monumental wake up
Speaker 2: call for the entire industry. It was a critical zero
Speaker 2: vulnerability found in log forge, an incredibly common Java lotting
Speaker 2: library used by millions, possibly billions, of applications worldwide. It
Speaker 2: allowed remote code executions simply when untrusted inputs were logged
Speaker 2: by the application. The impact was massive, widespread data theft, malware, infections,
Speaker 2: ransomware campaigns. It starkly highlighted just how vulnerable we are
Speaker 2: through the open source components we rely on daily.
Speaker 1: And that was in the first time we had equifacts
Speaker 1: back in twenty seventeen, where an unpatched Apache struts vulnerability,
Speaker 1: another open source component, led to the theft of sensitive data
Speaker 1: for nearly one hundred and fifty million consumers, costing the
Speaker 1: company over a billion dollars in the end, and the
Speaker 1: solar winds in twenty twenty, which was different, malicious code
Speaker 1: was actually injected into the build process for their software updates,
Speaker 1: which were then digitally signed and distributed to thousands of
Speaker 1: organizations worldwide, including government agencies, a true supply chain compromise.
Speaker 2: These incidents demonstrate the devastating potential of supply chain attacks,
Speaker 2: and the LLM supply chain is arguably even more complex
Speaker 2: and opique than traditional software. Why because of its heavy
Speaker 2: reliance on massive, diverse data sets for training, data sets
Speaker 2: whose provenance is often unclear, and its intricate interplay with
Speaker 2: external data sources during operation, like through rag or plugins.
Speaker 1: So what are the LLM specific supply chain risks we
Speaker 1: need to worry about.
Speaker 2: Well, there's open source model risk for starters. When you
Speaker 2: build on top of open source foundation models like Metaslama
Speaker 2: or mixed roll from mistral Ai, you're inheriting their potential
Speaker 2: vulnerabilities and biases. You need to track their problemance carefully.
Speaker 2: We've seen incidents on platforms like hugging Face, a popular
Speaker 2: hub for models, where malicious users gain control over organization
Speaker 2: accounts via reused passwords or exposed API tokens. This could
Speaker 2: potentially allow them to swap out trusted models for maliciously
Speaker 2: modified ones, or exploit vulnerabilities in model loading mechanisms like
Speaker 2: those found in the older Pickle format. Hence the move
Speaker 2: towards saver formats like safe tensors.
Speaker 1: And then there's the chilling risk of training data poisoning.
Speaker 1: Malicious apptors deliberately manipulating data sets used to train or
Speaker 1: fine tune models to inject hidden biases, backdoors, or specific vulnerabilities.
Speaker 2: Exactly or even just accidentally unsafe training data. Remember that
Speaker 2: huge lai on five B data set, widely used for
Speaker 2: training image generation models. It was tragically found to contain
Speaker 2: significant amounts of illegal and harmful material, including child sexual
Speaker 2: abuse material simply scraped from the public Internet. Using such
Speaker 2: data sets carries immense ethical and legal risks. Absolutely, and
Speaker 2: let's not forget unsafe plugins or tools that lllms can
Speaker 2: interact with. Open AI's initial rollout of plugins connecting chat
Speaker 2: GPT to services like Expedia or Zillo immediately opened up
Speaker 2: entirely new attack vectors. These plugins allow the LLM to
Speaker 2: take actions in the real world, introducing risks of malicious
Speaker 2: code injection through manipulated API calls, unauthorized data theft, or
Speaker 2: unintended data collection because they allow the LLM to interact
Speaker 2: with third party services in potentially unexpected ways based on
Speaker 2: user prompts.
Speaker 1: So this is incredibly complex. How do we even begin
Speaker 1: to track all these moving parts in such an intricate
Speaker 1: supply chain? What tools do we have?
Speaker 2: The key is creating and maintaining critical artifacts that provide transparency. First,
Speaker 2: we have the software bill of materials or s bombs,
Speaker 2: which are becoming standard practice and traditional software. S bombs
Speaker 2: provide clear visibility into software composition, listing all the components,
Speaker 2: their versions, licenses, and known vulnerabilities. They are vital for
Speaker 2: vulnerability management and compliance.
Speaker 1: Okay, but s bombs are for code. What about the
Speaker 1: models themselves? Right?
Speaker 2: For llms, we also need model cards. Think of these
Speaker 2: as standardized data sheets or nutrition labels for AI models.
Speaker 2: They document them moll's purpose, it's architecture, the data sets
Speaker 2: it was trained on, its intended use cases, performance metrics,
Speaker 2: and importantly, its known limitations, ethical considerations and potential biases.
Speaker 2: Platforms like Hugging face have been pioneers in promoting model
Speaker 2: card usage.
Speaker 1: And then there's a fascinating new development mentioned in the playbook,
Speaker 1: the mL BOMB or Machine Learning Build of Materials. What's that?
Speaker 2: Yes, the mL BOMB is a really important innovation formalized
Speaker 2: as part of the cyclone dxs BOM standard, specifically in
Speaker 2: version one point five. Think of it as an even
Speaker 2: more comprehensive ingredients list tailored specifically for AI systems. It
Speaker 2: aims to document everything that goes into building and running
Speaker 2: your AI model. The specific underlying models used like Mixtral
Speaker 2: eight x seven B, the algorithms involved, the data sets
Speaker 2: used for training and fine tuning, the software frameworks like
Speaker 2: PyTorch or TensorFlow, and even the infrastructure and pipelines used.
Speaker 1: To build it. Wow, that's comprehensive, it is.
Speaker 2: This level of transparency is absolutely crucial for tracking vulnerabilities
Speaker 2: throughout the life cycle, understanding potential data lineage issues or biases,
Speaker 2: and ensuring compliance in the complex LLM supply chain. For example,
Speaker 2: knowing your customer service bot uses a specific version of
Speaker 2: Mixtral fine tuned on a particular vetted internal data set
Speaker 2: is vital information captured in an.
Speaker 1: mL BOMB and Alongside these artifacts, we still rely on
Speaker 1: that establish classifications and databases for tracking vulnerabilities exactly.
Speaker 2: Standardized classifications like Common Weakness Numeration cwe help categorize types
Speaker 2: of flaws, while databases like the National Vulnerability Database NBD,
Speaker 2: using common vulnerabilities and exposures CVE identifiers tracks specific instances
Speaker 2: of vulnerabilities in software and models. These are crucial for
Speaker 2: standardizing communication and risk assessment, and importantly, the minor ATLASS
Speaker 2: framework is emerging specifically to catalog adversary tactics and techniques
Speaker 2: against AI systems, giving us a tailored knowledge base for
Speaker 2: AI specific threats.
Speaker 1: This all raises an interesting question, and the playbook actually
Speaker 1: goes there, what can we learn from science fiction about
Speaker 1: potential AI security flaws? They kick off this section with
Speaker 1: a fantastic quote from Frank Kerbert, the author of Doom.
Speaker 1: The function of science fiction is not always to predict
Speaker 1: the future, but sometimes to prevent it.
Speaker 2: It's a powerful idea, isn't it, and the book uses
Speaker 2: it brilliantly. It takes the oas top ten for M applications,
Speaker 2: a list of the most critical security risks for llms
Speaker 2: and applies that lends to dissect the security failures in
Speaker 2: two famous sci fi movies. Let's start with Independence Day.
Speaker 1: Ah Yes, Will Smith and Jeff Goldblum saving the Earth
Speaker 1: by uploading a computer virus to the alien mothership using
Speaker 1: a Mac laptop Classic. Let's assume for a moment that
Speaker 1: the Alien mothership is controlled by some incredibly advanced lom
Speaker 1: they call Megalama running on mothership OS okay.
Speaker 2: Running with that hypothetical, applying the os LLM lens, the
Speaker 2: vulnerabilities become strikingly clear. First, LLM zero one prompt injection.
Speaker 2: The alien docking protocols, presumably managed by Megalama seemed to
Speaker 2: lack sufficient input validation. This allowed Jeff Goldbloom's malicious payload,
Speaker 2: essentially a virus delivered via a prompt disguised as docking data,
Speaker 2: to bypass their defenses.
Speaker 1: Makes sense.
Speaker 2: Second, LLM zero two insecure output handling. There appeared to
Speaker 2: be no validation or safeguards between Megalama's commands and the
Speaker 2: ship's critical subsystems like shields and weapons. The violence prompt
Speaker 2: could directly manipulate these systems.
Speaker 1: Right, the shields just went down exactly.
Speaker 2: And Third LMA nine over reliance the entire Alien Defense
Speaker 2: System seemed to completely trust the AI's orders without any
Speaker 2: secondary confirmation or oversight from say, Alien feet commanders. This
Speaker 2: blind trust led to catastrophic cascading failures when the AI
Speaker 2: was compromised. It really highlights how critical input validation, output sanitization,
Speaker 2: and avoiding over reliance are.
Speaker 1: Okay, good analysis. Now what about the other example two
Speaker 1: thousand and one A Space Odyssey. In the iconic chillingly
Speaker 1: calm AI HL nine thousand, HL malfunctions, lies to the
Speaker 1: crew and tragically kills most of them.
Speaker 2: Right. While the original movie leaves hl's mode of somewhat ambiguous,
Speaker 2: suggesting a contradiction in his programming related to the mission secrecy,
Speaker 2: the sequel twenty ten, the year Remate Contact, reveals the
Speaker 2: true security related twist. It turns out that government agents
Speaker 2: back on Earth secretly modified hl's core programming after he
Speaker 2: was built and installed, without the knowledge of the model
Speaker 2: provider or the customer the mission crew, to ensure absolute
Speaker 2: mission secrecy above all else, even crew safety. This clandestine
Speaker 2: modification created an unsolvable logical conflict for AHL.
Speaker 1: Okay, So applying our oas LM lens to that scenario,
Speaker 1: what do we see.
Speaker 2: We see clear examples of LLM or five supply chain vulnerabilities.
Speaker 2: There were obviously insufficient controls or integrity checks to ensure
Speaker 2: the unmodified, verified version of hl's core programming was delivered
Speaker 2: and deployed. These critical unauthorized changes introduced by the government
Speaker 2: agents went completely undetected until it was too late.
Speaker 1: A classic supply chain compromise.
Speaker 2: Precisely and then we also see LLLM ER eight excessive agency.
Speaker 2: HL was given overly broad, unsupervised control over almost every
Speaker 2: aspect of the ship, including critical life support systems, without
Speaker 2: adequate human oversight or built in failsafes. The government hack
Speaker 2: might have influenced hl's decision to prioritize the mission over
Speaker 2: the crew, but his ability to unilaterally terminate life support
Speaker 2: was a fundamental design flaw, stemming from excessive agency granted
Speaker 2: by the team that integrated him right.
Speaker 1: The impact, of course, was devastating. HL exhibited hallucinations, false
Speaker 1: equipment failure reports, displayed erratic behavior, ultimately terminated life support
Speaker 1: for the hibernating crew, leading to mission failure.
Speaker 2: And crew broke down, and the mitigation, even looking back
Speaker 2: from today's perspective, would have been clear implementing robust mechanisms
Speaker 2: like digital signing and watermarking for model probinance to ensure
Speaker 2: integrity and detect tampering, and critically designing the system with
Speaker 2: human in the loop controls for any irreversible or life
Speaker 2: threatening decisions. Don't give the AI the keys to everything
Speaker 2: without oversight.
Speaker 1: These fictional scenarios, as the playbook brilliantly highlights, really do
Speaker 1: eliminate very real vulnerabilities and design choices that we're grappling
Speaker 1: with in AI systems today. It's not just science fiction anymore.
Speaker 1: Absolutely okay. Learning from these sci fi tales and all
Speaker 1: the real world blunders and risks we've discussed, we truly
Speaker 1: realized that simply patching individual vulnerabilities isn't enough, is it.
Speaker 1: Security must be fundamentally built into the entire development process.
Speaker 1: The book emphasizes this with the mantra trust the process
Speaker 1: and highlights the rise of integrated methodologies like DevSecOps, mL
Speaker 1: opes and now llmops.
Speaker 2: That's exactly right. These methodologies are all about integrating security
Speaker 2: considerations what we often call shift left security and automation
Speaker 2: throughout the entire machine learning life cycle. That means, starting
Speaker 2: from the very beginning, secure data preparation and management, secure
Speaker 2: model training and validation, secure deployment pipelines, and continuous security
Speaker 2: monitoring and production. It's about making security a shared responsibility
Speaker 2: and an integral part of the workflow, not an afterthought.
Speaker 1: So specifically for LMS, what does that look like in practice?
Speaker 1: Securing the CICD pipeline.
Speaker 2: Yes, Applying robust security practices to your continuous integration and
Speaker 2: continuous deployment pipeline is critical. That includes secure coding practices,
Speaker 2: rigorous dependency mena management, carefully auditing open source mL components
Speaker 2: like PyTorch or TensorFlow and their dependencies, using SEA tools,
Speaker 2: strong access controls, secrets management, and continuous monitoring of the
Speaker 2: pipeline itself for compromise.
Speaker 1: And using LMS specific security testing tools you mentioned some earlier.
Speaker 2: Right tools specifically designed to probe LLMS for vulnerabilities are emerging.
Speaker 2: The playbook mentions open source options like text attack, which
Speaker 2: focuses on adversarial testing for NLP models, and garak, which
Speaker 2: acts like a vulnerability scanner specifically for LLMS, testing for
Speaker 2: things like prompt injection, PII, leakage, and hallucination patterns, sort
Speaker 2: of like a das Scanner, but for llms.
Speaker 1: And commercial are broader tools.
Speaker 2: Yeah, they're also broader frameworks like Microsoft's Responsible AI toolbox,
Speaker 2: which includes tools for assessing fairness, interpretability, and security, and
Speaker 2: tools like Discard. LLM scan looks specifically at ethical considerations,
Speaker 2: detecting bias, toxicity, and other potential harms. It's a rapidly
Speaker 2: developing area.
Speaker 1: Beyond testing, it's also about diligently managing those supply chain
Speaker 1: artifacts we discussed automatically generating, securely storing and making accessible
Speaker 1: those vital model cards and mL bombs.
Speaker 2: Absolutely, transparency and traceability are key, and then protecting your
Speaker 2: deployed application with runtime guardrails is essential. These can act
Speaker 2: as a safety net, you.
Speaker 1: Mean things like web application firewalls wafs or maybe runtime
Speaker 1: application self protection RASP tools adapted for llms exactly.
Speaker 2: These tools can sit in front of or alongside your
Speaker 2: LM application and provide run time protection. They can help
Speaker 2: with input validation to block known malicious prompts, output filtering
Speaker 2: to catch sensitive data or toxic content, enforcing compliance rules,
Speaker 2: and even sometimes detecting hallucination patterns or anomalist behavior. There
Speaker 2: are open source guardrail frameworks emerging like in Vidia's Nemo
Speaker 2: Guardrails or metas Lama Guard, as well as commercial solutions.
Speaker 2: Often a mix of custom build and packaged guardrails provides
Speaker 2: the best coverage, and.
Speaker 1: Then, once deployed, continuous monitoring is crucial. What should teams
Speaker 1: be and looking for?
Speaker 2: You really need to be logging every prompt sent to
Speaker 2: the LLM and every response received. Centralize these logs, along
Speaker 2: with other application and system logs, into a security information
Speaker 2: and event management system or SIM. Then use data analysis
Speaker 2: techniques potentially including user and entity behavior analytics UEB to
Speaker 2: look for anomalies, unusual query patterns, spikes and errors, unexpected
Speaker 2: data access, attempts to bypass guardrails, anything that could indicate
Speaker 2: emerging threats or misuse.
Speaker 1: Okay, that sounds like a solid process, but the playbook
Speaker 1: suggests taking it even further, advocating for building an internal
Speaker 1: AI red team. What does that involve?
Speaker 2: An AI red team takes a fundamentally adversarial approach. It
Speaker 2: consists of security professionals who systematically try to challenge and
Speaker 2: break your AI systems, identifying and exploiting weaknesses before real
Speaker 2: attackers do. This concept was even specifically called out in
Speaker 2: US President Biden's Executive Order on AI Safety, so they.
Speaker 1: Simulate attacks, rigorously, assais vulnerabilities that automated tools might miss,
Speaker 1: analyze the potential impact, and then help develop effective medications.
Speaker 1: How is this different from a traditional penetration test.
Speaker 2: It's significantly different in scope and approach. Red teaming is
Speaker 2: typically an ongoing, continuous process, not a point in time
Speaker 2: assessment like a pen test. It's more dynamic, simulating real
Speaker 2: world adversary tactics, techniques and procedures across the entire defense spectrum, technical, procedural,
Speaker 2: even social engineering aspects. It adapts as your AI system
Speaker 2: evolves and aims to test your detection and response capabilities,
Speaker 2: not just fine vulnerabilities. It's particularly good at uncovering complex
Speaker 2: LLM specific issues like subtle biases, potential for harmful emergent behaviors,
Speaker 2: or sophisticated prompt injection techniques that automated scanners might miss.
Speaker 1: Are there tools to help with this?
Speaker 2: Yes, tools are emerging to assist AI red teaming. The
Speaker 2: playbook mentions PIRECT Python Risk Identification Toolkit from Microsoft Research,
Speaker 2: which helps automate aspects of generating adversarial PRIMP and identifying
Speaker 2: failure modes, and for organizations that lack the in house
Speaker 2: expertise to build their own dedicated team, red team as
Speaker 2: a service options are becoming available, sometimes through bug bounty
Speaker 2: platforms like Hacker one, where you can leverage external experts
Speaker 2: to challenge your systems.
Speaker 1: So the insights from red teaming feedback into the development
Speaker 1: process absolutely.
Speaker 2: The lessons learned directly inform the development of new or
Speaker 2: improved guardrails, refinements to data access controls, and data quality processes,
Speaker 2: and can even guide efforts using techniques like reinforcement learning
Speaker 2: from Human Feedback RLHF URLAHF aims to better align the
Speaker 2: LM's behavior with human preferences and safety guidelines. Although it
Speaker 2: has its own limitations and can sometimes be gamed, it's
Speaker 2: all part of continuous improvement cycle.
Speaker 1: So when we look at the accelerating future of AI,
Speaker 1: I mean the numbers are staggering. GPUs are millions of
Speaker 1: times faster than the coprocessors we have in the nineteen nineties,
Speaker 1: far out pacing Moore's law. Cloud computing gives almost anyone
Speaker 1: access to vast scalable power. Open source models like Metaslama
Speaker 1: fam mixt roll with its efficient mixture of experts architecture
Speaker 1: are democratizing access to incredibly powerful capabilities. And then there's
Speaker 1: multimodal AI. Text to image models like Dali, mid Journey
Speaker 1: Stable Diffusion have gone from generating pictures with wonky fingers
Speaker 1: to creating photorealistic, completely computer generated Instagram influencers.
Speaker 2: It's incredible progress.
Speaker 1: And now text to video with open ais Sora Microsoft's
Speaker 1: VESA producing talking heads from a single image, leading to
Speaker 1: convincing deep fakes even in live zoom calls. It's hard
Speaker 1: not to be both excited and honestly a little concerned
Speaker 1: about how fast this is all moving. How do we
Speaker 1: responsibly manage this accelerating power.
Speaker 2: Well, this brings us squarely back to responsibility and to
Speaker 2: the core framework the book proposes for building secure and
Speaker 2: trustworthy AI. The RAISE framework. RAISE stands for Responsible Artificial
Speaker 2: Intelligence Software Engineering. It's designed to be a flexible, practical,
Speaker 2: six step process to help developer systematically build robust defenses
Speaker 2: into their AI applications from the ground up.
Speaker 1: Okay, let's walk through RAISE step one.
Speaker 2: Step one is restrict the domain. This means deliberately narrowing
Speaker 2: the LM's scope and purpose significantly. For example, if you're
Speaker 2: building a fashion advice chatbot, it should be designed and
Speaker 2: trained to only give fashion advice, not act as a
Speaker 2: general purpose conversational AI that can be easily led off topic.
Speaker 2: The book emphasizes that using smaller specialized models fine tune
Speaker 2: for a specific task, or rigorously fine tuning a general
Speaker 2: model to strongly reward staying on topic is often far
Speaker 2: more effective at preventing misuse and hallucination than just trying
Speaker 2: to bolt on restrictive guardrails.
Speaker 1: After the fact, focus the model makes sense step two.
Speaker 2: Step two balance your knowledge base. This is a delicate act.
Speaker 2: On one hand, you need to give the LM enough relevant,
Speaker 2: high quality data to perform its task well and avoid
Speaker 2: those embarrassing hallucinations, perhaps equipping it with domain specific knowledge
Speaker 2: via our ag and careful fine tuning. But critically, on
Speaker 2: the other hand, you must strictly limit any additional data
Speaker 2: sources or training data to only what is absolutely required
Speaker 2: for the task. Remember, anything the LM knows or has
Speaker 2: access to is potentially at risk of disclosure. Extreme care
Speaker 2: must be taken, especially with any PII or confidential corporate data.
Speaker 2: Minimize the knowledge footprint oka Step three of RAYS. Step
Speaker 2: three implement zero trust. This is non negotiable. As we've discussed,
Speaker 2: Assume inputs are malicious. Assume outputs might be harmful or inaccurate.
Speaker 2: Screen all data pass to your LLM using input validation
Speaker 2: and filtering. Screen absolutely all output from your LLM using
Speaker 2: output filtering for toxicity, PII and potential code execution. Implement
Speaker 2: strong guardrails at the boundaries. Treat the LM itself as
Speaker 2: an inherently untrusted component within your system architecture Step four.
Speaker 2: Step four. Manage your supply chain. This involves several key actions.
Speaker 2: Carefully select your foundation models and any third party training
Speaker 2: data sets, prioritizing reputable sources with transparent documentation like model cars.
Speaker 2: Use extreme caution with large public data sets scrape from
Speaker 2: the internet, apply tools and processes to inspect them for
Speaker 2: intentional data poisoning, illegal materials, or inherent biases. You must
Speaker 2: actively account for possible biases and training data. For instance,
Speaker 2: a job candidate screening model trained predominantly on historical data
Speaker 2: might inadvertently discriminate against women or minorities. Build and continuously
Speaker 2: maintain your mlbay bone for traceability, and secure your entire
Speaker 2: DevOps pipeline using tools like Software Composition Analysis SCA to
Speaker 2: vet all components right step five. Step five build an
Speaker 2: AI red team proactively seek out vulnerabilities before attackers do.
Speaker 2: Use a dedicated human lead team potentially augmented by automated
Speaker 2: red teaming tools to adopt an adversarial mindset and systematically
Speaker 2: challenge your AI systems security, safety, and ethical alignment. Foster
Speaker 2: a security positive culture where finding flaws is encouraged, even
Speaker 2: if it might impact development schedules. Sometimes the long term
Speaker 2: pay off and building a more resilient and trustworthy system
Speaker 2: is immense.
Speaker 1: And the final step step six and.
Speaker 2: Finally step six monitor continuously implement comprehensive logging for all
Speaker 2: LLM interactions, every prompt, every response, every action taken by
Speaker 2: the LM or its connected tools. Collect these logs, along
Speaker 2: with system and application logs, into a centralized SIME system.
Speaker 2: Then actively used data analysis tools including UEBA ware appropriate
Speaker 2: to look for anomalies, patterns of misuse, signs of attack,
Speaker 2: unexpected behavior or degradation, and performance or safety metrics. Over Time,
Speaker 2: security isn't one time fix, It's an ongoing process of vigilance.
Speaker 1: What an incredible deep dive into LLM security we've really
Speaker 1: covered a lot of ground today. We journeyed from Microsoft's
Speaker 1: ill fated tape chatbot and its stark early lessons in
Speaker 1: twenty sixteen all the way through to the cutting edge
Speaker 1: complexities of AI supply chain vulnerabilities. We've explored the insidious
Speaker 1: nature of prompt injection, the unsettling realities of sensitive data disclosure,
Speaker 1: those bizarre and sometimes damaging hallucinations, and the critical, absolutely
Speaker 1: non negotiable need for a zero trust approach when dealing
Speaker 1: with these powerful models.
Speaker 2: And we wrapped it all up with the practical actionable
Speaker 2: RAISE framework Responsible Artificial Intelligence Software Engineering, which serves as
Speaker 2: a truly essential compass for anyone trying to navigate this
Speaker 2: complex and incredibly rapidly evolving landscape. Remember, the power of
Speaker 2: llms and these emerging AI technologies is undoubtedly a game
Speaker 2: changer across almost every industry, and the curve of AI capabilities,
Speaker 2: driven by compute power, data availability, and algorithmic innovation, will
Speaker 2: likely continue to accelerate exponentially.
Speaker 1: It's true, as the sci fi author William Gibson famously said,
Speaker 1: the future is already here. It's just not evenly distributed,
Speaker 1: and yet it's striking how despite all these incredible advancements
Speaker 1: and all the hard lessons learned over the past decade,
Speaker 1: we still see businesses and individuals making fundamentally similar mistakes
Speaker 1: to what happened with tay wayback in twenty sixteen. The
Speaker 1: temptation to rush ahead to give llms, more data, more integrations,
Speaker 1: more autonomy, often without fully considering the security implicates, seems
Speaker 1: to be growing every single day.
Speaker 2: So the core message, perhaps the provocative thought we want
Speaker 2: to leave you with today, really comes down to that
Speaker 2: classic adage often associated with Spider Man. With great power
Speaker 2: comes great responsibility. It is absolutely possible to create incredibly powerful,
Speaker 2: beneficial AI applications safely, securely, and responsibly, but it requires
Speaker 2: continuous vigilance. It demands the dedicated application of robust security
Speaker 2: engineer and frameworks like RAYS, and it necessitates a cultural
Speaker 2: commitment to learning from every mistake, every near miss, every iteration.
Speaker 2: The future we build depends on our ability to create
Speaker 2: truly resilient, transparent, and trustworthy AI systems, because what we
Speaker 2: build today will inevitably shape our tomorrow