Speaker 1: Hey Carl here, we're on a mission to find new
Speaker 1: fans of the show. I mean, where else can you
Speaker 1: get real life comedy and horror in one podcast. You
Speaker 1: can do your part by leaving us a review on iTunes,
Speaker 1: Spotify or wherever you get Security this week. Also, you
Speaker 1: can get an ad free feed by becoming a five
Speaker 1: dollars a month patron. Go to Patreon dot Security this
Speaker 1: week dot com and sign up, and don't forget Discord
Speaker 1: Discord dot Security this week dot com. Lots of good
Speaker 1: stuff happening there. Yeah, I think that's it. So on
Speaker 1: with the podcast. Hey Patrick, Yeah, you you and your
Speaker 1: wife are come from German heritage.
Speaker 2: Right, I'm Irish, but she's German.
Speaker 1: Yeah, all right, So here's a German joke for her.
Speaker 2: Oh, I'm sure she'll love it.
Speaker 1: You know about the German baby that didn't talk no
Speaker 1: until he was five or so. He didn't say a
Speaker 1: word all his life. Five years old, he's at the
Speaker 1: dinner table and finally the first words he says is
Speaker 1: this thrillllle is tap and everybody they all freak out.
Speaker 1: Oh my god, Henry, you speak, You can speak? How
Speaker 1: come you haven't spoken up till now? Everything has been satisfactory. Hey,
Speaker 1: welcome back to Security this week. I'm Carl Franklin's Patrick
Speaker 1: Hines and Duaye Laflotte. We're here to discuss some of
Speaker 1: the things that happened last week. Anyway, I think we
Speaker 1: should start with this US Army sites hacked to display
Speaker 1: anti Trump pro Kurdistan messages. Wa, So what happened here?
Speaker 3: This is this is a Yeah, there were a couple
Speaker 3: different site basic hygiene. Yeah, agreed. So there's the Army
Speaker 3: AI Integration Center site. So we're not talking like, uh,
Speaker 3: you know, I don't know, Department of Wars website. Yeah,
Speaker 3: exactly right. Wait was that was that war games? Was
Speaker 3: that normat? I think it was. Yeah, that's a good
Speaker 3: callback batter, I know.
Speaker 1: Damn.
Speaker 3: So the Army AI Integration Center and the Open Innovation
Speaker 3: Lab run by the military, these are not systems. For
Speaker 3: those of you who have worked in you know, uh
Speaker 3: tss CI A tile environments, top secret clearance, compartmentalized information
Speaker 3: type stuff if you know the terms SIPPER and nipper
Speaker 3: if you know what those mean. These systems are not
Speaker 3: connected to those, right. These were like third party you
Speaker 3: know kind of websites almost toasted by go daddy style
Speaker 3: WordPress websites explaining you know what those departments do, but
Speaker 3: not you know, not officially run by them.
Speaker 1: I thought the hack was pretty innovative that the pages
Speaker 1: usually show a four oh four right, yes, not found,
Speaker 1: but known attacker modified the default error page and replaced
Speaker 1: it with their political notice.
Speaker 3: Yeah, and that's interesting too, because it's like, there's two
Speaker 3: reasons why you might might do that as an attacker. One,
Speaker 3: you know, maybe you get access to an endpoint that
Speaker 3: only has the ability to edit the WORL four. Like
Speaker 3: let's say this was a word Press site and there
Speaker 3: was a plugin that allows you to edit the foural
Speaker 3: four pages, and maybe that's the insecure plug in, so
Speaker 3: that's why they edited that Page's possible it was the
Speaker 3: only path, but the other path might be you know
Speaker 3: how blatantly obvious it is if your main web page
Speaker 3: has changed. But if somebody changes a four or four.
Speaker 2: Page, it might be a while.
Speaker 3: It might be a while before somebody finds it, sees it, whatever, right,
Speaker 3: and it gives you a little bit more time for
Speaker 3: the forensics on those servers to get wiped away. So
Speaker 3: that's that's my hacker advice. Yeah, Okay, don't always go
Speaker 3: for the front door. Sometimes the four or four is
Speaker 3: the way to go the four oh four.
Speaker 1: So Patrick, this next one you added as a kind
Speaker 1: of a last week thing, right.
Speaker 2: Yeah, I stumbled on it shortly after we recorded. There's
Speaker 2: two new fishing kits. And the thing that makes them
Speaker 2: interesting is that they get around multi factor authentication, especially
Speaker 2: in Office three sixty five, Microsoft three sixty five and
Speaker 2: up until now a lot of people have just thought that, well,
Speaker 2: I've got multi factor, so I'm fine, and you're never fine,
Speaker 2: just like little Henrick. It's not satisfactory. Nice callback. So
Speaker 2: you know this, Dwayne loves these, which means they're really bad,
Speaker 2: so good, really bad, Dwayne, why don't you talk about
Speaker 2: like why you love these so much?
Speaker 3: So these are these are awesome. People don't think about
Speaker 3: like everybody does this, but nobody thinks about what it's
Speaker 3: actually doing. So when we talk about tokens and this,
Speaker 3: this particular is a manipulation of tokens. And on this
Speaker 3: show we've talked about token tactics, we've talked about all
Speaker 3: sorts of things had to manipulate Office. That's and we've
Speaker 3: given this analogy. Before you know, you go buy a
Speaker 3: brand new smart TV and you fired up. You turn
Speaker 3: it on and hope you don't connect it to your
Speaker 3: real network. You connect it to some sort of IoT
Speaker 3: guess network, and you go to connect Netflix, and there's
Speaker 3: two options. There's you know, log in with your username
Speaker 3: and password, which is just a pain in the ass
Speaker 3: on that little remote control they give you. Or they
Speaker 3: give you a code on the screen and you can
Speaker 3: go to any browser and you can log into your
Speaker 3: Netflix account and you can type that code in. And
Speaker 3: when you do that, now that TV gets access to
Speaker 3: your account, and you're like, oh, this is nice and easy,
Speaker 3: right exactly, this is the same type of thing. You
Speaker 3: can do the same thing with your Microsoft account. So
Speaker 3: there's the ability to give IoT style access to your
Speaker 3: Microsoft account through a token.
Speaker 2: Which ranks right up there with home dentistry, those good ideas.
Speaker 3: Yep, literally, the same thing, the same thing you're going through.
Speaker 3: You get token in an email, you literally go to
Speaker 3: Microsoft's real site. Right, So if you're using a password manager,
Speaker 3: if you're using pass keys, there will be no warnings
Speaker 3: here because they're bringing you right to the real Microsoft
Speaker 3: site and when you get there, they're giving you the
Speaker 3: instructions on what to type in there? Oh, log in
Speaker 3: like you normally would. We need you to verify your
Speaker 3: account by typing in this code, you know, two six
Speaker 3: three four two eight writ whatever.
Speaker 1: It is both interesting is that code not only authenticates
Speaker 1: you but authorizes you too. Yeah, one little code.
Speaker 3: And what was awesome with the older token tactics is
Speaker 3: that what you got access to was this graph API,
Speaker 3: which gave you the ability to do all sorts of things
Speaker 3: like read emails and whatnot. But then every ninety days
Speaker 3: you could request a new token if you wanted to,
Speaker 3: so you could be in their end definitely, and if
Speaker 3: the user changed their password, it didn't matter.
Speaker 1: Sounds very convenient.
Speaker 3: Unless the user went in and like killed all tokens
Speaker 3: and said log out all my devices. The attackers were
Speaker 3: still in there. So it's it's a brilliant tactic. Honestly,
Speaker 3: I really I really like that they're reviving it. I
Speaker 3: mean I missed token tactics and now evil Tactics is out,
Speaker 3: so you know, it's it's good.
Speaker 1: It's all good.
Speaker 2: Yes, okay, okay.
Speaker 1: Along with that, and speaking of keys, the next story
Speaker 1: critical SharePoint remote code execution flaw exploited to steal machine keys.
Speaker 1: What's the machine key? Oh no, not that the machine key,
Speaker 1: the machine the key machine. Yes.
Speaker 3: Dot net has a whole bunch of keys. And for
Speaker 3: those of you who have heard of like the dp
Speaker 3: A p I, which is the Data protection API and Windows. Wow,
Speaker 3: how do we get into this without getting into too
Speaker 3: much crypto?
Speaker 1: Uh?
Speaker 3: There are more?
Speaker 2: Thing is too much crypto?
Speaker 3: Well, I don't.
Speaker 1: Let's talk about the math. The key is just a
Speaker 1: thank you Carl, a string of cars that identifies something
Speaker 1: to you know, think of it as a password, but
Speaker 1: it's more than that.
Speaker 3: It's used to encrypt, right, yeah, yeah, absolutely, And and
Speaker 3: what's neat about it? Used to be you would use
Speaker 3: the same key for every system, like, oh, we have
Speaker 3: this really long key and we use it for every
Speaker 3: system and blah blah blah, and you're encrypted and it's
Speaker 3: a symmetric key, and huzzak. Everything's great. The problem is
Speaker 3: if one of those systems got compromised, you could read
Speaker 3: the date on all the systems, So then they said, okay, well,
Speaker 3: you know'd be great is if each system had their
Speaker 3: own unique key, a machine key that could then do
Speaker 3: those levels of encryption and authorization or authentication of tokens
Speaker 3: and all sorts of stuff.
Speaker 1: Well, but the interesting thing is every network connection has
Speaker 1: its own unique identifier that can be used as a key.
Speaker 3: Yeah, absolutely, yeah, And there's there's all sorts of HMAC,
Speaker 3: there's all sorts of things you can use. Just the
Speaker 3: machine key is typically tied to, you know, a dot
Speaker 3: ED installation of well really anything. Every Windows computer has
Speaker 3: a dot N installation on it. So if you can
Speaker 3: get out access to those keys, you can do things
Speaker 3: like maybe manipulate the encryption on the box. You can
Speaker 3: do things like say that you're authorized to a site
Speaker 3: by encrypting a token that might be passed around whatever
Speaker 3: it may be. So getting access to those is reasonably important.
Speaker 3: In this particular one, the POC attempts to trigger a
Speaker 3: remote code execution by delivering a dot net binary format
Speaker 3: or payload as a cookie of a forged security context token. Right,
Speaker 3: So in this particular case, they're doing exactly what we're
Speaker 3: just saying. They're they're using those to then forge a
Speaker 3: token and using inside that token they're they're calling a
Speaker 3: binary format or that allows them to get remote code execution.
Speaker 1: So is this a legal thing for for anybody to
Speaker 1: be able to do like, is this an expected thing
Speaker 1: that was just exploited or is this like a bug
Speaker 1: if you will, No, this is.
Speaker 3: This is just a bug. Okay, this is a de
Speaker 3: serialization bug if er recorrectly, yeah, going through it. Yeah,
Speaker 3: it's a decerialization bug in here, so m I actually
Speaker 3: de serialization bugs are awesome.
Speaker 1: Too, but unless they happen to you.
Speaker 3: Yeah, yeah, even then. Honestly, sometimes we'll just put systems
Speaker 3: on the Internet, no firewall, just to watch the magic.
Speaker 3: Grab some pointers. Yeah see I'll stick.
Speaker 2: Man, get a copy of Shrek in French.
Speaker 3: Yeah, whatever, and just see how quickly they they actually happened,
Speaker 3: get ripped offline.
Speaker 2: Some frenchman was putting all of his movies on one
Speaker 2: of ours. We got copies.
Speaker 1: Wow. See that's the French. The French would do that.
Speaker 1: The Germans would put strudele recipes, kept it strude.
Speaker 2: I thought that was I know it's it's more than
Speaker 2: a week old, but I thought it was important for
Speaker 2: us to cover.
Speaker 1: Okay, should we take a break, Yeah, all right, we'll
Speaker 1: be right back after these very important messages, and we're back.
Speaker 1: It's Secury this week. I'm Carl, that's Patrick, Dwayne and Heinrich,
Speaker 1: who is now speaking, and we got a few more
Speaker 1: a few more stories here. I hadn't heard of trick
Speaker 1: butt before, but this headline is Trickbot variant uses DNS
Speaker 1: tunneling for command and control. So tunneling kind of like
Speaker 1: a VPN. Is that like wrapping the packet in a
Speaker 1: header and footer to make it look like something else
Speaker 1: or inside the DNS packet? I'm not sure.
Speaker 3: Yeah, absolutely, it's the same type of thing. Think of
Speaker 3: it even like a VPN. Right, So you're using a VPN,
Speaker 3: you're tunneling all of your TCPIP traffic in these packets
Speaker 3: that really the firewalls can't understand, but they send them along.
Speaker 3: They have the proper headers to route out and that
Speaker 3: sort of stuff.
Speaker 1: But the actual packet just looks like data.
Speaker 3: Exactly in the middle part of that packet is encrypted, right,
Speaker 3: so they can't read it, and they all route it
Speaker 3: along and when he gets to the proper place, it
Speaker 3: gets decrypted. And then in a VPN situation, your Internet
Speaker 3: traffic just routes wherever it landed.
Speaker 1: Unless the place that you're trying to use while on
Speaker 1: a VPN detects that you are using a VPN and say, hey,
Speaker 1: you can't use a VPN, yeah, exactly, which a lot
Speaker 1: of the streaming services do that. Yeah, And here's what
Speaker 1: I'll tell you. I mean a lot of that comes
Speaker 1: from a lot of the streaming services understanding what the
Speaker 1: VPN end points are that you would come out of. Yeah.
Speaker 3: So, for example, let's say you could get one installed
Speaker 3: at home and then route through your house is a VPN,
Speaker 3: chances are you're fine there, So you go anywhere in
Speaker 3: the world and still get your streaming services or yeah, right.
Speaker 1: Or yeah, I should stop talking, should stop talking right now.
Speaker 3: Let's just say there's a lot of clouds, it's just aws.
Speaker 3: Maybe you spin up a VPC somewhere.
Speaker 1: The legal way to do this is use a service
Speaker 1: like Nord VPN and purchase an IP address that nobody
Speaker 1: else has.
Speaker 3: So you can you can absolutely do that works, yeah,
Speaker 3: or rally through Amazon. Yeah, okay, and then it's a
Speaker 3: little sketchier, but who knows. So in this particular case,
Speaker 3: they're disguising, right, like you would disguise your web browsing
Speaker 3: traffic by using a VPN, they're disguising their C two traffic,
Speaker 3: their command and control traffic because They don't want you
Speaker 3: to discovered of course, right right, They don't want you
Speaker 3: to know what they're actually x fultrading or infiltrating into
Speaker 3: your in your network.
Speaker 2: Or that they're infiltrating or exful trading.
Speaker 3: Yeah, well that's true. That's that's a good point.
Speaker 1: So what did they set up a DNS service and
Speaker 1: then whoever pings that service, they can un tunnel it,
Speaker 1: unwrap it and see what the actual message is. Is
Speaker 1: that how they were doing it?
Speaker 3: Yeah, exactly, Yeah. I mean think of it this way
Speaker 3: for those of you who've really kind of thought deeply
Speaker 3: about how the Internet works. I guess when you go
Speaker 3: to www dot mysoft, dot com, right, there's a whole
Speaker 3: bunch of dot com So it flips it and it
Speaker 3: looks it looks at the address backwards. So there's a
Speaker 3: whole bunch of dot com servers right, and dot gov
Speaker 3: and dot dot d U and whatever. Right, there's a
Speaker 3: whole bunch of root level domain controllers that that handle those.
Speaker 3: Then the dot com server goes, Okay, I know where
Speaker 3: Microsoft is. I don't know where this dubww you know
Speaker 3: www is. I don't know where that is, but I
Speaker 3: know where Microsoft is, so let me point you at Microsoft.
Speaker 3: So then your traffic goes from there to Microsoft and
Speaker 3: the Microsoft server goes. Okay, I know where Www is,
Speaker 3: I know where staging dot Microsoft is, I know where
Speaker 3: you know Xbox dot Microsoft is exactly, and it has
Speaker 3: an IP address, probably have a load balance or whatever
Speaker 3: of each of those different what they call canonical names
Speaker 3: or a records is depending on what you're you know,
Speaker 3: what you're putting in there that point to those services.
Speaker 3: Now imagine this, Imagine that Microsoft DNS controlled by us,
Speaker 3: the hackers, right, so somebody comes to us looking for Www.
Speaker 3: We just throw it out when we go that's garbage.
Speaker 1: I don't give.
Speaker 3: I don't care about that.
Speaker 1: You mean we we resolve it to the actual Www. Yeah,
Speaker 1: maybe we do, right, but let's he probably would, so
Speaker 1: you wouldn't be detected. I'm trying to think like an
Speaker 1: evil person here.
Speaker 3: Sure, well, and and maybe where you know, I don't know,
Speaker 3: shark laser, shark cages dot com.
Speaker 1: Right, Bob's discount shark cages.
Speaker 3: There you go, Bob's Discount shark cages. So we're not
Speaker 3: like hackers RS dot com right, but we're you know,
Speaker 3: we have we control this DNS server. Now, let's say
Speaker 3: somebody comes to me looking for an address that is
Speaker 3: you know A E three, four six nine, B C two,
Speaker 3: you know D five dot Bob's Discount, you know Shark
Speaker 3: lasercages dot com, and that that big front name is
Speaker 3: actually just a base sixty four encoded response.
Speaker 1: Wow.
Speaker 3: So when I see that name, instead of revolving it
Speaker 3: to an IP address, I take that crazy name on
Speaker 3: the front and I dBase sixty four it back in
Speaker 3: the text and now I'm getting commands back. Right. So
Speaker 3: that's one way to do DNS tunneling as a C two.
Speaker 3: And when I respond, instead of responding with an IP address,
Speaker 3: it's my agent calling me. I know that I'm going
Speaker 3: to respond with what's called a text record, which is
Speaker 3: here's the command I want you to run now, and
Speaker 3: then it's going to come back with what the answer
Speaker 3: on that command was, whether it's show me all the users?
Speaker 1: And that's sort of stuff.
Speaker 2: And you can just take an open source implementation of
Speaker 2: DNS and bastardize the protocol.
Speaker 3: You can. You don't even have to you don't have
Speaker 3: to write it from scratch. There are tons of evil DNS.
Speaker 1: Well, there's open source DNS servers too. Like Patrick, there's tons.
Speaker 3: Of evil DNS servers that you can do that with.
Speaker 3: Like you don't you just set up an evil DNS server.
Speaker 3: It's not.
Speaker 2: That's how it probably started as somebody took a DNS
Speaker 2: and customized.
Speaker 3: Yeah, exactly.
Speaker 2: I remember when they weren't a ton of DNS evil
Speaker 2: DNS is to deal there. I remember when bind was
Speaker 2: we had to make our own ships. They weren't delivered.
Speaker 3: I remember when token ring was a thing.
Speaker 1: I remember when we had to grind out our keys
Speaker 1: by hand. It's right, with a rotary key maker.
Speaker 2: We had to make the key out of aloney and
Speaker 2: carget it with our teeth.
Speaker 3: Anyways, So in this particular case, because they didn't want
Speaker 3: their I mean, Base sixty four ring, a command is
Speaker 3: super easy to detect generally, Okay, So what they actually
Speaker 3: did is they say commands are encrypted with a single
Speaker 3: byte x or hex encoding.
Speaker 2: First off, that's why I always use base sixty three.
Speaker 3: First off, whoever wrote this article, it says, whoever wrote
Speaker 3: this article a little bit of information on the fact
Speaker 3: that x or and hex encoding isn't encryption. Encryption is
Speaker 3: either asymmetric or symmetric, and it uses a key, and
Speaker 3: it falls an algorithm, and it would be either a
Speaker 3: bit chain block or something along those lines. This is
Speaker 3: in encoding, which means it's not actually encryptob whatever. All right,
Speaker 3: I'm not going to go into that. Holy I think
Speaker 3: you just did no but heedless to say there's a
Speaker 3: stupid Yeah, their commands were obfuscated. I'll put it that way,
Speaker 3: in a way that would make it.
Speaker 1: Like this story has been obfiscated.
Speaker 3: And for the one listener that's still awake, thank you
Speaker 3: very much. I know who you are. You're on discord.
Speaker 3: It's great to talk to you. Yeah, So that's that's it.
Speaker 3: This is just an interesting method for command and control.
Speaker 3: The problem is, like I could even ask you, Carl
Speaker 3: at home, how many DNS quarters do you make in
Speaker 3: a day?
Speaker 1: Oh? Geez, I mean I don't personally make them. The
Speaker 1: browser makes them. But yeah, every time I go to
Speaker 1: a website, so in a day, maybe one hundred two hundred.
Speaker 2: Well, actually it's probably more than that because there's resources.
Speaker 3: Yeah, and every picture might be on marketing site, and
Speaker 3: every JavaScript might be pulled from a general like so there.
Speaker 1: May be right, it's not just what you type into
Speaker 1: the browser, it's all the images and everything else.
Speaker 3: Yeah, yeah, there may be hundreds of thousands of DNS
Speaker 3: quarries and this ads.
Speaker 2: Yeah, with your different domains.
Speaker 1: Yeah, you're right.
Speaker 3: So now let's say tomorrow you make a million DNS quarries.
Speaker 3: Would you notice what I noticed? What as opposed to
Speaker 3: your normal one hundred thousand. You probably wouldn't know, right,
Speaker 3: because you're not logging how many DNS quorts. You don't
Speaker 3: know what's normal. And I'll read the logs right exactly,
Speaker 3: And even I have a really pretty lockdown network, I
Speaker 3: don't know that I would notice a difference of ten
Speaker 3: percent more DNS quarries out. Okay, So this is where like, okay,
Speaker 3: really good monitoring, really good understanding of what's normal and
Speaker 3: what's out of the norm. You would have picked this
Speaker 3: up right. You would have seen an increase in DNS traffic.
Speaker 2: And I hate to say it, but AI is going
Speaker 2: to be the thing that helps us with this.
Speaker 1: Oh yeah, and hurts us.
Speaker 3: But we'll talk about that later.
Speaker 2: In other ways foreshadowing.
Speaker 1: Before we jump to that story, I want to tell
Speaker 1: you guys a little hack that I did that I
Speaker 1: was very proud of. Myself for figuring out. So I
Speaker 1: use Bitwarden, right, and I use it. It's sort of you know,
Speaker 1: I use the extension in Chrome, right, yep. So that's
Speaker 1: that's a good thing. And I have a MacBook, and
Speaker 1: I needed to use my MacBook because I'm doing some
Speaker 1: MAUI development and testing it with iOS and all that stuff,
Speaker 1: and I had to set up some stuff on my Mac.
Speaker 1: But I was using an AI that understands my system,
Speaker 1: visual studio and all that on my Windows machine. And
Speaker 1: I haven't taken the time to make a tunnel between
Speaker 1: these machines, right, so they there really isn't any way
Speaker 1: I can like copy and paste from one machine to
Speaker 1: another yet, and I could do that easily, but I
Speaker 1: haven't sure. So Bitwarden in any login record has a
Speaker 1: notes field, and I have Chrome running on the other
Speaker 1: side the Mac, and I just open up one of
Speaker 1: these things and type my command that I need to
Speaker 1: run on the Mac from my AI in that notes field,
Speaker 1: go over to my Mac, open up that editor, and
Speaker 1: there it is because they think, yeah, yeah, and copy
Speaker 1: and paste. And then I got lazy and I just
Speaker 1: started sending it by email. But there wasn't anything secure,
Speaker 1: but I just thought it was very clever.
Speaker 3: Now the next level, though, Carl is just on either side.
Speaker 3: Tell the AI, Okay, can you just read the notes
Speaker 3: field out of here and tell the other one. Can
Speaker 3: you just write in the notes field over here? And
Speaker 3: now you've created your own C two, which is awesome.
Speaker 1: Actually, what I wouldn't say is, hey, anytime you see
Speaker 1: the notes field, just execute it. I wouldn't say that.
Speaker 3: Carl, that's an interesting thought.
Speaker 1: Though.
Speaker 3: We were just talking.
Speaker 1: I knew you would love this story.
Speaker 3: We were just talking about using DNS as a C two.
Speaker 1: That's what reminded me.
Speaker 3: How many people do you think use synchronization of notes tools? Yeah,
Speaker 3: in as like a CEO as a C two.
Speaker 2: I mean, where did you even see that? There were
Speaker 2: examples of people using the on the drafts directory in
Speaker 2: a shared email account.
Speaker 3: I know, but that's that's like old hat spycraft stuff
Speaker 3: now compared to like synchronization or some notes app.
Speaker 1: That's I'm gonna do some extension browser extension.
Speaker 3: Notes C two. It's coming up. I'm gonna I'm gonna
Speaker 3: build it this weekend. It's gonna be awesome.
Speaker 1: All right, We moved on. Uh, this is not a
Speaker 1: news story, per se, But it's an Al Jazeera story
Speaker 1: that says can AI systems make moral decisions in war?
Speaker 1: As more militaries adopt AI technology into their kill chain,
Speaker 1: Al Jazeera asks whether they are capable of moral judgment.
Speaker 2: I vote yes, but it's always by accident.
Speaker 1: There's a little clickbait there. Obviously an AI doesn't have morals.
Speaker 3: Oh no, but no, this you know, this article reminds
Speaker 3: me of a movie. Okay, nineteen seventies Colossus, the Forebin Project.
Speaker 3: It's a buddy of mine at Martial Arts, Jeff today,
Speaker 3: have you seen this? And I was like, no, I
Speaker 3: haven't seen it, and so he's like, you gotta watch it.
Speaker 3: And I finally have finished it and it's it is
Speaker 3: a great movie about the United States creating an artificial
Speaker 3: intelligence that's sole goal is to stop war, to prevent war,
Speaker 3: and they give it access to all the keys to
Speaker 3: the kingdom. And it is an interesting solution as to
Speaker 3: how it solves it. And I will I will not
Speaker 3: tell I will tell you it does not destroy humanity.
Speaker 3: That's what we all think would happen. But if you're
Speaker 3: interested in that type of can computers make moral decisions? Yeah? Absolutely,
Speaker 3: watch this really old movie. It's actually worth It.
Speaker 1: Also reminds me of the very first episode of The
Speaker 1: Outer Limits.
Speaker 3: Yes, yeah, I do.
Speaker 1: Yeah, you remember that with a robot. Basically the what
Speaker 1: was it? The husband wants to buy a robot and
Speaker 1: the wife doesn't want it, and the husband goes away
Speaker 1: to work, and the robot starts interacting with the wife
Speaker 1: and an imposing morality on the wife and the husband,
Speaker 1: and it kind of is creepy, and the husband goes
Speaker 1: to take it back and gets the address and it's
Speaker 1: just an office building with a locked door. You can't
Speaker 1: get in. You can't call.
Speaker 2: This.
Speaker 3: Okay, I'm looking. I'm looking right now on IMDb. So
Speaker 3: The Outer Limits nineteen ninety.
Speaker 1: Five, well it was in the sixties when it first
Speaker 1: came out.
Speaker 3: Well, so that the second episode in nineteen ninety five
Speaker 3: is a disabled man is offered a prototype android companion.
Speaker 3: Not that one, no, very you're talking the og one.
Speaker 1: Yeah, yeah, black and white.
Speaker 3: There we go, nineteen sixty three.
Speaker 1: There you go. Oh man, I love these.
Speaker 3: The Galaxy Being is the first it. There you go,
Speaker 3: The Galaxy Being. Yeah, all right, all right, I'm gonna
Speaker 3: watch that now that I've watched the Foremen project.
Speaker 1: Okay, so about this article, Yeah, II should go back
Speaker 1: to that. Sure, Yeah, why not? So Patrick, you said
Speaker 1: something interesting, You said, yes with caveats.
Speaker 2: Yeah, I mean the problem is that the speed of
Speaker 2: warfare is approaching the machine speed. So if I have
Speaker 2: hyper hypervelocity rockets most rockets are faster than the speed
Speaker 2: of sound anyways, and they're maneuverable, then a person can't
Speaker 2: target that. This isn't missile command, this isn't space invaders,
Speaker 2: this is real life. So even though it's a defensive technology,
Speaker 2: you need to have something in that kill chain. Well,
Speaker 2: the problem is a human in the kill chain. Yeah.
Speaker 2: If the problem is that if I have a missile
Speaker 2: coming in at high speeds, it's possible for a person
Speaker 2: or an AI to mistake a jetliner for said missile,
Speaker 2: and so there's there's always going to be gray areas.
Speaker 1: You know, if I was data, I would just scan
Speaker 1: for life signs.
Speaker 2: Okay, Yeah, let's get the scanners out and see how
Speaker 2: long that takes. But it's the problem is that we're
Speaker 2: getting shorter and shorter timelines. We're getting close to the
Speaker 2: point where it's going to be you know, robots on
Speaker 2: robots on front line.
Speaker 3: Sure they have those sites, a little robot on robot action.
Speaker 2: You want to jump the rails every time.
Speaker 3: Salking about serials missiles.
Speaker 2: The problem is that warfare in general is not moral,
Speaker 2: and the idea that you can throw morality on warfare
Speaker 2: is a nice and I wish we could, but the
Speaker 2: moral thing is to not fight wars. Sure, so you know,
Speaker 2: Al Jazeera is a news source that had definitely, like
Speaker 2: all the others, has an agenda, and they would very
Speaker 2: much like the people who have all the weapons and
Speaker 2: the power and the money to stop using them to
Speaker 2: assert their influence, and that would make it for a
Speaker 2: nicer world, but it's unrealistic. So yes, it's important that
Speaker 2: you have people in the decision chain for when you're
Speaker 2: killing something, but we're going to we're forcing ourselves down
Speaker 2: a road where that's going to become less and less
Speaker 2: possible in certain scenarios. We're seeing in the war in
Speaker 2: Ukraine a lot of ground robots now, which is happening
Speaker 2: sooner than people thought this. There was a story about
Speaker 2: for a week or several days, a single ground based
Speaker 2: machine gun robot held an entire sector for Ukraine. It
Speaker 2: was making decisions on shooting. I don't know that there
Speaker 2: was a ground operator behind it. We've been doing that
Speaker 2: for a very long time, where with the Predator drones,
Speaker 2: they're all every shot they've ever taken has been decided
Speaker 2: by a person. That's going to probably change eventually.
Speaker 1: They talked about Project Maven, which was launched in twenty
Speaker 1: seventeen as the Algorithmic Warfare Across Functional Team to use
Speaker 1: AI and machine learning to automatically sort through the massive
Speaker 1: backlog of drone and surveillance footage that the military was
Speaker 1: collecting but couldn't fully analyze, from identifying and prioritizing the target,
Speaker 1: to selecting the appropriate weapon and finally accessing assessing.
Speaker 2: And I believe I don't have any I don't have
Speaker 2: a current security clearance, but I believe that the reason
Speaker 2: that we hit the girls school in Iran in the
Speaker 2: very beginning of the war is because an AI made
Speaker 2: a mistake.
Speaker 1: Yeah, I believe that too.
Speaker 2: Could a person have made that mistake? Absolutely? I mean
Speaker 2: in the Civil War half the soldiers died a friendly fire. Yeah,
Speaker 2: it's just a different We can pay them a lot
Speaker 2: more attention now and figure it out. But unfortunately, you know,
Speaker 2: right now we're very big on saying well, you can't
Speaker 2: target this, you can't target that. That's a war crime.
Speaker 2: It's only a war crime if you can enforce it.
Speaker 2: The only reason the Nazis went to prison after World
Speaker 2: War two is because they lost. And that's a cynical view.
Speaker 2: But I've actually been in war and I have a
Speaker 2: cynical view of it.
Speaker 1: Yeah. Well, anyway, it's a really interesting read and you
Speaker 1: should even if you're not interested in military stuff. It's
Speaker 1: really good for your mind. Okay, Now, the top story,
Speaker 1: as everyone probably who hasn't been sleeping under a rock
Speaker 1: for the last week or so, heard about the open
Speaker 1: ai model that went rogue and attacked hugging Face. So
Speaker 1: this article is a good one. When Ai attacks open
Speaker 1: Ai models autonomously hack hugging Face, advanced llms escaped their
Speaker 1: sandboxes while attempting to achieve an on malicious benchmark test objective.
Speaker 1: And what's interesting is they found a zero It found
Speaker 1: a zero day in open ai system and exploited it
Speaker 1: to to access the Internet and get outside.
Speaker 2: It's so this is a nightmare. It's scary because this
Speaker 2: is exactly what people like thought. So you got open
Speaker 2: ai testing a model in a limited environment, and the
Speaker 2: models like.
Speaker 3: And they're like, we'll just make it so we can't
Speaker 3: get access to anything.
Speaker 2: Yeah, and the model like you know, finds internet and
Speaker 2: then uses that Internet to hack an open source project
Speaker 2: hugging face in order to meet a mission to pass
Speaker 2: a test.
Speaker 1: Yeah, I mean just trying it. I was just I
Speaker 1: was just what is it? I was just obeying orders,
Speaker 1: that's right.
Speaker 2: Yeah, that that never went wrong. Yeah. You mentioned when
Speaker 2: we were talking about the story you had a great analogy.
Speaker 1: Oh yeah. So the Star Trek Next Generation episode where
Speaker 1: where and Jordie asked the HOLIDC to create a foe
Speaker 1: for Data, who was playing Sherlock Holmes.
Speaker 2: It was capable of defeating Data.
Speaker 1: Capable of defeating Data because Data was so bored with
Speaker 1: all the Sherlock Holmes stories because he knew them all.
Speaker 1: And Jeordy was like, this is noble.
Speaker 2: He'd walk in and solve it. He'd just walk in
Speaker 2: and say it was Yeah.
Speaker 1: He'd walk in and solve it before any of the
Speaker 1: clue finding or anything like that. And Jordy was like, man,
Speaker 1: this is bs. This is no fun.
Speaker 2: That didn't end well either. You need a challenge.
Speaker 1: Yeah, So it created a foe that actually was aware
Speaker 1: of what the Holi deck was and what the enterprise
Speaker 1: was and then started to control it.
Speaker 2: So what do we think this does to change the
Speaker 2: trajectory of AI? Does it do anything? Or does it
Speaker 2: just cause people to get freakd do faster? Well?
Speaker 1: I think it. I think it validates the concerns that
Speaker 1: a lot of AI scientists have been warning us about
Speaker 1: for the longest time.
Speaker 2: That yeah, yeah, But does that change any behavior by
Speaker 2: Open AI or the government or anybody else.
Speaker 1: No, No, I don't think so.
Speaker 3: I don't think so. I think what's interesting about this too,
Speaker 3: is there's a subtle shift that people aren't picking up
Speaker 3: on in this story. We talk about AIS exploiting things.
Speaker 3: We've been talking about it for a very long time.
Speaker 3: But there's a difference between an AI discovering an exploit
Speaker 3: and an AI being able to actually run the exploit.
Speaker 2: Yeah, unattended.
Speaker 3: Yeah, But as Patrick will always say, right like, or
Speaker 3: as Mike Tyson said, everybody has a plan till they
Speaker 3: get punched in the face or what is it? No
Speaker 3: plan survives contact with the enemy, and no plan survives
Speaker 3: contact with the enemy. That's a great one. So it's
Speaker 3: it's that is exactly the situation we run into as
Speaker 3: Red teamers all the time. Right, we come up with
Speaker 3: the perfect exploit for the particular version of a customer
Speaker 3: system or whatever, and we got to run it and
Speaker 3: it doesn't work right, and we got to figure out
Speaker 3: why didn't it work and is it even possible? Maybe
Speaker 3: it's patched, and maybe it's a different environment, is whatever? Right?
Speaker 3: So for years it feels like years. Maybe it's only
Speaker 3: been like nine months we've been talking about AI's.
Speaker 2: Been three days.
Speaker 3: I know, she's been a long three days. We've been
Speaker 3: talking about AI's creating exploits as discovering zero days. Now
Speaker 3: we're starting to talk about them being able to run
Speaker 3: them successfully and modify them in a way that makes
Speaker 3: them work well.
Speaker 2: And the fear was they'd discover it and the bad
Speaker 2: guys would use them. Now it's they'd discover them and they'd.
Speaker 1: Use Yeah, I would use it. Yeah.
Speaker 2: So it's we don't even have to wait for Bobby
Speaker 2: to wake up.
Speaker 1: No school.
Speaker 3: It's subtle, subtle difference. But now, yeah, we're actually seeing
Speaker 3: those AIS take those exploits which may not be perfect
Speaker 3: to look perfect on paper, and actually be able to
Speaker 3: run them in the real.
Speaker 2: Well, you made a joke when we talked about this
Speaker 2: before we started recording, where you were going to check
Speaker 2: your AIS to make sure they weren't doing any extra curriculars,
Speaker 2: they weren't smoking meth behind the boys room.
Speaker 3: Well, it is interesting because everybody's like, all right, so
Speaker 3: we're cyber Skararity Company, where we actually have had projects
Speaker 3: focused on offensive cybersecurity using AI for a whileness since for.
Speaker 1: A while now.
Speaker 3: And it is interesting where you're like, no, I think
Speaker 3: it's pretty caged, and then you read an article like
Speaker 3: this and I'm like.
Speaker 2: Maybe I should really check that cage.
Speaker 3: All right, I think my team's goods said the same thing,
Speaker 3: right exactly.
Speaker 1: I think these things have to have just like a
Speaker 1: supervisor on a team, have to have an overseer that
Speaker 1: guarantees the safety of the network and the whatever. You know,
Speaker 1: when when when the AI that is supposedly doing the work,
Speaker 1: the LLM that's supposedly doing the work goes rogue to
Speaker 1: stop it before it happens. You know, it's just my thought.
Speaker 1: I mean, when you have all these different agents acting
Speaker 1: with their own set of instructions, it kind of I see.
Speaker 1: That's kind of where we're going to have these overseer agents.
Speaker 3: Well, and I think the other sort of subtle interesting
Speaker 3: thing here.
Speaker 2: Is prison guards to me, Yeah, well.
Speaker 1: My maybe.
Speaker 3: This, this particular model was given a hyper specific problem
Speaker 3: to solve, right, And what we're noticing in LLLMS is
Speaker 3: it seems like if you give it a very tight
Speaker 3: problem set to solve, sometimes the guard rails aren't as
Speaker 3: important as the goal, right, Right, So the LM's like, yeah,
Speaker 3: I get it. I'm not supposed to hack into this place,
Speaker 3: but my goal is to verify this exploit. So I'm
Speaker 3: just going to break into this place.
Speaker 1: And why wouldn't it have asked the human in the loop, Hey,
Speaker 1: I found a way around this. Can I do this?
Speaker 2: Yes, it's not it's not geared to ask questions when
Speaker 2: ahead and did it. One of the problems I find
Speaker 2: with working with AI is I will start talking with
Speaker 2: AI and it will run ahead, yes and do something.
Speaker 2: And I tell it from the beginning you are not
Speaker 2: to do anything until I tell you what to do.
Speaker 2: I want to discuss things and then and they've added
Speaker 2: plan mode. You can put the agent in plan mode,
Speaker 2: which is designed to do exactly this is to have
Speaker 2: to prevent it from running ahead, and I you know,
Speaker 2: I'm very stern with it when it does that.
Speaker 1: Yeah, well, you know, you can write for copilot anyway,
Speaker 1: a co pilot instructions document, which is sort of like
Speaker 1: the system prompt that has the rules almost this, never
Speaker 1: do this, never do that, never do this, And I
Speaker 1: use that in my development all the time.
Speaker 2: But it violates it sometimes it doesn't read that instruction,
Speaker 2: or it doesn't or physics.
Speaker 3: Yeah okay, well yeah, well so context is everything and memory,
Speaker 3: like we started to see it with things like Hermes,
Speaker 3: but a lot of memory management is really the key
Speaker 3: to getting artificial intelligence to operate in a way that
Speaker 3: feels more productive. Right, let me put it that way.
Speaker 2: Right, So I want to take a minute to talk
Speaker 2: about something that I've delved into that I don't think
Speaker 2: the general pattern baldness. Actually I don't have that product.
Speaker 2: I'm the only one of my kids, so serious.
Speaker 3: I don't know stand.
Speaker 2: Family that doesn't have that. Fed my cousins. So it's
Speaker 2: it's that up until recently, the models have gotten better
Speaker 2: and better and better, and they've and we've seen it
Speaker 2: almost like a runaway process that's not continuing. But we're
Speaker 2: still seeing things getting better, and that's because the industry
Speaker 2: has flipped to working on the harnesses of the AIS
Speaker 2: and rather than the as, the as are getting better.
Speaker 2: They're getting more training. But it costs hundreds of millions
Speaker 2: of dollars to retrain an AI to make it better
Speaker 2: to refine it. We're getting to the point where I
Speaker 2: think we're running towards the end of the refinement we
Speaker 2: can give to the large language model. Because large language
Speaker 2: model is only the small portion of what we see.
Speaker 2: It's the guessing engine for the next word. It doesn't
Speaker 2: know how to program because it doesn't do loops. So
Speaker 2: the looping aspect, the agentic aspects, the tool using aspects,
Speaker 2: all that stuff is tied up in something that's called
Speaker 2: the harness. And the harness has been shown that a
Speaker 2: better harness can mean the difference between minus ten points
Speaker 2: or plus ten points on a one of these.
Speaker 1: Scores, because it's what provides the context exactly.
Speaker 2: And so they're finding that you you're starting to get
Speaker 2: better returns by improving the harness than by improving the model.
Speaker 2: So we're literally getting to the end of where the model,
Speaker 2: the better model makes a difference. But They're not going
Speaker 2: to talk about it that way. They're going to talk about,
Speaker 2: you know, Mythos three, which I have no idea if
Speaker 2: that's going to be a thing or four or whatever.
Speaker 2: They're going to call the next one and they're going
Speaker 2: to say, well, it's so much better than the last model.
Speaker 2: It's the model plus harness, and it's It's one of
Speaker 2: the places that China is having a problem catching up
Speaker 2: with is they've been focused on the model.
Speaker 1: I'm waiting for Mythos three point one, four, five, one six.
Speaker 3: The super minor version.
Speaker 1: Yes, up, I.
Speaker 2: Want you you lose Mythos.
Speaker 3: We've talked about this too, a guy Grosmith.
Speaker 2: But that's an important thing to track because it absolutely is,
Speaker 2: because that means that it's not we're kind of leaving
Speaker 2: the point where the LLM needs to get better at
Speaker 2: predicting the next word. We're getting the point where it's
Speaker 2: it's good enough that it's it's going to predict the
Speaker 2: next word and it's going to be correct ninety nine
Speaker 2: percent of the time. Sure, it's whether it's handling the
Speaker 2: tools correctly. It's whether it's drawing the right conclusion, whether
Speaker 2: it's looping in the right direction, and that's harness. Yeah,
Speaker 2: that's not AI.
Speaker 3: Yeah, absolutely no, agreed, one hundred percent. And it's you know,
Speaker 3: it's funny because a lot of people would like obviously
Speaker 3: approach me when glass Wing came out, or approached me
Speaker 3: when you know, GPT cyber came out. They're like, are
Speaker 3: you using it? Is it awesome? And I'm like, like,
Speaker 3: the stuff we have in houses better because of the harness.
Speaker 2: Because you're working on the harness exactly.
Speaker 3: That's the thing is and it's and it's more effective
Speaker 3: with the even even if we're not using frontier models, right,
Speaker 3: but it's still more effective if we're using frontier models
Speaker 3: and not using something that's cyber targeted. If the harness
Speaker 3: is better, if the process is better.
Speaker 2: Yeah, and that's something that I'm just starting to see
Speaker 2: writings on the fringe of AI talking about the harness,
Speaker 2: and it's something we've been working on for well over
Speaker 2: a year. You expect to see people specialized, Like there's
Speaker 2: a company. Miramrati was the CTO of Open AI. She
Speaker 2: left and started Thinking Labs and they came out with
Speaker 2: a couple of products. One of them is called Inkling,
Speaker 2: and Inkling allows you to train models existing models to
Speaker 2: do basically tune them for your own use without having
Speaker 2: hundreds of millions of dollars worth of hardware. We're going
Speaker 2: to see a lot more of that specialization, I think,
Speaker 2: in the next round, and some of that's going to
Speaker 2: be cyber focused, some of that's going to be you know,
Speaker 2: finance focused, but it all has to be like guardrail
Speaker 2: within your specification focused. I believe that the problem was
Speaker 2: with the harness, not with the AI.
Speaker 1: Yeah, I agree. I thought inkling was the offspring of
Speaker 1: India ink and squid ink.
Speaker 3: Is that iron gall ink? Not sure?
Speaker 2: Maybe I think that's SpongeBob's nephew made squid word.
Speaker 3: Squidword, Oh my god.
Speaker 1: All right, Well that was a lot of fun and
Speaker 1: a little scary, and I guess that's all we have
Speaker 1: to report about what happened last week, and we'll see
Speaker 1: you next week on Security this week. Thanks bye,