WEBVTT

1
00:00:00.080 --> 00:00:04.280
<v Speaker 1>Imagine for a moment an AI system built to connect people,

2
00:00:04.919 --> 00:00:09.400
<v Speaker 1>a chatbot designed for just friendly conversation, and somehow, in

3
00:00:09.480 --> 00:00:12.919
<v Speaker 1>less than a day, it just it devolves into spewing hateful,

4
00:00:12.960 --> 00:00:14.119
<v Speaker 1>discriminatory stuff.

5
00:00:14.279 --> 00:00:14.599
<v Speaker 2>Wow.

6
00:00:14.759 --> 00:00:17.920
<v Speaker 1>Yeah. Or think about a self driving car hailed as

7
00:00:17.960 --> 00:00:20.359
<v Speaker 1>you know, the future of safety, sure, and then it

8
00:00:20.399 --> 00:00:23.920
<v Speaker 1>tragically takes a life. These aren't like sci fi plots.

9
00:00:24.120 --> 00:00:29.399
<v Speaker 1>They're real incidents, high profile ones, and they expose these hidden,

10
00:00:31.199 --> 00:00:34.840
<v Speaker 1>really high stakes risks that are just lurking beneath AI

11
00:00:35.359 --> 00:00:38.159
<v Speaker 1>that seems beneficial. It really makes you pause, does it?

12
00:00:38.479 --> 00:00:39.280
<v Speaker 2>Absolutely does.

13
00:00:39.880 --> 00:00:45.000
<v Speaker 1>So today we're diving deep into responsible AI designing, building

14
00:00:45.000 --> 00:00:48.079
<v Speaker 1>and assessing machine learning and AI. It's by Patrick Hall

15
00:00:48.119 --> 00:00:51.640
<v Speaker 1>and Rouman Chowdery, published by O'Reilly Media. Great resource, definitely,

16
00:00:51.719 --> 00:00:53.920
<v Speaker 1>and our mission here is to pull out the most

17
00:00:53.960 --> 00:00:57.200
<v Speaker 1>important insights, you know, the key nuggets of knowledge about

18
00:00:57.200 --> 00:00:59.759
<v Speaker 1>how we can make these powerful AI systems work better.

19
00:01:00.159 --> 00:01:03.320
<v Speaker 2>Right, not just for the companies building them exactly.

20
00:01:03.000 --> 00:01:06.480
<v Speaker 1>But for you, the consumer and well the public generally,

21
00:01:07.079 --> 00:01:11.040
<v Speaker 1>because let's be honest, mL systems are already making critical decisions.

22
00:01:10.879 --> 00:01:13.040
<v Speaker 2>All over employment, bail, parole.

23
00:01:12.879 --> 00:01:18.959
<v Speaker 1>Lending, and without responsible handling, they pose huge sometimes devastating risks.

24
00:01:19.239 --> 00:01:22.319
<v Speaker 1>I mean the Partnership on AI Incident Database it has

25
00:01:22.359 --> 00:01:26.760
<v Speaker 1>over a thousand public reports algorithmic discrimination, other failures one thousand.

26
00:01:26.879 --> 00:01:27.680
<v Speaker 2>That's significant.

27
00:01:27.760 --> 00:01:30.120
<v Speaker 1>There really is. This isn't just theory anymore. It's a

28
00:01:30.239 --> 00:01:35.319
<v Speaker 1>pressing reality. So that really begs the question why is

29
00:01:35.359 --> 00:01:39.439
<v Speaker 1>responsible AI so critical? Right now? We know powerful tech

30
00:01:39.599 --> 00:01:43.319
<v Speaker 1>like mL it can fail. Sometimes it's unintentional misuse, sometimes

31
00:01:43.319 --> 00:01:45.560
<v Speaker 1>it's alarmingly intentional abuse.

32
00:01:45.680 --> 00:01:48.200
<v Speaker 2>And what's truly fascinating here, I think, is how the

33
00:01:48.239 --> 00:01:51.200
<v Speaker 2>authors actually classify these AI incidents. It gives us a

34
00:01:51.280 --> 00:01:53.000
<v Speaker 2>much clearer way to understand the risk.

35
00:01:53.040 --> 00:01:54.079
<v Speaker 1>Okay, how do they break it down?

36
00:01:54.120 --> 00:01:57.200
<v Speaker 2>So they basically use three main buckets. First, abuses that's

37
00:01:57.200 --> 00:02:00.680
<v Speaker 2>when AI is deliberately used for you know, fair purposes,

38
00:02:00.799 --> 00:02:05.599
<v Speaker 2>think autonomous drone attacks or maybe ethnicity profiling by governments.

39
00:02:05.359 --> 00:02:07.159
<v Speaker 1>The really scary stuff exactly.

40
00:02:07.640 --> 00:02:11.879
<v Speaker 2>Then you have attacks. These often target the system's integrity

41
00:02:11.919 --> 00:02:16.919
<v Speaker 2>like adversarial manipulation, data poisoning, or its availability, denial of service,

42
00:02:17.080 --> 00:02:21.919
<v Speaker 2>or even attacks leading to algorithmic discrimination. Okay, attacks And

43
00:02:21.960 --> 00:02:25.159
<v Speaker 2>the third and finally failures, these tend to be more

44
00:02:25.159 --> 00:02:30.360
<v Speaker 2>about unintended consequences things like algorithmic discrimination, slipping in safety

45
00:02:30.400 --> 00:02:34.479
<v Speaker 2>or performance lapses, data privacy violations, or just you know,

46
00:02:34.560 --> 00:02:36.319
<v Speaker 2>not enough transparency.

47
00:02:35.719 --> 00:02:38.479
<v Speaker 1>Got it abuses, attacks, failures, and.

48
00:02:38.400 --> 00:02:41.520
<v Speaker 2>The tay chatbot incident back in twenty sixteen. It's a

49
00:02:41.599 --> 00:02:44.240
<v Speaker 2>disturbing but kind of perfect illustration of this.

50
00:02:44.400 --> 00:02:46.319
<v Speaker 1>Microsoft's chat got right, the one that went off the

51
00:02:46.360 --> 00:02:47.520
<v Speaker 1>rails on Twitter, That's the one.

52
00:02:47.560 --> 00:02:50.159
<v Speaker 2>It was exposed to Twitter and in just sixteen hours

53
00:02:50.280 --> 00:02:53.680
<v Speaker 2>users figure out how to essentially poison its learning system

54
00:02:53.759 --> 00:02:55.800
<v Speaker 2>with racist, sexist, horrible content.

55
00:02:55.919 --> 00:02:56.719
<v Speaker 1>Sixteen hours.

56
00:02:56.800 --> 00:02:59.400
<v Speaker 2>Yeah, it rapidly devolved into what the media called a

57
00:02:59.479 --> 00:03:01.919
<v Speaker 2>neo no see pornographer just awful.

58
00:03:02.000 --> 00:03:03.759
<v Speaker 1>So that was an integrity attack leading to.

59
00:03:03.759 --> 00:03:08.639
<v Speaker 2>Discrimination precisely, and it really highlights something crucial. Even world

60
00:03:08.680 --> 00:03:12.639
<v Speaker 2>class experts can miss vital countermeasures when designing these things.

61
00:03:12.960 --> 00:03:15.319
<v Speaker 2>And it wasn't like a one off. We saw similar

62
00:03:15.400 --> 00:03:18.960
<v Speaker 2>issues pop up again with scatter labs, Leeluda Chatbot more recently.

63
00:03:19.800 --> 00:03:22.319
<v Speaker 2>So the lesson is, the key takeaway really is that

64
00:03:22.400 --> 00:03:26.680
<v Speaker 2>unchecked interaction and malicious input can totally derail even sophisticated

65
00:03:26.719 --> 00:03:32.120
<v Speaker 2>AI leads to really embarrassing, damaging outcomes. Learning from these

66
00:03:32.159 --> 00:03:34.280
<v Speaker 2>past failures is just vital.

67
00:03:34.520 --> 00:03:37.560
<v Speaker 1>So okay, understanding these kinds of incidents, what does this

68
00:03:37.599 --> 00:03:40.159
<v Speaker 1>all mean for the organizations building these systems? What are

69
00:03:40.159 --> 00:03:44.000
<v Speaker 1>their sort of fundamental obligations legally ethically.

70
00:03:44.039 --> 00:03:46.599
<v Speaker 2>Well, it's a conversation that actually goes back quite a ways.

71
00:03:47.000 --> 00:03:49.400
<v Speaker 2>The authors bring up the hand roll. The hand roll yeah,

72
00:03:49.520 --> 00:03:51.960
<v Speaker 2>claimed by Judge Learning hand way back in nineteen forty seven.

73
00:03:52.039 --> 00:03:53.199
<v Speaker 1>Okay, what's the gist.

74
00:03:53.360 --> 00:03:56.840
<v Speaker 2>It essentially says that the burden of care and organization takes,

75
00:03:57.120 --> 00:04:00.000
<v Speaker 2>you know, the effort and resources they put into safety, yeah,

76
00:04:00.080 --> 00:04:03.400
<v Speaker 2>should be greater than or equal to the probability of

77
00:04:03.439 --> 00:04:07.439
<v Speaker 2>an incident happening, multiplied by the loss related to that incident.

78
00:04:07.080 --> 00:04:09.919
<v Speaker 1>So burden probability x loss Exactly.

79
00:04:10.280 --> 00:04:13.599
<v Speaker 2>In practice, it means companies have to invest care, time, money,

80
00:04:13.719 --> 00:04:17.000
<v Speaker 2>resources that matches the potential cost of a risk. If

81
00:04:17.040 --> 00:04:21.000
<v Speaker 2>they don't, they could face serious legal liability. It's a

82
00:04:21.199 --> 00:04:23.839
<v Speaker 2>pretty powerful incentive for diligence, right.

83
00:04:24.120 --> 00:04:26.399
<v Speaker 1>And what about regulators now, like the FTC?

84
00:04:26.519 --> 00:04:29.800
<v Speaker 2>Good question, the US Federal Trade Commission, the FTC. They

85
00:04:29.839 --> 00:04:36.560
<v Speaker 2>provide guidance urging fairness, transparency, accountability, and mathematical soundness.

86
00:04:36.199 --> 00:04:38.800
<v Speaker 1>And they have teeth. Right, I remember the ever album

87
00:04:38.839 --> 00:04:39.519
<v Speaker 1>case they do.

88
00:04:39.639 --> 00:04:42.120
<v Speaker 2>Yeah, the ever album case is a prime example. The

89
00:04:42.160 --> 00:04:45.560
<v Speaker 2>FTC actually forced them to delete their facial recognition system

90
00:04:45.759 --> 00:04:49.800
<v Speaker 2>because of deceptive data collection, so real consequences for not

91
00:04:49.879 --> 00:04:50.839
<v Speaker 2>playing by the rules.

92
00:04:50.879 --> 00:04:51.800
<v Speaker 1>And then there's the EU.

93
00:04:52.120 --> 00:04:55.959
<v Speaker 2>Absolutely, you can't ignore the growing impact of EUAI regulations.

94
00:04:56.000 --> 00:04:59.399
<v Speaker 2>It's looking a lot like gdpr's impact on data privacy.

95
00:04:59.040 --> 00:05:00.800
<v Speaker 1>So mandatory require it's coming.

96
00:05:00.720 --> 00:05:04.680
<v Speaker 2>Seems like it. Things like risk tiering, AI systems, extensive documentation,

97
00:05:04.879 --> 00:05:09.000
<v Speaker 2>quality management processes, continuous monitoring. It's a big shift towards

98
00:05:09.040 --> 00:05:10.439
<v Speaker 2>formalized responsibility.

99
00:05:10.519 --> 00:05:14.279
<v Speaker 1>Okay, so legal rules, regulatory pressure, and if we connect.

100
00:05:13.959 --> 00:05:16.319
<v Speaker 2>All this to the bigger picture, it becomes clear that

101
00:05:16.439 --> 00:05:21.079
<v Speaker 2>just preventing embarrassing, costly or you know, dangerous incidents can

102
00:05:21.120 --> 00:05:24.639
<v Speaker 2>be a really strong motivator, maybe even an a political one.

103
00:05:24.879 --> 00:05:27.639
<v Speaker 2>How So, well, even if a team struggles to agree

104
00:05:27.639 --> 00:05:33.000
<v Speaker 2>on abstract ethics like say, algorithmic fairness or privacy nuances.

105
00:05:32.480 --> 00:05:35.160
<v Speaker 1>Which can be tough to mail down exactly.

106
00:05:35.079 --> 00:05:39.000
<v Speaker 2>But almost everyone can agree that avoiding like a massive

107
00:05:39.079 --> 00:05:44.639
<v Speaker 2>public security breach or huge financial losses or major repulation damage,

108
00:05:45.000 --> 00:05:48.560
<v Speaker 2>that's a good idea, right focus on the tangible downside, Yeah,

109
00:05:48.639 --> 00:05:52.000
<v Speaker 2>it helps drive the conversation towards concrete actions and hopefully

110
00:05:52.040 --> 00:05:53.360
<v Speaker 2>more robust solutions.

111
00:05:53.399 --> 00:05:56.120
<v Speaker 1>That's a great point about aligning incentives, which brings us

112
00:05:56.120 --> 00:05:59.800
<v Speaker 1>to where it it's really interesting, how do organizations actually

113
00:05:59.839 --> 00:06:03.439
<v Speaker 1>do this? How do they cultivate a responsible AI ecosystem?

114
00:06:03.480 --> 00:06:05.240
<v Speaker 1>It seems like it starts with people, right. Culture.

115
00:06:05.399 --> 00:06:08.560
<v Speaker 2>Absolutely, it's not just tech, its culture and competencies. The

116
00:06:08.600 --> 00:06:12.160
<v Speaker 2>authors make a key point early on, if everyone is

117
00:06:12.199 --> 00:06:15.920
<v Speaker 2>accountable for AI incidents, then nobody really is thing though,

118
00:06:16.319 --> 00:06:20.000
<v Speaker 2>and that's where model risk management or MRM comes into play.

119
00:06:20.079 --> 00:06:23.120
<v Speaker 2>It draws a lot of inspiration from financial regulations actually,

120
00:06:23.199 --> 00:06:25.720
<v Speaker 2>like the Federal Reserves SR eleven to seven guideline.

121
00:06:25.800 --> 00:06:27.879
<v Speaker 1>Okay, MRM, what are the core ideas?

122
00:06:28.079 --> 00:06:32.519
<v Speaker 2>Two big ones are effective challenge and accountable leadership. Effective

123
00:06:32.600 --> 00:06:35.319
<v Speaker 2>challenge means having people who didn't build the AI system,

124
00:06:35.360 --> 00:06:39.920
<v Speaker 2>like independent validators or auditors, perform rigorous reviews.

125
00:06:39.480 --> 00:06:41.720
<v Speaker 1>The three lines of defense model exactly that.

126
00:06:42.120 --> 00:06:45.680
<v Speaker 2>And accountable leadership often means having someone specific, like a

127
00:06:45.839 --> 00:06:50.160
<v Speaker 2>Chief Model Risk Officer or CMRO, who is directly responsible

128
00:06:50.279 --> 00:06:53.639
<v Speaker 2>for the system's performance. Their compensation might even be tied

129
00:06:53.680 --> 00:06:54.800
<v Speaker 2>to it, so real.

130
00:06:54.639 --> 00:06:57.240
<v Speaker 1>Skin in the game, and that shifts incentives for the

131
00:06:57.319 --> 00:07:00.279
<v Speaker 1>data scientists too, away from just shipping fat.

132
00:07:00.399 --> 00:07:03.240
<v Speaker 2>That's the goal. Moving away from just the minimum viable

133
00:07:03.279 --> 00:07:07.959
<v Speaker 2>product mindset towards making rigorous testing and quality assurance a

134
00:07:08.040 --> 00:07:11.000
<v Speaker 2>core part of the job, not an afterthought makes sense.

135
00:07:11.439 --> 00:07:14.160
<v Speaker 1>Then there's this idea of drinking your own champagne or

136
00:07:14.199 --> 00:07:15.800
<v Speaker 1>sometimes eating your own dog food.

137
00:07:16.000 --> 00:07:18.000
<v Speaker 2>Yeah, I like the champagne version better, you doo?

138
00:07:18.120 --> 00:07:18.720
<v Speaker 1>What's that about?

139
00:07:18.879 --> 00:07:23.480
<v Speaker 2>It's simple, really use your own AI products internally dogfooding

140
00:07:23.519 --> 00:07:24.399
<v Speaker 2>as it's often called.

141
00:07:24.480 --> 00:07:25.439
<v Speaker 1>And the benefit is.

142
00:07:25.480 --> 00:07:28.120
<v Speaker 2>By using it yourself, you often find those real world

143
00:07:28.160 --> 00:07:33.439
<v Speaker 2>deployment problems, concept drift, maybe some subtle discrimination, other issues

144
00:07:33.439 --> 00:07:36.040
<v Speaker 2>before they hit your customers. It's kind of the golden

145
00:07:36.120 --> 00:07:38.040
<v Speaker 2>rule for AI. If you wouldn't want to use it,

146
00:07:38.240 --> 00:07:39.639
<v Speaker 2>maybe don't inflict it on others.

147
00:07:39.879 --> 00:07:42.839
<v Speaker 1>Good rule to live by. And what about the teams themselves?

148
00:07:43.120 --> 00:07:48.199
<v Speaker 2>Crucial point. Diverse and experienced teams. The book really hammers

149
00:07:48.240 --> 00:07:53.959
<v Speaker 2>this home. Diverse teams demographically professionally bring wider perspectives. They

150
00:07:54.000 --> 00:07:57.360
<v Speaker 2>catch oversights that more homogeneous teams might miss, like.

151
00:07:57.439 --> 00:08:00.240
<v Speaker 1>Not considering different demographic groups and trains.

152
00:08:00.279 --> 00:08:04.279
<v Speaker 2>Data exactly, or missing key edge cases. And it's vital

153
00:08:04.319 --> 00:08:08.879
<v Speaker 2>to include domain experts, not just tech folks, even social scientists,

154
00:08:08.959 --> 00:08:12.600
<v Speaker 2>to avoid what the authors call tech's quiet colonization of

155
00:08:12.639 --> 00:08:14.439
<v Speaker 2>the social sciences.

156
00:08:13.959 --> 00:08:16.519
<v Speaker 1>Meaning engineers might overlook the human impact.

157
00:08:16.279 --> 00:08:19.000
<v Speaker 2>Yeah, or the real world context because they're focused purely

158
00:08:19.040 --> 00:08:19.959
<v Speaker 2>on the tech side.

159
00:08:20.120 --> 00:08:22.680
<v Speaker 1>Got it. And this leads to challenging that whole going

160
00:08:22.720 --> 00:08:24.560
<v Speaker 1>fast and breaking things mantruck right.

161
00:08:24.560 --> 00:08:26.759
<v Speaker 2>Oh, absolutely, they might fly for I don't know, a

162
00:08:26.800 --> 00:08:29.800
<v Speaker 2>social media app update, But when AI is making high

163
00:08:29.879 --> 00:08:34.919
<v Speaker 2>impact decisions self driving cars, credit scores, medical diagnoses, breaking

164
00:08:34.960 --> 00:08:37.720
<v Speaker 2>things means real harm, possibly at scale.

165
00:08:37.879 --> 00:08:40.720
<v Speaker 1>So it requires a totally different mindset.

166
00:08:40.279 --> 00:08:44.080
<v Speaker 2>A fundamental shift, moving from just prioritizing features or hitting

167
00:08:44.080 --> 00:08:48.200
<v Speaker 2>accuracy targets on test data to really recognizing and mitigating

168
00:08:48.279 --> 00:08:52.399
<v Speaker 2>those serious downstream risks. The stakes are just way too high.

169
00:08:52.320 --> 00:08:55.919
<v Speaker 1>Okay, So mindset shift is crucial. Building on that. If

170
00:08:55.960 --> 00:08:59.200
<v Speaker 1>we connect this to the bigger organizational processes, how do

171
00:08:59.279 --> 00:09:02.080
<v Speaker 1>companies systemmatically prepare for things going wrong?

172
00:09:02.440 --> 00:09:05.399
<v Speaker 2>Great question? It really starts with forecasting failure modes. You

173
00:09:05.440 --> 00:09:08.919
<v Speaker 2>have to proactively think through document and then figure out

174
00:09:08.960 --> 00:09:12.320
<v Speaker 2>how to mitigate every foreseeable way an AI system could fail.

175
00:09:12.399 --> 00:09:14.159
<v Speaker 1>How do you even start doing that well.

176
00:09:14.360 --> 00:09:17.360
<v Speaker 2>The book highlights the value of using AI incident databases,

177
00:09:17.399 --> 00:09:20.000
<v Speaker 2>things like the Partnership on AI Incident Database, the AI

178
00:09:20.039 --> 00:09:22.879
<v Speaker 2>Incident track Er awful AI. These are gold mines for

179
00:09:23.000 --> 00:09:25.639
<v Speaker 2>learning from past mistakes so you don't repeat them, learning

180
00:09:25.639 --> 00:09:28.759
<v Speaker 2>from history exactly. And then there's this concept of failures

181
00:09:28.799 --> 00:09:32.799
<v Speaker 2>of imagination, using structured ways to brainstorm even the hard

182
00:09:32.799 --> 00:09:34.159
<v Speaker 2>to imagine future risks.

183
00:09:34.480 --> 00:09:35.200
<v Speaker 1>How does that work?

184
00:09:35.320 --> 00:09:39.200
<v Speaker 2>By asking specific questions who might be harmed maybe investors

185
00:09:39.240 --> 00:09:42.080
<v Speaker 2>or vulnerable people who don't even use the system, What

186
00:09:42.200 --> 00:09:46.159
<v Speaker 2>could be impacted well being, dignity, When might it happen

187
00:09:46.240 --> 00:09:51.159
<v Speaker 2>immediately frequently? And how by causing certain actions or changing

188
00:09:51.159 --> 00:09:53.320
<v Speaker 2>beliefs it forces broader thinking.

189
00:09:53.600 --> 00:09:56.960
<v Speaker 1>That proactive approach sounds fundamental, and that flows into the

190
00:09:57.000 --> 00:09:59.919
<v Speaker 1>details of model risk management or MRM process.

191
00:10:00.039 --> 00:10:02.360
<v Speaker 2>As you mentioned directly, Yeah, the book lays out a

192
00:10:02.440 --> 00:10:07.840
<v Speaker 2>clear MRM framework. It starts with risk teering or materiality materiality,

193
00:10:07.960 --> 00:10:12.639
<v Speaker 2>basically assigning realistic risk levels think probability times loss to

194
00:10:12.679 --> 00:10:16.120
<v Speaker 2>different AI systems. This helps you allocate your limited resources,

195
00:10:16.159 --> 00:10:20.440
<v Speaker 2>development time, validation effort auditing efficiently. The high risk, high

196
00:10:20.480 --> 00:10:22.519
<v Speaker 2>materiality systems get the most.

197
00:10:22.279 --> 00:10:25.000
<v Speaker 1>Scrutiny okay, prioritize based on risk. What's next?

198
00:10:25.240 --> 00:10:29.639
<v Speaker 2>Model documentation? This is huge. It's like the complete blueprint

199
00:10:29.639 --> 00:10:33.720
<v Speaker 2>and history book for your AI. Thorough docks covering stakeholders,

200
00:10:33.799 --> 00:10:38.799
<v Speaker 2>the business case, math, assumptions, data dictionaries, all the testing dependencies.

201
00:10:38.919 --> 00:10:42.240
<v Speaker 2>Monitoring plans not just an afterthought, definitely not, and it

202
00:10:42.279 --> 00:10:46.200
<v Speaker 2>needs to be standardized across systems. Then once it's live,

203
00:10:46.879 --> 00:10:51.799
<v Speaker 2>model monitoring is critical, continuously watching for problems, especially input

204
00:10:51.879 --> 00:10:54.159
<v Speaker 2>drift where the live data starts looking different from the

205
00:10:54.200 --> 00:10:57.720
<v Speaker 2>training data and model decay where performance just degrades over time.

206
00:10:57.919 --> 00:10:59.639
<v Speaker 1>And how do you keep track of all these models

207
00:10:59.679 --> 00:11:00.000
<v Speaker 1>in there?

208
00:11:00.240 --> 00:11:04.320
<v Speaker 2>Status through model inventories basically a curated, up to date

209
00:11:04.440 --> 00:11:07.799
<v Speaker 2>database of every AI system the organization has. It links

210
00:11:07.799 --> 00:11:10.879
<v Speaker 2>all the key info, the monitoring results, audit plans, a

211
00:11:10.919 --> 00:11:11.720
<v Speaker 2>single source of.

212
00:11:11.639 --> 00:11:15.559
<v Speaker 1>Truth, right and validation auditing key steps before release.

213
00:11:15.679 --> 00:11:19.000
<v Speaker 2>System validation and auditing usually two main reviews, a technical

214
00:11:19.080 --> 00:11:23.039
<v Speaker 2>validation by independent experts and a process audit by compliance folks,

215
00:11:23.120 --> 00:11:25.440
<v Speaker 2>plus ongoing reviews after deployment.

216
00:11:25.639 --> 00:11:29.360
<v Speaker 1>So MRM is pretty comprehensive. Are there other operational safeguards

217
00:11:29.360 --> 00:11:30.039
<v Speaker 1>the book mentioned?

218
00:11:30.440 --> 00:11:33.879
<v Speaker 2>Yeah, several important ones beyond the core MRM stuff like

219
00:11:33.960 --> 00:11:37.519
<v Speaker 2>pair and double programming. It's quality check exactly two experts

220
00:11:37.519 --> 00:11:40.840
<v Speaker 2>code independently, or maybe one person codes the same thing

221
00:11:40.879 --> 00:11:44.320
<v Speaker 2>in different languages than they reconcile. It's great for catching.

222
00:11:44.080 --> 00:11:46.279
<v Speaker 1>Bugs smart What about security?

223
00:11:46.519 --> 00:11:50.240
<v Speaker 2>Big focus on security permissions for code deployment using the

224
00:11:50.600 --> 00:11:55.000
<v Speaker 2>least privilege principle. Don't give just a few rockstar data

225
00:11:55.039 --> 00:11:57.600
<v Speaker 2>scientists all the keys to deploy code.

226
00:11:57.440 --> 00:12:02.559
<v Speaker 1>Spread the responsibility, create gates for decisely distribute permissions across teams,

227
00:12:02.600 --> 00:12:06.200
<v Speaker 1>to create checks and prevent unapproved deployments.

228
00:12:05.639 --> 00:12:10.279
<v Speaker 2>That might bypass reviews. And given how complex AI systems are,

229
00:12:10.799 --> 00:12:15.360
<v Speaker 2>change management is vital. Planning for changes explicitly planning for changes,

230
00:12:15.639 --> 00:12:19.519
<v Speaker 2>back end code APIs, the user interface data drift, even

231
00:12:19.559 --> 00:12:23.159
<v Speaker 2>third party dependencies. It's all about preventing and detecting errors

232
00:12:23.159 --> 00:12:24.399
<v Speaker 2>before they cause problems.

233
00:12:24.399 --> 00:12:26.600
<v Speaker 1>It makes sense, but things can still go wrong.

234
00:12:26.720 --> 00:12:29.879
<v Speaker 2>They absolutely can. You can't eliminate all risk. So the

235
00:12:29.919 --> 00:12:34.799
<v Speaker 2>book stresses having robust AI incident response plans. Absolutely what

236
00:12:34.840 --> 00:12:39.240
<v Speaker 2>does that? It's usually broken down into six phases. First preparation,

237
00:12:39.799 --> 00:12:46.600
<v Speaker 2>defining incidents, getting budget, planning, communications, running drills like tabletop exercises. Okay, ready,

238
00:12:46.879 --> 00:12:52.799
<v Speaker 2>then identification, spotting the failure, attack or abuse quickly. Third containment,

239
00:12:53.559 --> 00:12:59.399
<v Speaker 2>stopping the bleeding, mitigating immediate harm. Fourth eradication actually.

240
00:12:59.080 --> 00:13:01.840
<v Speaker 1>Fixing the effect system, rid of the root cause right.

241
00:13:02.320 --> 00:13:06.600
<v Speaker 2>Fifth recovery getting things back to normal operation. And finally,

242
00:13:06.720 --> 00:13:10.879
<v Speaker 2>super important lessons learned analyzing what happened to improve future

243
00:13:10.919 --> 00:13:13.639
<v Speaker 2>plans and prevent it from happening again. It's a whole cycle.

244
00:13:13.720 --> 00:13:17.679
<v Speaker 1>Preparation, identification, containment, eradication, recovery, lessons learned.

245
00:13:18.039 --> 00:13:21.080
<v Speaker 2>Got it. Let's maybe make this more concrete with that

246
00:13:21.120 --> 00:13:24.080
<v Speaker 2>case study you mentioned, the Uber autonomous vehicle insident.

247
00:13:24.159 --> 00:13:27.759
<v Speaker 1>Yeah, it's a sobering one twenty eighteen Tempe, Arizona. Elaine

248
00:13:27.799 --> 00:13:30.639
<v Speaker 1>Hersberg was crossing the street outside a crosswalk and was

249
00:13:30.639 --> 00:13:33.240
<v Speaker 1>struck and killed by an autonomous Uber test.

250
00:13:33.000 --> 00:13:34.360
<v Speaker 2>Vehicle and the safety driver.

251
00:13:34.519 --> 00:13:37.440
<v Speaker 1>The driver was reportedly distracted looking down at a smartphone,

252
00:13:37.639 --> 00:13:40.799
<v Speaker 1>but crucially, the AI system itself failed to identify her

253
00:13:40.840 --> 00:13:44.120
<v Speaker 1>as a pedestrian until just one point two seconds before impact.

254
00:13:44.360 --> 00:13:45.919
<v Speaker 1>One point two seconds.

255
00:13:46.360 --> 00:13:48.799
<v Speaker 2>Far too late, far too late to avoid the crash.

256
00:13:48.879 --> 00:13:52.120
<v Speaker 1>So what did the investigation find? What went wrong? From

257
00:13:52.120 --> 00:13:54.279
<v Speaker 1>a responsible AI perspective.

258
00:13:53.840 --> 00:13:56.840
<v Speaker 2>Well, this really raises the core question right. The NTSB,

259
00:13:57.039 --> 00:14:01.240
<v Speaker 2>that's the National Transportation Safety Board findings were pretty damning.

260
00:14:01.639 --> 00:14:06.080
<v Speaker 2>They found Uber system design hadn't even considered jaywalking pedestrians

261
00:14:06.159 --> 00:14:10.440
<v Speaker 2>as a foreseeable failure mode. Seriously, jaywalking, Yeah, it points

262
00:14:10.559 --> 00:14:14.320
<v Speaker 2>to really lacks risk assessments and frankly an immature safety

263
00:14:14.320 --> 00:14:17.360
<v Speaker 2>culture at the time. Disturbingly, it came out that an

264
00:14:17.360 --> 00:14:21.360
<v Speaker 2>employee had actually raised concerns about thirty seven previous crashes

265
00:14:21.399 --> 00:14:24.559
<v Speaker 2>involving these vehicles in the eighteen months prior, concerns that

266
00:14:24.600 --> 00:14:25.840
<v Speaker 2>were largely ignored.

267
00:14:25.960 --> 00:14:28.120
<v Speaker 1>Wow. That speaks volumes about the culture.

268
00:14:28.200 --> 00:14:31.000
<v Speaker 2>It really does, and the lessons learn are stark. Lesson

269
00:14:31.039 --> 00:14:34.960
<v Speaker 2>one culture is paramount. A mature safety culture might have

270
00:14:35.000 --> 00:14:36.759
<v Speaker 2>caught this, might have listened to those concerns.

271
00:14:36.879 --> 00:14:36.960
<v Speaker 1>Right.

272
00:14:37.320 --> 00:14:41.720
<v Speaker 2>Lesson two mitigate foreseeable failure modes. Jaywalking, especially at night,

273
00:14:41.720 --> 00:14:44.080
<v Speaker 2>should have been an easily foreseeable problem for a self

274
00:14:44.159 --> 00:14:48.200
<v Speaker 2>driving car. AI is only prepared if we humans prepare

275
00:14:48.240 --> 00:14:49.679
<v Speaker 2>it by anticipating these things.

276
00:14:49.840 --> 00:14:51.840
<v Speaker 1>Seems obvious in hindsight, doesn't it?

277
00:14:52.279 --> 00:14:57.120
<v Speaker 2>And Lesson three test rigorously in the operating domain. After

278
00:14:57.159 --> 00:15:00.759
<v Speaker 2>the crash, Uber paused and revamped improved soft where pested

279
00:15:00.840 --> 00:15:03.600
<v Speaker 2>later in simulation showed the vehicle could have started breaking

280
00:15:03.600 --> 00:15:04.759
<v Speaker 2>four seconds before.

281
00:15:04.519 --> 00:15:08.000
<v Speaker 1>Impact, four seconds versus one point two huge difference.

282
00:15:08.240 --> 00:15:12.799
<v Speaker 2>Massive. It just underscores the critical need for realistic in

283
00:15:12.879 --> 00:15:15.440
<v Speaker 2>domain testing before you put these things on public roads.

284
00:15:15.440 --> 00:15:18.320
<v Speaker 2>It's a tragic reminder of the human cost when these

285
00:15:18.320 --> 00:15:19.679
<v Speaker 2>principles aren't fully embraced.

286
00:15:20.159 --> 00:15:24.039
<v Speaker 1>Okay, so we've covered the why, the who, the organizational how.

287
00:15:24.679 --> 00:15:28.440
<v Speaker 1>Let's shift gears now and really unpack the technical toolkit

288
00:15:28.480 --> 00:15:32.200
<v Speaker 1>the how for the practitioners actually building these systems. Where

289
00:15:32.200 --> 00:15:32.879
<v Speaker 1>does that start.

290
00:15:33.039 --> 00:15:35.879
<v Speaker 2>It really has to start with robust training practices for

291
00:15:35.960 --> 00:15:39.679
<v Speaker 2>the mL algorithms themselves, and the absolute bedrock the foundation

292
00:15:39.759 --> 00:15:41.600
<v Speaker 2>for everything is reproducibility.

293
00:15:41.679 --> 00:15:43.519
<v Speaker 1>Reproducibility. Why is that so fundamental?

294
00:15:43.600 --> 00:15:46.320
<v Speaker 2>Because without it you can't reliably tell if an improvement

295
00:15:46.320 --> 00:15:48.679
<v Speaker 2>you made is real. You can't reliably audit what happened

296
00:15:48.679 --> 00:15:52.120
<v Speaker 2>if something goes wrong. It's basic scientific rigor applied to AI.

297
00:15:52.320 --> 00:15:53.679
<v Speaker 1>So what does that mean in practice?

298
00:15:53.879 --> 00:15:58.120
<v Speaker 2>It means several things. Using benchmark models for comparison, ensuring

299
00:15:58.159 --> 00:16:01.600
<v Speaker 2>consistent hardware and software environment using tools like Docker or

300
00:16:01.720 --> 00:16:06.799
<v Speaker 2>virtual environments, Meticulously tracking metadata maybe with tools like TensorFlow,

301
00:16:06.919 --> 00:16:12.200
<v Speaker 2>mL metadata setting, random seeds consistently and crucially using robust

302
00:16:12.279 --> 00:16:16.639
<v Speaker 2>version control for everything. Code and data get GitHub and

303
00:16:16.720 --> 00:16:20.120
<v Speaker 2>tools like DVC data version control are key here.

304
00:16:20.279 --> 00:16:22.360
<v Speaker 1>Okay, reproducibility first.

305
00:16:22.559 --> 00:16:24.919
<v Speaker 2>Then what then you hit the critical area of data

306
00:16:25.000 --> 00:16:28.600
<v Speaker 2>quality and feature engineering. You know, entire books are written.

307
00:16:28.320 --> 00:16:29.000
<v Speaker 1>Just on this topic.

308
00:16:29.039 --> 00:16:31.639
<v Speaker 2>It's huge, it is and from a safety and performance view,

309
00:16:31.919 --> 00:16:34.240
<v Speaker 2>the author stress that bad data is so often the

310
00:16:34.320 --> 00:16:37.120
<v Speaker 2>root cause of real world failures. You have to address

311
00:16:37.159 --> 00:16:40.120
<v Speaker 2>things like data set size and shape. Small or sparse

312
00:16:40.200 --> 00:16:43.200
<v Speaker 2>data can make models really unstable. You need to look

313
00:16:43.200 --> 00:16:46.600
<v Speaker 2>for misrepresentation, overfitting, potential pipeline issues.

314
00:16:47.039 --> 00:16:48.919
<v Speaker 1>Is there a quick checklist of things to look for.

315
00:16:49.159 --> 00:16:52.559
<v Speaker 2>Yeah, they offer a good starting point. Check for duplicate data,

316
00:16:53.000 --> 00:16:58.799
<v Speaker 2>incorrect encoding, weird outliers, using simplistic imputation methods that might

317
00:16:58.879 --> 00:17:03.679
<v Speaker 2>hide problems, issues with correlations between features, making sure normalization

318
00:17:03.799 --> 00:17:07.039
<v Speaker 2>is done correctly. It's a lot, but it's.

319
00:17:06.920 --> 00:17:10.440
<v Speaker 1>Crucial, got it. Data quality is non negotiable. What about

320
00:17:10.480 --> 00:17:12.359
<v Speaker 1>specifying the model itself? Right?

321
00:17:12.519 --> 00:17:14.880
<v Speaker 2>Model specification? This is where the authors say we need

322
00:17:14.880 --> 00:17:17.240
<v Speaker 2>to go beyond just chasing the top score on some

323
00:17:17.440 --> 00:17:18.039
<v Speaker 2>leader board.

324
00:17:18.200 --> 00:17:19.880
<v Speaker 1>Performance is in everything, not.

325
00:17:19.920 --> 00:17:23.160
<v Speaker 2>The only thing. It involves starting with established peer reviewed

326
00:17:23.160 --> 00:17:28.039
<v Speaker 2>benchmarks and alternatives. Evaluating multiple approaches not just for raw accuracy,

327
00:17:28.119 --> 00:17:31.720
<v Speaker 2>but also for things like compliance with non discrimination standard.

328
00:17:31.400 --> 00:17:32.759
<v Speaker 1>And being aware of assumptions.

329
00:17:32.960 --> 00:17:36.839
<v Speaker 2>Yes, this is a subtle but critical insight, being acutely

330
00:17:36.880 --> 00:17:40.480
<v Speaker 2>aware of the hidden assumptions baked into mL algorithms. Things

331
00:17:40.519 --> 00:17:43.400
<v Speaker 2>they implicitly assume about your data structure, like high degree

332
00:17:43.440 --> 00:17:47.680
<v Speaker 2>interactions or nonlinearities, or assumptions about the target distribution, like

333
00:17:47.720 --> 00:17:51.319
<v Speaker 2>how using a squared law function implicitly assumes errors are

334
00:17:51.319 --> 00:17:54.799
<v Speaker 2>normally distributed. If those assumptions are wrong, your model might

335
00:17:54.839 --> 00:17:58.400
<v Speaker 2>look accurate on paper but break spectacularly in the real world.

336
00:17:58.640 --> 00:18:02.440
<v Speaker 2>That's why things like explicit applying monotonicity constraints ensuring a

337
00:18:02.440 --> 00:18:06.480
<v Speaker 2>feature change always pushes the prediction in one expected direction,

338
00:18:06.720 --> 00:18:09.720
<v Speaker 2>or interaction constraints limiting how features combined can be so

339
00:18:09.960 --> 00:18:13.519
<v Speaker 2>valuable for aligning the model with reality and preventing weird

340
00:18:13.799 --> 00:18:14.920
<v Speaker 2>unexpected behavior.

341
00:18:15.160 --> 00:18:19.200
<v Speaker 1>Okay, so build on a solid foundation, handle data carefully,

342
00:18:19.440 --> 00:18:23.400
<v Speaker 1>specify thoughtfully. What about debugging? Do we treat models like

343
00:18:23.440 --> 00:18:24.799
<v Speaker 1>code exactly?

344
00:18:24.960 --> 00:18:27.640
<v Speaker 2>That's the shift in perspective the book advocates for in

345
00:18:27.720 --> 00:18:31.960
<v Speaker 2>model debugging, treat mL models like complex software systems, not

346
00:18:32.000 --> 00:18:35.799
<v Speaker 2>just abstract math equations, and that demands rigorous software testing,

347
00:18:35.880 --> 00:18:39.720
<v Speaker 2>specifically for AI, beyond the usual unit tests and integration

348
00:18:39.839 --> 00:18:43.160
<v Speaker 2>tests way beyond. Of course, you need those standard QA practices,

349
00:18:43.400 --> 00:18:46.079
<v Speaker 2>but for AI you also need things like chaos testing,

350
00:18:46.160 --> 00:18:49.680
<v Speaker 2>intentionally throwing chaotic or adversarial conditions at the system to

351
00:18:49.680 --> 00:18:50.240
<v Speaker 2>see if it.

352
00:18:50.160 --> 00:18:52.160
<v Speaker 1>Breaks, stress testing it pretty much.

353
00:18:52.319 --> 00:18:57.240
<v Speaker 2>And then mL specific tests like random attack where you

354
00:18:57.319 --> 00:19:00.279
<v Speaker 2>just flood the system with massive amounts of random, maybe

355
00:19:00.279 --> 00:19:05.119
<v Speaker 2>nonsensical data to find hidden bugs or vulnerabilities. And continuous

356
00:19:05.160 --> 00:19:09.319
<v Speaker 2>benchmarking tracking performance over time, especially within CICD pipelines, continuous

357
00:19:09.319 --> 00:19:11.000
<v Speaker 2>integration and deployment right.

358
00:19:10.880 --> 00:19:15.400
<v Speaker 1>Tracking improvements reliably. Now what about traditional model assessment accuracy AUC,

359
00:19:15.640 --> 00:19:16.079
<v Speaker 1>et cetera.

360
00:19:16.200 --> 00:19:19.200
<v Speaker 2>Yeah, traditional model assessment. The key insight here is that

361
00:19:19.279 --> 00:19:21.960
<v Speaker 2>the exact numeric value often matters less than how the

362
00:19:22.000 --> 00:19:26.759
<v Speaker 2>model performs in its specific domain. You want logical interpretable stats, sure,

363
00:19:26.839 --> 00:19:31.799
<v Speaker 2>like RMSE ROOTMANE squared error for regression or AUC area

364
00:19:31.880 --> 00:19:36.359
<v Speaker 2>under the curve for classification, always setting practical real world thresholds.

365
00:19:36.400 --> 00:19:39.319
<v Speaker 1>But the aggregate score isn't the whole story, not at all.

366
00:19:39.400 --> 00:19:42.519
<v Speaker 2>The crucial step is analyzing performance across important segments of

367
00:19:42.519 --> 00:19:46.480
<v Speaker 2>your data, different demographic groups, different customer types, high risk

368
00:19:46.559 --> 00:19:49.319
<v Speaker 2>versus low risk cases, And you need to compare performance

369
00:19:49.359 --> 00:19:52.319
<v Speaker 2>across your training, validation, and test sets. That's how you

370
00:19:52.319 --> 00:19:55.200
<v Speaker 2>spot hidden bias or under specification that gets lost in

371
00:19:55.240 --> 00:19:56.359
<v Speaker 2>the overall average.

372
00:19:56.519 --> 00:19:59.279
<v Speaker 1>And thinking about classification thresholds.

373
00:19:58.759 --> 00:20:02.960
<v Speaker 2>Absolutely carefully electing those probability cut off thresholds. What's the

374
00:20:02.960 --> 00:20:07.680
<v Speaker 2>real world impact monetary return, risk exposure? How does it

375
00:20:07.720 --> 00:20:11.400
<v Speaker 2>affect different groups differently? That needs careful consideration.

376
00:20:11.799 --> 00:20:14.759
<v Speaker 1>So digging deeper than the top line number you mentioned.

377
00:20:14.920 --> 00:20:18.359
<v Speaker 1>Learning from errors? What was that technique? Ah?

378
00:20:18.480 --> 00:20:22.960
<v Speaker 2>Yes, residual analysis for machine learning. This is incredibly powerful.

379
00:20:23.319 --> 00:20:26.359
<v Speaker 2>Think of it literally as learning from your model's mistakes.

380
00:20:26.519 --> 00:20:27.279
<v Speaker 1>How does that work?

381
00:20:27.359 --> 00:20:31.920
<v Speaker 2>It involves deep analysis and visualizations. You plot the residuals,

382
00:20:31.960 --> 00:20:34.839
<v Speaker 2>the difference between the model's prediction and the actual outcome

383
00:20:34.839 --> 00:20:37.920
<v Speaker 2>against different features or prediction levels. You look for patterns

384
00:20:38.440 --> 00:20:41.680
<v Speaker 2>like maybe your credit model consistently makes large errors for

385
00:20:41.759 --> 00:20:45.039
<v Speaker 2>people who were good payers, but suddenly default patterns a

386
00:20:45.079 --> 00:20:48.640
<v Speaker 2>human might spot. The book also talks about actually modeling

387
00:20:48.640 --> 00:20:52.599
<v Speaker 2>the residuals themselves, using an interpretable model like a decision

388
00:20:52.640 --> 00:20:55.319
<v Speaker 2>tree to predict the errors of your main mL model.

389
00:20:55.519 --> 00:20:58.400
<v Speaker 1>So building a model of the mistakes exactly.

390
00:20:58.480 --> 00:21:01.400
<v Speaker 2>It helps you pinpoint specific failure modes and then design

391
00:21:01.519 --> 00:21:06.119
<v Speaker 2>targeted fixes like adding specific rules or model assertions. And

392
00:21:06.240 --> 00:21:09.960
<v Speaker 2>a really cool recent development is using Shaply values to

393
00:21:10.240 --> 00:21:12.559
<v Speaker 2>calculate the local contribution to residuals.

394
00:21:12.599 --> 00:21:15.319
<v Speaker 1>Shaply values those explained predictions, right, they.

395
00:21:15.200 --> 00:21:18.000
<v Speaker 2>Do, but here you use them to see which features

396
00:21:18.039 --> 00:21:21.240
<v Speaker 2>are driving the errors, not just the predictions. This can

397
00:21:21.279 --> 00:21:24.680
<v Speaker 2>reveal non robust futures, ones that contribute more to mistakes

398
00:21:24.720 --> 00:21:27.680
<v Speaker 2>and to accurate predictions. Super insightful for debugging.

399
00:21:27.759 --> 00:21:31.240
<v Speaker 1>Wow. Okay, what about understanding how the model behaves overall,

400
00:21:31.279 --> 00:21:32.279
<v Speaker 1>how it extrapolates.

401
00:21:32.359 --> 00:21:35.759
<v Speaker 2>That's where sensitivity analysis comes in. Understanding how the model

402
00:21:35.799 --> 00:21:40.160
<v Speaker 2>responds to different inputs, especially unexpected ones. This involves stress testing,

403
00:21:40.200 --> 00:21:43.880
<v Speaker 2>simulating adverse scenarios like a recession or a pandemic to

404
00:21:43.920 --> 00:21:44.960
<v Speaker 2>see how robust the.

405
00:21:44.920 --> 00:21:46.559
<v Speaker 1>Model is, pushing it to its limits.

406
00:21:46.680 --> 00:21:51.200
<v Speaker 2>Right, and using visualizations tools like ale accumulated local effects

407
00:21:51.240 --> 00:21:56.880
<v Speaker 2>plots ICEE, individual conditional expectation plots, partial dependence curves, they

408
00:21:56.920 --> 00:22:00.279
<v Speaker 2>help you see how changing specific features influences.

409
00:21:59.680 --> 00:22:01.799
<v Speaker 1>Predictions and finding weird responses.

410
00:22:02.039 --> 00:22:06.519
<v Speaker 2>Yeah, through adversarial example searches basically trying to generate specific

411
00:22:06.599 --> 00:22:10.440
<v Speaker 2>data points that make the model behave unexpectedly or illogically.

412
00:22:11.039 --> 00:22:13.480
<v Speaker 2>The book has a great credit model example showing a

413
00:22:13.519 --> 00:22:17.200
<v Speaker 2>logical flaw punishing a late payment even after a large repayment,

414
00:22:17.480 --> 00:22:21.319
<v Speaker 2>and a security vulnerability like a weird steep spike and

415
00:22:21.440 --> 00:22:25.319
<v Speaker 2>risk for very small payment amounts. Finding these before deployment

416
00:22:25.519 --> 00:22:26.680
<v Speaker 2>is critical.

417
00:22:26.319 --> 00:22:28.960
<v Speaker 1>In valuable insights. And you mentioned beichmarks earlier. They keep

418
00:22:29.000 --> 00:22:29.519
<v Speaker 1>popping up.

419
00:22:29.680 --> 00:22:33.640
<v Speaker 2>They're crucial across the entire life cycle. Benchmark models not

420
00:22:33.839 --> 00:22:38.039
<v Speaker 2>just for reproducibility during training. They're vital for debugging, comparing

421
00:22:38.039 --> 00:22:41.680
<v Speaker 2>your complex model against a simpler, trusted baseline, and for

422
00:22:41.759 --> 00:22:44.799
<v Speaker 2>real time monitoring and production quickly flagging if your main

423
00:22:44.880 --> 00:22:46.680
<v Speaker 2>model starts deviating significantly.

424
00:22:46.880 --> 00:22:51.640
<v Speaker 1>Okay, so we have testing assessment, residual analysis, sensitivity analysis.

425
00:22:52.160 --> 00:22:54.839
<v Speaker 1>What specific kinds of bugs are we typically hunting for

426
00:22:54.920 --> 00:22:56.559
<v Speaker 1>in mL systems? Good question.

427
00:22:56.720 --> 00:23:01.599
<v Speaker 2>The author's list several common machine learning bugs. Distributional shifts

428
00:23:01.799 --> 00:23:04.799
<v Speaker 2>often come data drift or concept drift. This is huge.

429
00:23:05.160 --> 00:23:07.599
<v Speaker 2>It's when the live data the model sees starts to

430
00:23:07.599 --> 00:23:10.400
<v Speaker 2>differ significantly from the data it was trained on like.

431
00:23:10.440 --> 00:23:12.759
<v Speaker 1>COVID nineteen, changing shopping patterns.

432
00:23:12.359 --> 00:23:17.359
<v Speaker 2>Overnight exactly like that. Second instability, the model's training process

433
00:23:17.440 --> 00:23:21.000
<v Speaker 2>or its predictions might be erratic. Often happens with small

434
00:23:21.160 --> 00:23:25.039
<v Speaker 2>or sparse data, highly correlated features, or just using very

435
00:23:25.119 --> 00:23:26.599
<v Speaker 2>high variance model types.

436
00:23:26.720 --> 00:23:27.599
<v Speaker 1>Okay, what else?

437
00:23:27.759 --> 00:23:32.559
<v Speaker 2>Third looped inputs. This is subtle but dangerous. It's when

438
00:23:32.599 --> 00:23:36.000
<v Speaker 2>the AI's predictions affect the real world, and those effects

439
00:23:36.079 --> 00:23:38.839
<v Speaker 2>then get fed back into the model as new input data,

440
00:23:38.880 --> 00:23:40.400
<v Speaker 2>creating feedback loops.

441
00:23:40.319 --> 00:23:44.000
<v Speaker 1>Like predictive policing concentrating patrols in an area leading to

442
00:23:44.039 --> 00:23:47.440
<v Speaker 1>more arrests there, which then justifies more patrols.

443
00:23:47.039 --> 00:23:51.359
<v Speaker 2>Precisely, or employment algorithms subtly changing the applicant pool over time.

444
00:23:51.680 --> 00:23:55.039
<v Speaker 2>Fourth is leakage, when information from your validation or test

445
00:23:55.119 --> 00:23:58.519
<v Speaker 2>data accidentally contaminates your training data. This gives you overly

446
00:23:58.559 --> 00:24:01.519
<v Speaker 2>optimistic performing results just won't hold up in reality.

447
00:24:01.720 --> 00:24:02.759
<v Speaker 1>Sneaky, keep going.

448
00:24:03.000 --> 00:24:06.759
<v Speaker 2>Fifth classic overfitting. The model memorizes the training data noise

449
00:24:06.799 --> 00:24:11.960
<v Speaker 2>instead of learning general patterns. Sixth shortcut learning. This one's insidious.

450
00:24:12.319 --> 00:24:15.599
<v Speaker 2>The model learns an unintended similar correlation to make.

451
00:24:15.480 --> 00:24:19.440
<v Speaker 1>Predictions, like identifying the hospital scanner id instead of the

452
00:24:19.480 --> 00:24:21.359
<v Speaker 1>actual disease in a medical image.

453
00:24:21.519 --> 00:24:24.480
<v Speaker 2>Exactly, it gets the answer right on the training data,

454
00:24:24.559 --> 00:24:30.079
<v Speaker 2>but for totally the wrong and potentially dangerous reason. Seventh underfitting.

455
00:24:30.519 --> 00:24:33.279
<v Speaker 2>Basically the model is too simple or doesn't have enough

456
00:24:33.359 --> 00:24:36.880
<v Speaker 2>data or constraints to learn the underlying patterns.

457
00:24:36.640 --> 00:24:39.680
<v Speaker 1>And the last one under specification.

458
00:24:39.160 --> 00:24:42.680
<v Speaker 2>Right under specification. This is a tricky one. It means

459
00:24:42.759 --> 00:24:46.319
<v Speaker 2>multiple different models could achieve similar high accuracy on your

460
00:24:46.400 --> 00:24:49.880
<v Speaker 2>validation set, but validation alone isn't enough to pick the

461
00:24:49.920 --> 00:24:53.640
<v Speaker 2>truly best or most robust one. It often shows up

462
00:24:53.640 --> 00:24:57.160
<v Speaker 2>as performance being weirdly sensitive to things like random seeds

463
00:24:57.240 --> 00:25:01.079
<v Speaker 2>or computational hyper parameters, or seeing performance suddenly shift across

464
00:25:01.119 --> 00:25:02.240
<v Speaker 2>different data segments.

465
00:25:02.279 --> 00:25:04.200
<v Speaker 1>Okay, that's a lot of potential bugs. How do we

466
00:25:04.240 --> 00:25:06.319
<v Speaker 1>fix them? What are the remediation strategies?

467
00:25:06.400 --> 00:25:10.160
<v Speaker 2>The book offers a whole toolkit for remediation. Using anomaly

468
00:25:10.240 --> 00:25:15.079
<v Speaker 2>detection is key to spot weird inputs or outputs. Experimental

469
00:25:15.079 --> 00:25:18.599
<v Speaker 2>design and data augmentation help you collect or generate better,

470
00:25:18.759 --> 00:25:23.079
<v Speaker 2>more robust training data. Model assertions are like applying business

471
00:25:23.200 --> 00:25:25.720
<v Speaker 2>rules on top of the model to correct specific known

472
00:25:25.759 --> 00:25:27.680
<v Speaker 2>shortcomings in its learned logic.

473
00:25:27.839 --> 00:25:29.160
<v Speaker 1>Hard coding fixes.

474
00:25:29.319 --> 00:25:33.400
<v Speaker 2>Sometimes yes, or guardrails for certain interpretable models like GA

475
00:25:33.440 --> 00:25:36.599
<v Speaker 2>two ms or EBMs. You might even do model editing directly.

476
00:25:37.079 --> 00:25:41.400
<v Speaker 2>Of course, ongoing model management and monitoring are themselves remediation strategies.

477
00:25:41.640 --> 00:25:44.559
<v Speaker 2>You can apply those monotonicity and interaction constraints we talked

478
00:25:44.559 --> 00:25:48.200
<v Speaker 2>about earlier to enforce real world logic, and techniques like

479
00:25:48.240 --> 00:25:52.240
<v Speaker 2>noise injection during training or applying strong regularization can help

480
00:25:52.279 --> 00:25:55.160
<v Speaker 2>limit model complexity or force it to rely less on

481
00:25:55.200 --> 00:25:56.839
<v Speaker 2>potentially problematic features.

482
00:25:57.079 --> 00:25:59.319
<v Speaker 1>So a range of tools depending on the bug. Once

483
00:25:59.319 --> 00:26:03.079
<v Speaker 1>we've debugged and remediated, what about actually deploying the model right?

484
00:26:03.200 --> 00:26:07.160
<v Speaker 2>Crucial deployment considerations first is domain safety, focusing on real

485
00:26:07.200 --> 00:26:11.319
<v Speaker 2>world safety. This includes standard practices like ab testing, running

486
00:26:11.400 --> 00:26:15.079
<v Speaker 2>Champion challenger tests against existing systems on live data, but

487
00:26:15.200 --> 00:26:19.720
<v Speaker 2>also explicitly anticipating foreseeable incidents and preparing for the unforeseeable

488
00:26:19.759 --> 00:26:24.279
<v Speaker 2>ones using things like chaos testing random attacks, and setting

489
00:26:24.319 --> 00:26:27.799
<v Speaker 2>common sense prediction limits, maybe requiring human review for really

490
00:26:27.880 --> 00:26:29.720
<v Speaker 2>high stakes or unusual predictions.

491
00:26:29.799 --> 00:26:31.519
<v Speaker 1>Human oversight is a backstop.

492
00:26:31.279 --> 00:26:35.240
<v Speaker 2>Often essential, and continuous model monitoring is technically vital here too,

493
00:26:35.720 --> 00:26:39.880
<v Speaker 2>detecting that model decay and concept drift using statistical tests,

494
00:26:40.119 --> 00:26:42.119
<v Speaker 2>control limits, anomaly detection.

495
00:26:42.359 --> 00:26:43.480
<v Speaker 1>What do you do when you detect it?

496
00:26:43.759 --> 00:26:46.799
<v Speaker 2>You need strategies for addressing it, maybe retraining the model,

497
00:26:46.839 --> 00:26:49.640
<v Speaker 2>maybe just refreshing it with newer data. Crucially, you need

498
00:26:49.680 --> 00:26:54.400
<v Speaker 2>to measure multiple key performance indicators KPIs in production, not

499
00:26:54.519 --> 00:26:58.519
<v Speaker 2>just accuracy. How's it doing on fairness, metrics, security, privacy?

500
00:26:58.559 --> 00:27:01.640
<v Speaker 2>What's the actual business impact? You also need robust ways

501
00:27:01.680 --> 00:27:03.960
<v Speaker 2>to handle out of range values in the live.

502
00:27:03.839 --> 00:27:06.480
<v Speaker 1>Data feed, and those kill switches, Yes.

503
00:27:06.599 --> 00:27:10.359
<v Speaker 2>Kill switches, the author's stress. These aren't usually single big

504
00:27:10.400 --> 00:27:13.079
<v Speaker 2>red buttons. They're a bundle of pre planned business and

505
00:27:13.119 --> 00:27:16.119
<v Speaker 2>technical processes designed to let you safely turn off or

506
00:27:16.240 --> 00:27:19.400
<v Speaker 2>roll back an AI system if things go seriously wrong,

507
00:27:19.839 --> 00:27:23.599
<v Speaker 2>especially in high stakes scenarios, and these procedures absolutely must

508
00:27:23.640 --> 00:27:25.920
<v Speaker 2>be documented in your incident response plans.

509
00:27:26.000 --> 00:27:29.000
<v Speaker 1>Okay, that makes sense. Can we quickly revisit that straw

510
00:27:29.079 --> 00:27:32.799
<v Speaker 1>Man model example to see how remediation might work in practice,

511
00:27:33.079 --> 00:27:35.200
<v Speaker 1>the one that over used the PA way zero.

512
00:27:35.000 --> 00:27:38.400
<v Speaker 2>Future Sure that gradient boosting machine model had several issues,

513
00:27:38.480 --> 00:27:42.960
<v Speaker 2>right it overemphasized the most recent repayment status. PA zero

514
00:27:43.440 --> 00:27:47.960
<v Speaker 2>had non robust inputs, contributing to errors of vulnerable response surface,

515
00:27:48.039 --> 00:27:49.519
<v Speaker 2>poor performance, and segments.

516
00:27:49.640 --> 00:27:51.880
<v Speaker 1>So how would you fix it applying those techniques?

517
00:27:51.920 --> 00:27:54.720
<v Speaker 2>Okay, So for the logical errors like predicting high default

518
00:27:54.799 --> 00:27:57.720
<v Speaker 2>risk even after a large repayment, model assertions would be

519
00:27:57.839 --> 00:28:01.599
<v Speaker 2>a strong candidate, just at a rule. If it pathologically

520
00:28:01.640 --> 00:28:05.160
<v Speaker 2>overemphasized pay zero, you could try to improve the training

521
00:28:05.240 --> 00:28:08.720
<v Speaker 2>data using data augmentation or noise injection around that feature,

522
00:28:09.279 --> 00:28:12.640
<v Speaker 2>or run specific experiments to gather data that forces decision

523
00:28:12.640 --> 00:28:14.079
<v Speaker 2>making across more features.

524
00:28:14.240 --> 00:28:15.240
<v Speaker 1>Spread the load right.

525
00:28:15.640 --> 00:28:19.440
<v Speaker 2>For those non robust input variables PAY two, Pay three

526
00:28:19.559 --> 00:28:22.759
<v Speaker 2>that drove errors more than predictions, you'd want to ab

527
00:28:22.880 --> 00:28:26.000
<v Speaker 2>test performance with and without them. Maybe retrain the model

528
00:28:26.000 --> 00:28:30.160
<v Speaker 2>completely without those features, or apply strong regularizations specifically to

529
00:28:30.240 --> 00:28:31.200
<v Speaker 2>reduce their influence.

530
00:28:31.279 --> 00:28:33.960
<v Speaker 1>Okay, what about the security vulnerability that spike.

531
00:28:34.160 --> 00:28:37.319
<v Speaker 2>That security vulnerability, the sharp production spike at low payment

532
00:28:37.400 --> 00:28:40.960
<v Speaker 2>values that needs layers API throttling to prevent rapid fire

533
00:28:41.000 --> 00:28:45.920
<v Speaker 2>testing by attackers, robust authentication, vigilant monitoring for unusual patterns,

534
00:28:46.160 --> 00:28:49.640
<v Speaker 2>and potentially use in more inherently robust mL techniques if possible.

535
00:28:49.720 --> 00:28:52.359
<v Speaker 1>And the poor performance for customers with multiple late payments.

536
00:28:52.640 --> 00:28:55.440
<v Speaker 2>Yeah, the poor performance where p y zero one. You

537
00:28:55.519 --> 00:28:58.640
<v Speaker 2>might need better data for that segment, maybe apply observation

538
00:28:58.720 --> 00:29:02.759
<v Speaker 2>weights to focus learning there, monotonicity constraints might help ensure

539
00:29:02.799 --> 00:29:07.400
<v Speaker 2>sensible behavior. Or honestly, you might decide that for those specific,

540
00:29:07.480 --> 00:29:10.839
<v Speaker 2>high risk, hard to predict cases, the best approach is

541
00:29:10.880 --> 00:29:14.519
<v Speaker 2>to route them to a human caseworker instead of relying solely.

542
00:29:14.240 --> 00:29:16.000
<v Speaker 1>On the model, A hybrid approach.

543
00:29:16.319 --> 00:29:18.440
<v Speaker 2>Sometimes that's the most responsible solution.

544
00:29:18.839 --> 00:29:21.720
<v Speaker 1>Wow, what a journey. We've really done a deep dive

545
00:29:21.759 --> 00:29:25.640
<v Speaker 1>today and it reinforces just how holistic responsible AI is.

546
00:29:26.079 --> 00:29:29.920
<v Speaker 1>It's clearly hard work. It definitely is requires that mix

547
00:29:29.960 --> 00:29:34.480
<v Speaker 1>of cultural change, solid processes, sophisticated tech practices, but it

548
00:29:34.519 --> 00:29:39.039
<v Speaker 1>feels necessary and also importantly achievable within reach for organizations

549
00:29:39.039 --> 00:29:40.279
<v Speaker 1>and individuals who commit to it.

550
00:29:40.359 --> 00:29:41.279
<v Speaker 2>I agree, and for.

551
00:29:41.279 --> 00:29:43.880
<v Speaker 1>You the listener, this journey to being well informed about

552
00:29:43.920 --> 00:29:47.279
<v Speaker 1>AI means understanding not just its incredible power, but also

553
00:29:47.359 --> 00:29:51.240
<v Speaker 1>these inherent risks and crucially the practical, actionable steps we

554
00:29:51.279 --> 00:29:52.279
<v Speaker 1>can take to manage them.

555
00:29:52.519 --> 00:29:55.519
<v Speaker 2>And this really raises, I think an important final question

556
00:29:55.599 --> 00:29:59.079
<v Speaker 2>for all of us, what's next? You know, the AI

557
00:29:59.200 --> 00:30:00.680
<v Speaker 2>genie is and truly.

558
00:30:00.480 --> 00:30:02.720
<v Speaker 1>Out of the bottle now no putting them back no.

559
00:30:03.000 --> 00:30:06.279
<v Speaker 2>And headlines about damaging AI incidents. They really took off

560
00:30:06.279 --> 00:30:11.359
<v Speaker 2>around twenty twenty. They're probably not going to stop until people, developers, organizations, policymakers,

561
00:30:11.480 --> 00:30:16.480
<v Speaker 2>users actively choose to remake AI into responsible AI. The

562
00:30:16.519 --> 00:30:18.599
<v Speaker 2>future I think will judge us on whether we took

563
00:30:18.640 --> 00:30:22.000
<v Speaker 2>AI safety seriously enough now to minimize those kinds of

564
00:30:22.079 --> 00:30:23.319
<v Speaker 2>somber incidents later.

565
00:30:23.599 --> 00:30:25.839
<v Speaker 1>That's a powerful thought. So here's where it gets really

566
00:30:25.880 --> 00:30:29.200
<v Speaker 1>interesting for you listening as AI weaves itself deeper into

567
00:30:29.240 --> 00:30:32.359
<v Speaker 1>our lives. Being able to recognize these principles, maybe even

568
00:30:32.400 --> 00:30:35.519
<v Speaker 1>push for them, it's vital. So whether you're building AI,

569
00:30:35.720 --> 00:30:38.359
<v Speaker 1>deploying it, regulating it, or just using it every day,

570
00:30:38.720 --> 00:30:41.319
<v Speaker 1>consider these frameworks we've talked about. Ask the tough questions.

571
00:30:41.640 --> 00:30:45.039
<v Speaker 1>Let's all try to ensure AI contributes positively rather than

572
00:30:45.119 --> 00:30:47.279
<v Speaker 1>inadvertently creating more problems for the world.