Hello, and welcome to This anthro Life, a podcast about the little things
we do as people that shape the course of humanity. I'm your host,
Adam Gamwell. Today we're going to talk about responsible AI, artificial intelligence,
and the future of the technology industry. But before we get into it,
let me share a story with you. Recently, a law enforcement agency in
the US deployed an AI system to predict which of its officers might need counseling
based on their social media posts. The system was meant to detect indicators of
mental health issues and help get struggling officers the help that they need. At
first, clans, it seemed like a great use of AI, but when
the system was analyzed, it turned out that the vast majority of the at
risk officers were people of color or women, were both expressing their concern about
systemic racism and sexism within their departments. In other words, the AI system
was flagging officers who were speaking out against injustices, rather than those who might
need help in counseling for mental health issues. The story highlights the urgent need
for responsible AI development and implementation, and that's precisely what we'll be talking about
today with our guests David Graywitterer and Don Nephis. David Graywitterer is a technology
and ethics researcher and PhD candidate at Carnegie Mellon University. He has conducted extensive
research on the intersection of ethics, technology, and labor as well as works
as a computer scientist. Don Nephis is a senior research scientist it Intel Labs,
and her work focuses on designing and studying systems that support personal informatics and
data driven decision making, as well as exploring the ethical implications of these systems.
Today, Don and David are going to help us explore three questions.
First, what are the limitations of the current modularity system in software development and
what impact can it have on user experience, data privacy, and end user
consumption. If you're not familiar with what a modular system is for development,
no worries. We're going to get into that in the earlier part of the
episode. Second, what are the implications of a siloed approach to user research
and design on AI ethics and responsible development? And how might we ensure that
the needs of a wider range of stakeholders are taken into account During development including
end users. And finally, what are the potential solutions to the power imbalances
between labor that's the folks that are doing the development and management, the folks
that are directing how that labor happens, and how might we build frameworks of
accountability and collective action across the technology industry landscape. So this conversation is fundamentally
important as AI is becoming more and more ubiquitous in our everyday lives and touching
more parts of how we move about our everyday worlds. We hope that this
conversation will help you better understand the complex and multifaceted challenges of responsible AI development
and implementation. We're going to be walking through some of the arguments that David
and Don put together in a new paper. We cannot wait to dive into
this with you, and so we'll jump right into it after a message from
this episode sponsor. So I just want to say Don and David super excited
to have you on the podcast today. It was fun to connect with David
at the Scity for Applied Anthropology this year in February, and we actually just
ran into each other and then we got to kind of talking about the work
that David was doing and immediately caught my attention as this is a conversation we
need to have on the podcast and helped bring to wider audiences. And I
was delighted also find that he's working with the amazing Don Nephis and great to
have you back on the program as well, and both the very timely topic.
But then also I think the work that you're both doing in terms of
bringing computer science into conversation with anthropology and anthropological thinking is super important. So
first off, I want to say to you both, thank you for being
on the show and excited to chat with you today. Excited to be here,
great to be here. Right on, David, tell us to your
superhero origin story, you know, how you found your way into this space,
and then then we'll jump over to Don Sure. So I am Peter
scientists by training. I come to AI from more of a technical perspective.
My first class in IT was thinking about how to build neural networks and such,
and in my research I tend to interview software engineers and try and understand
how they work, how they use tools, you know, So I come
at this more from a technical perspective. Having taken AI courses, but also
trying to understand how software engineers work, how they use tools, how they
make choices, how they make decisions, how they collaborate. And that led
to a few different internships NASA at Microsoft and then also at Intel, where
I met Don and the more I learned about the existing research and responsible AI,
the more I at least saw that it relied on a particular idea of
what developers thought was their problem or their responsibility and what kind of power they
might have to action on what they thought their job was their sort of agency,
and that then led to me conceiving of some of my dissertation research as
thinking about will actually what kind of responsibility do individual developers feel over the downstream
impact of what they create? And given that, what do they feel they
can actually do about it? And the more I've worked across disciplinary lines such
as it's done, the more I've tried to think about those more individual factors
of responsibility and agency within sort of broader discourses and contexts that constrain and to
guide those. So I'll pick up next. So I've been an anthropologist in
industry for an embarrassingly long time, and I turned to responsible AI back when
it was really just exploding around twenty seventeen, twenty eighteen, where gender shades
had just come out. Folks were saying that this is a big deal,
this is a real issue, and internally, my lab director Lama Knoxman had
started leading the charge about, Okay, what do we need to do in
house to sort ourselves out? So we were looking in only what do we
need to do? You know, Intel does put out AI models, we
also have a lot of customers that use AI for various purposes, and so
the question of both how do we have to govern what kinds of models we
put out in the world, which is kind of where the developer experienced perception
understanding slash notion to supply chain became salient. And where does Intel sit in
all of this? Because you know, we're not Google, we're not Meta,
we don't really build the sort of stuff that you commonly see in the
news, So we're do our responsibilities begin and end? Became this question that
became part of my job to answer. So here I am here to answer
that question. Boy, folks, do we have the answer for you today?
I appreciate you both showing that and it's a really interesting arena again to
think about the importance of interdisciplinary approaches to these kinds of problems, right,
because one, as we're recording this in twenty twenty three, AI is becoming
more ubiquitous. I mean, the thing is, it's already been kind of
stealthily ubiquitous and a lot of technologies we might have used the word automation a
little bit more, but now with things like you know, chat GBT and
open AI and propic getting more getting more pressed, and there's just the stupid
explosion of AI chatbots on the iOS and Android app stores now that if they're
becoming just more common tools on the consumer side. And so I think what's
really interesting and I'm excited to explore with with both of you today, is
a lot of the work that you're looking at it is on the production the
developer side, right, And so oftentimes it's like how do things get shaped
before they even find their way over to consumer facing products for features? So
I think it might be useful for our listeners take zoom out a little bit
and describe the world that David and I have been working in, you know,
at a very broad level. One of the things you'll see regularly and
responsible AI reports and publications and even policies is developers should blah blah blah,
right, they should test for bias or developers should be transparent about la la
la la la, And it goes on and on. There's about a million
inventories you can find online that's sort of like have you done X Y and
say okay. And the problem that we're poking at is we're sort of saying,
okay, wait, which ones? Like which ones do you mean exactly
when you say developers should blah blah blah. And that's a problem that arises
in many different contexts. Right. So, for example, even in policymaking
right now, one of the things that many policymakers are struggling with or trying
to figure out how do you have a policy around, is, for example,
what happens when a model AI model gets open sourced? Right? This
discussions happened before the generative AI kind of became hot in the news. Right,
Even something as simple as does it detect a face? Like is it
a face or not? Never mind facial recognition, just is it a face
or not? Right? What's the responsibility when you put this out into open
source? And all of a sudden. You know, anybody in their brother
can do this, that and the other thing with it. And so that's
why we started to reframe. Okay, you know there's the notion of developers,
but who developers are actually situated in a political economy of sorts. If
you're an anthropologist, that's how you describe it. If you're a business person,
you describe it as an ecosystem, or you might also describe into supply
chain terms, which is what we did, both because that invokes kind of
corporate social responsibility, but also because that's how developers themselves see the world like
they see it in terms of a chain where they made a little part and
the little part is modularized and exactly the same way as you'd have a container
on a ship. Right, the language isn't coincidental at all. And another
scholars have been writing about how that way of imagining the world kind of inflects
what people do. Right, So we've said, okay, fine, you've
organized yourselves. There are many different actors, both companies and individuals, assembled
along what they see as a chain, and what we're saying is, okay,
how do you act that's a social circumstance so how do you act in
that to get the results that's going to actually work out for people? So
that's why we frame the problem in the way that we did. I think
that we've seen, at least in my view, responsible AI emerge as a
largely a set of principles we might have heard of, you know, fair
AI, accountable AI, transparent AI, and that sort of in a lot
of company context, but in a lot of like policy and regulatory context too,
seems to outline the boundaries of what constitutes responsible AI. So we want
to build these things in certain ways. We want to build what we're building
in a in a fair way, make sure the AI is fair, doesn't
discriminate. But what that does, in my view, and some of my
research has shown, is it sort of scopes what people see as their their
responsibility. You know, if we're going to build a fair AI, we
will build it in such a way that it is you know, doesn't have
bias in the model perhaps, But once that leaves our hands, that's where
perhaps some of our responsibility might stop. You know, the design attributes,
the fair, accountable, transparent design attributes, or what scope responsibility In many
conceptions of responsible aim and less thinking downstream less thinking about how the thing we
build may be used, and perhaps also thinking less upstream too, like you
know, oh, well, our data that we're getting from the web or
whatever has biased in it, but we didn't create that data, so that's
not on us in some sense, and that's where we came from, you
know, thinking about where I at least came from thinking about these principles and
what their perhaps shortcomings or limitations are, and then began thinking about how they
might limit the kind of scope that individual developers told to and believing certainly that
responsible I is important, then may find the limits or the scope of what
they see their job as being. And that's then where the AI supplied shape
or value chain sort of notion became helpful, and that it sort of helped
us zoom out from one particular developers context or sense of responsibility to seeing how
they relate to the many other developers because the many other modules that are necessary
to create finished AI systems, and how responsibility does or does not sort of
transfer throughout that those chains. I think that's that's a really important framing as
you're inquiring about this, because there is, of course the literal business and
lived reality of how developers work in a system, right, and I think
I think of echoes too of I mean this sounds weird to say this,
but like assembly line factory labor too right that you have your part of the
car and Henry Ford's factory, you build this part of the car and you
do that over and over again, and so you're not responsible for when the
car gets on the road or are you as one of the questions that you'll
are poking as and I think you know you're absolutely right there too done where
it's the language that we use shapes how we approach the process, what we
think about what it is that we're quote unquote supposed to be doing. And
so that kind of question I was putting us up with to his like responsible
AI as again a set of practices, as like where did it come from?
Versus a sense of responsibility that an individual hasn't And I think it's really
important that you've exploded that, say, we have to look between these two
in terms of if we're talking about developers should do something, which developers,
who's making it and where David to kind of drawn on some of the language
you brought up to in terms of Are we upstream? Are we downstream?
Where are we in the supply chain? Are we very close to consumer end
products and features? Are we at the first base code right where a consumer
will never see that unless there's some get a nerd that wants to dive into
it if it becomes open source later. Right? And so one thing I'm
curious to think about, two is this kind of typical process, you know
down you begin walking us into this space, and so thinking about what is
that development process typically look like? So we have the both the metaphor and
I think the reality right of working in these modules, right, these like
these sections of ship as it were that they end of help move part one
to Part two to Part three to part four. But then also I think
the other piece that I want to hear from you two is like how developers
also see themselves within these these modules. So David, your perspective, Howard,
developers thinking about themselves in the development process as part of this chain,
as part of these different modules, what does that look like for them?
Starting with something you sort of said a little bit ago thinking about modularity.
I mean I sit in a PhD in software engineering program where we read sort
of the papers that said, Hey, one day, systems are going to
become so big we can't think about them all at once. We have to
think about modularity, and this will enable all these wonderful benefits such as division
of labor in the factory context you just talked, and so really, you
know, modularity is held up in software engineering as sort of an unquestioned core
principle that allows us to build scale, to build large systems and have someone
be responsible for one thing and then someone else be responsible for using that thing,
but not knowing how it works. So in terms of how does AI
finished AI systems today get built, modularity allows it so that not each developer
has to understand the inner workings of the whole system, so that we can,
for example, oh, this is the standard data set. I got
it off the shelf. I don't know exactly how it was collected, and
I don't know exactly you know the issues at play, but we agree that
this is the standard module for this kind of thing. Or this is the
facial recognition model that we're going to use, and really don't need to know
the minute details of how it works, but we can tune it to to
work in this particular context. Or whatever for this particular product. So none
of this is terribly controversial. I mean, people use these sort of kickstarts
to make software engineering and building ani systems faster, to make it so they
can be built larger without having to intimately understand the inner workings, and developers
would of see this as part of being a good developer. You ought not
make the person using your module understand too much about how you built the thing
because it ought to work and just you know, render its interface easily understandable
to the next person. You know, as David is talking, well,
I'll say two things. First is that I think doing this work with David
helped me appreciate that there are times and places where it actually does make sense
to not have to get into the inner workings of every single little thing,
particularly when you're building complex systems. I think I've come to appreciate that more.
At the same time, it has all the hallmarks of what Charles Perraut
would call normal accidents. Right, you build up system that is sufficiently complex
where no one person necessarily understands the whole because you can't cognitively, you just
can't. That's a pretty good scenario for opening the door to failure, right,
because you have systems that layer upon systems upon systems, and then all
of a sudden you in that layer and you have a through line to badness
happening of some kind. His example is nuclear power accidents, right, Why
they continue to happen despite all this guards thrown on top and thrown on top
and thrown on top. Right. In this context, we have something similar
where everything things do layer over time and it is actual work. I mean
part of the problem here is it takes work to actually go digging through.
Oh wait, is the problem actually and the training data side, right,
what the models take in and learn from? Is it in Oh? Actually
something went wrong with the model itself, Like they didn't tune in the right
way? Is it in way the entity of the company putting it out in
the world actually could have made better choices about what context? Or is it
in the end? Use? Right? One of the things we know from
the anthropology is, of course using technology is a creative endeavor, right,
That's just true. So so there's this distance that you've got to travel through,
and it takes a lot. It takes both a lot of work and
a lot of kind of intellectual agility to go through all of that difference,
right, you know, modularities there for a reason, and really what we're
calling for is an apparatus to help folks traverse through it. Right. So
it might be that instead of just saying, look, okay, here's my
box, I did my work right, maybe it's things like asking the person
from whom you received your parts, right, your data whatever it is,
or your open source model, what's in this box? Like what do I
need to know about this box? Not necessarily what's in it, but what
do I need to know about how it behaves? Right? And building that
reflex a bit better, like things like that that kind of deepening the connections
or sort of the thing we're after here, and not necessarily just like what
do I need to know what's in this box to use it? Which is
kind of the state of affairs now, but like for purposes of thinking about
ethical questions, like what do I need to know about what's in this box?
For thinking about my own relation to the next person who might use what
I'm building and taking away, oh this is a standard component as sort of
a common refrain, and thinking more no, like I'm building a system that
is using a standard component. But I still want what I'm building to not
cause harm down for the next user or for the people who deploy it downstream,
or on whom it's deployed downstream. You know, I want to think
more about like what I'm depending on, you know, who I'm depending on,
and what the not just will it work? Will it fulfill the immediate
goal of what I'm trying to build, but which is already fulfilled through modularity
and sort of the interfaces that permit functionality, but thinking more about asking those
same kind of questions that don't outlined for ethical reasons, as like an analog.
If you're writing a dissertation or a research article, you are in essence
doing something similar of pulling a library sometimes literally of previous research and ideas and
thinkers to the amalgam them in a different way that can then be consumed in
a new way for different audiences. And we may not think about maybe we
should causing harm the same way when we're when we're putting a paper, dissertation
or book out into the world. But I mean, if we are putting
something that people can consume, obviously a book doesn't make decisions or maybe shapeway
traffic light works the way that generative AI may at some point. But the
modularity example, I think it's really interesting. Then I do think that there's
there's good ways to echo that. I think in a typical research process too.
And just again, so if folks either don't work in computer science or
like it still feels like a black box, then part of it is this
too, like what resources are we drawing from ahead of time to then build
into our system now to then see where where it goes next. And we
may find ourselves at one point and when a book itself can also then you
know, write it's next chapter. But that's I think we're not quite there
yet. I mean, maybe we are. I want to read exactly.
You know. It's funny you mentioned writing, Adam. I mean, I
think one of the things I learned in writing with a computer science student is
that, like mondularity is really epistemological. It's not just I know what my
job is and I do my job. It's like it gets into how you
think, to the point where computer scientists actually don't use word processors. This
is a thing. It's called overleaf, latex or late I've heard all the
you know, many different ways of pronouncing it. And it's wild, because
okay, it's wild if you're an anthropologist, right, it's totally normal if
you're computer scientists. But I am an anthropologist, so I'm gonna call it
wild. And what happens is like the way you write is you code and
then you render, right, So it's lines, I mean, it's lines
of texts. It's legible. There are a couple of commands that if you're
smart, where you could learn potentially I did not. But other than that,
it's like like the final text is a separate thing. It's like you
know, compiling code, and it's meant to be like code is meant to
be built in modules like good Apparently texting computer science is also meant to be
in clean modules. And I couldn't do it. I could not write this
paper like because there were these parts and there was no there were no visual
assets to actually build an interlocking argument in a way that I normally would.
And Jeremy Bonkers, you're very confident, you know, happy with with Lake
textbook. Well, again, it's the unquestioned norm in my field. Like
I also had to learn this early on in my PhD, and you know,
didn't have fun either um, but like to collaborate with anyone, and
at least within computer science is sort of expected to go back to the point
you know, Don was making um like, yeah, this is designed just
to with particular like rational goals in mind, like it's supposed to save labor,
and that you can literally import package, use package, like, oh,
you want a package, it's going to make really fancy tables and format
them nicely, so you ideally can save labor or at least if you know
how to use them. You import a package, you import someone else's fancy
table module and and then use it. But once something goes wrong with one
of those packages, good luck trying to fix the problem. Or also then
once you try and sort of render the thing or work with people who are
used to what computer sciences were called busy lake, what you see is what
you get editors like word, that's where problems and or hilarity ensues. The
modularity is sort of seen as a technical practice for people in my field.
It's seen as just simply the way we build code. And we can have
debates about the right way of modularizing things, you know, to fulfill one
value or one goal or another. But one of what what Don practly pointed
out is she as she spoke there was it functions more as an epistemic culture.
It sort of transcends the purely technical purpose and then begins to be used
as a way we think for the way we organize different concerns, separate concerns
to use the lingo in our field. And the hardest thing about is trying
to explain the point of this paper to people within my field is drawing the
connection between modularity as a technical practice, which we've talked about the implications and
benefits and disadvantages of here, and also to explain how that sort of then
transcends and it becomes an epistemic culture where it's seen as the right way to
organize relations, or the right way to do work, or the right way
to do much broader thing. And so as I try to explain that the
implications of modularity, it's hard to get people who see it as a technical
practice and purely that to see some of it's perhaps what we might call like
ethical side effects or implications in the way we think as as computer scientists.
Yeah, at the end of the paper, we have these three different proposals,
right, so everybody recognizes that this is a real problem. You know,
what do you do about it? And one of the things we say
is, look what you do about it depends precisely on how committed you are
to modularity in the first place. And so if you are like, this
is your bread and butter, this is what you do, right, Like,
the action you take is even better modularity. That solves the problem.
Right, So you assign somebody to do the biases, right, you frog
marks your user experience people in to anticipating the usages, so you know,
and then you can define like what the right usages should be going forward,
and what's otoscope right. Even more, you just divide them the modules according
to meeting the requirements of the chain, right, and then all the way
the other side. We essentially had a kind of a just reject the notion
of modularity entirely, right. So there are examples at this where if we
look at Mary Gray's work, we look at what the folks over in dare
are doing, which was a research institute recently set up after the Google got
rid of it. Many of its ethicis right. What they're doing is they're
saying that we do deep community work. Right, we take our time and
we make people as much of a part of the development process, so much
so that it doesn't matter what technology we end up with or not. The
point is to meet people's needs period. Right. If this is not a
technic analogy problem project, this is a problem solving project. And if we
have the technical skills that actually helped, great, if not also great,
we're going to solve the problem. And then in between we have a notion
that types some more advantage of the institutional forms that businesses often use, right,
might be contracts, a bunch of other stuff that is at neither end.
The amazing part was the people who read this paper who come from David's
part of the world, We're like, you can't reject modularity, Like,
how would that even work? Like you can't kild anything, right, That's
just not even possible. But then on the other side we had sts reviewers
saying the implication was, I know, you're just putting in this improved modularity
thing, is like this giveaway, but really the point is to reject it.
Obviously, because the whole point of the papers leading up to rejecting modularity,
right, just softballing in the middle, Right, yeah, yeah,
yeah, we're just softballing in I'm a come from industry, you know,
like you know those people, you know, so as like, wow,
two highly different points folks got from the paper was wild? Is that a
function of the fact that anthropologists and computer scientists are approaching this together, that
there are these multiple interpretations. I think that the responses like I mentioned that
I've gotten from computer scientists to speak to why I think it's important to think
about how we might improve ethics within dominant logics and modularity that exists today.
I mean, like, it's both funny that the first question people tend to
ask me when I in computer science, when I talk about this paper is
oh, well, how can we modularize ethics, Like what are the interfaces
we can reveal from our modules that will like can you do an API call
for bias or you know, like people will start start trying to then see
it as a modularized or modularizeable problem, essentially presenting modularity is a dominant ideology
that sort of subsumes and organizes other concerns like ethics beneath it. At the
same time, modular systems are what structure large systems today and other people in
my field will ask, well, like where does it end? Like if
we can't depend on modules, can we not depend on compilers? So modularity
is designed to manage complexity? How else would we build large systems? Now,
again, that reveals sort of an idea that, you know, we
ought to build large systems, and that's sort of the natural goal we must
do that, the natural goal we must hold m So you know, a
practical critique there. But and the more I think about it, the more
like I think this speaks to sort of a different orientation from at least computer
science and anthropology, which is like computer science tends to see the value in
building new things, like building systems. And I actually had someone asked me
recently like how long did you have to build things before you were allowed to
just comment, you know, and reading this article or and of course like
they were joking with me, they're a friend, you know. But the
first goal of this kind of work, if this paper at least is not
to say this is how we might build things better. The tagline at my
department that's printed art t shirts is build it better. So that sort of
shows you something about the orientation of software engineering departments. Perhaps, whereas thinking
more carefully about sort of the epistemological questions embedded in the way we build things
is seen as much more in anthropology at least, you know, Don can
speak more to that. But I see that as more what I've learned from
working with Don, what I've learned from collaborating with an anthropologist here. But
like Vie, the common refrain of like, well, you pointed out a
bunch of problems, Now what spurred us? Just think about the menu of
options approach? Like okay, So you'll have some readers that are very much
wanting to know what they can do better within the ethos of mondularity. But
then of course we'll have other readers who are like not necessarily added to or
sort of schooled in mondularity, who want to imagine other different futures that might
have no immediately applicable within many organizational context sort of practical ability to be implemented.
I would not call very grades project not practical to implement. It clearly
was within certain organizational structures. Yeah, like that's all I mean, you
know, all props to Microsoft, but it did happen. You know,
she is that Microsoft. So there are times in places where you can do
that, certainly nonprofits and geo's foundations, Right, I mean there's a whole
world out there that, with proper science policy, could actually proliferate and could
actually really develop and elaborate the kinds of broader social and technical technological trajectories that
we might want to have. Right, That's not the world we live in
now because we have an unequal situation. But I think they're entirely practical for
that side of them. Yeah, And we get to that a little bit
towards the end when we talk about the idea of like there's this unquestioned idea
of like general purpose modules, like well, general purpose for who like the
standard bricks or the standard shipping containers or modules that you can take off the
shelf to do certain things. Well, do the existing general purpose modules exist
to serve all of the kinds of things we might use software for equally or
do they tend to push towards particular ways of using software for particular people,
for particular needs business needs? So like what would you know? I don't
know, As Don mentioned, perhaps like public funding for software maintenance and creation
and sort of thinking of it as a public good approach. Perhaps or you
know, implications for policy in that direction. We're going to take a quick
break. Just wanted to let you know that we're running ads to support the
show down. We'll be right back. Is there something about A in general
that sparks this kind of conversation of what is my responsibility? If I back
up ten years and I'm developing the code for Facebook's social graph, we think
about that a bit differently in terms of a knowledge graph. And if I'm
putting my notes together in Obsidian or something, right, I guess something about
AI that makes us say we actually need this pause. I mean, obviously
there's been you know, Donn You mentioned the problems at Google with with the
firing of Tina Jibru and other issues like silencing or firing ethicists for raising questions.
And I'm just thinking out out here, wondering both from your paper and
through our conversation and just other things that we've seen recently. Sam Alt and
the CEO of Open Ai, was testifying for a congress. You know,
I was also perusing Lex Fridman's podcast and he had Max Techmark was talking about
the case for halting AI development, or at least just slowing it. Down
stop being in a quote unquote arms race to be the first out of the
gate, because GPT four is actually just the baby program. We think it's
a very pretty baby, but it's the first iteration of many more complex things
that are to come. And so this begs the bigger question, how will
AI change society? Or are we just more conscious now that we have social
media that's changed society in many ways, mostly for worse, some for better,
that we're thinking more intentionally about the impact AI might have. In other
words, is there something about AI that begs us to rethink the development process?
Such as modularity? It seems that there's a sense of urgency from the
examples that we're talking through that A will get out of control at some point
if we don't put safeguards in place. And so I'm curious for your perspective,
how is AI reshaping how we approach the problem or process of development.
You know, the issue is always what counts as AI is a moving kind
of goalposts, and you know, when you get into the weeds of it.
As far as I understand it, very few people actually use the term
AI, although recently it's become more popular as as it becomes the term of
the popular press, and I think you're right to point out that if you
look at technologies and harm right, there's a long list of them right through
the ethnographic and sociological record that we could point to, and arguably responsibility for
that should have been taken well before machine learning came on the scene, right.
I would love for somebody to actually dig into this properly. My guess
is that when something like gender shades comes up, where it's so palpably odious
that you have somebody not being seen literally because of who they are, you
can no longer deny or say that, oh, tech is neutral and the
only problem is how you use it, right, because clearly not so that
might have been part of why we're here. I will say that I think
the current sort of elaborations of the moment where folks have varying reasons to be
concerned about generative AI shall we say personally as an anthropologist, I would say
that some of them are more realistic than others. When I am concerned about,
again as an individual, is being distracted by made up harms at the
expense of, you know, the actual arms that we exist, And not
only that, but for the flavors of machine learning that aren't generative. Are
we going to suddenly give up because now we're worried about something that puts speech
together, all of a sudden, we're anthropomorphizing, putting an imagination of personhood
where it arguably doesn't belong, And so that's just fueling the fire. I
think also, I have friends who are getting PhDs and more of the technical
aspects behind AI, and there's a huge disconnect with between what they do and
what, like most systems marketed as AI are are doing. I think like
almost any software is now labeled AI software, at least in the public way
it's message or the way it's marketed. And yeah, I think that can
be distracting and sort of lead towards conceiving of AI ethics and questions of superhuman
possible like sort of science fiction ey scenarios like Terminator and the AI that might
kill us or might you know AGI. What I'm drawing, what I'm going
towards here artificial general intelligence singularly, and with that is the very lifelike chat
GPT, sort of human appearing convincingly putting together strings of words that sound very
much like a human. But when you actually have domain expertise in what it's
loviating about our less san But also in some of my more recent work,
I've sort of begun to question the impact of the conceptual category of AI in
responsible AI discussions, because, as we talked about earlier, like there's a
certain kind of question that's like, how do we build this thing ethically?
And there's another kind of question that's like, how do we use this or
how will we permit this system to be used ethically? In what ways?
Can we sell the system in what ways? And who should we sell the
system too? And scoping the question to specifically responsible AI scopes the question at
least in my view, to a design question like Okay, well we're using
AI to do this thing. Now we figure out how to design it ethically,
and now we figure out how to build it in an ethical way.
We want we want to be fair, we want to be accountable, transparent,
whatever, right now, That to me is another kind of trick that
distracts us from well the question of whether we're not we need to use AI
in the first place, machine learning or statistical techniques in the first place to
do this thing, but then also renders out of scope some of the more
difficult to answer questions when you're an individual software developer in a business. And
this speaks to our study too. Do I want my software being used by
the military? Do I want my software being used in this way for this
purpose that I might have concerns with? So AI in itself can scope the
kind of questions in a more narrow sense to questions of design rather questions of
use, And that might be something we want to rethink as we think about
AI ethics, because AI is certainly the term of the moment and the term
of art used in both business and in a lot of development context too.
But I think a lot of the same questions can be asked about software ethics
or about technology ethics more broughtly and and the and not doing so can have
certain implications for what is legitimate discuss or No. That's an incredibly helpful framework
to think with because as an outsider to software developments, as I think about
what we're talking through here, there is this question of the role that AI
is also helping us. Then reflect back on to your point, David,
like the broader elements of software development in the first place, right, Like,
how is something that's like hot on people's minds, helping us reflect back
on the entire production process. I think it's really valuable to ask this idea
of if we're falling into a moment of we're trying to design something, then
we might miss or scope out the question of ethics because we're not talking about
it's use case. As someone who has taught in a design program of taught
design thinking methodologies and research, I think about this because teaching UX classes user
experience classes. You know, we always talk about remember one of your first
things of doing user research is to find out who it is that you're designing
for. And when we are talking about designed versus research, right, is
there a level of research or user research that kind of goes into how we
might think about this too, of either how quickly or at one point ethical
questions do get asked? Like do they tend to get outsourced to the research
department? I know in the paper you notice in the development chain it's like
they tend to get pushed into different model cards later. But is the research
team also an area that we tend to see the ethics questions get siloed or
moved over? Yeah, it's like twofold right, So often user experience research
comes in at the end of the chain and there's an actual interface, there's
an actual product, like at that level, then that's when the user experienced
folks typically have something to say. I think it's also the case that it's
useful to do research kind of at that innovation level, which is one of
the reasons I do state until for as long as I do, is that
that is what we do. You know, we do food social research in
the process of doing AI innovation itself, specifically to set trajectories in a way
that are going to have social utility and uh, you know, a sense
of responsibility from the start, right. That's something a big company can do
that arguably is harder for smaller companies. What I am concerned about is being
a nerd and understanding other people's nerdiness. And at some level I do get
this, oh like like chasing the technological curiosity, like oh wait, if
I could just make it do this, you know, like, but then
you end up in a situation where your user is like the technological trajectory itself.
And that's that's why we're here, right, right, I mean,
we have people publicly making up uses on behalf of new forms of AI for
no good reason and rights. I've seen statements to the effect of well generative
AI of course will positively impact x y z, but there are negative implications.
It's like, have you demonstrated that, No, Like you're just saying
things now. There's a lot of just saying stuff and arguably more research is
necessary. But then you get that flip side of when research as a whole
enterprise. Right, Universities, think tanks all the rest of it are sitting
around chasing the trajectory that's essentially been set by a combination of very powerful,
usually white men, and you know the way that the engineering is unfolding.
Right then, like at some certain level, those wise minds might be better
spent looking at other, more socially productive things. Right, So we're in
this kind of vicious loop of chasing the things that you know, really need
significant levels of control at the expense of things that could potentially be better.
That's a great point that how are different people's expertise like have to be deployed,
right, Like, we have finite amounts of time and energy, and
so if we're chasing these rabbit holes, sometimes sometimes legit threat sometimes not right,
you know, then it does take away from these other these other projects
that have incredible value. That's a really interesting question too. Yeah, there
are a few points that done accidentally laid lay down there that I just want
to pick up on. One we make in the in the paper, which
is the idea that like user research leaves out actually a lot of people nowadays
because software systems are deployed on people who are not necessarily users in the sense
we think about them. There, people who are subject to algorithmic systems are
rarely the people that we're sort of seen as the pain user. And so
when we frame it as a user research problem, where it's probably thinking about
our customers, are probably thinking about a particular frame of person who willfully engages
with the system, or or at least knowingly engages with the system, rather
than the much more pervasive effects that happen or that can happen for people who
aren't in the user role. You can think about like maybe the police department
are the users of the predicted policing software, but the data subjects are the
ones whose thought crime is being predicted or future crime is being predicted. Right,
those are data subjects, and those are not, at least with what
I've been exposed to in my classes and some of my work is Yeah,
users are thought of as one way, and we're hoping to think about the
broader stakeholders, the broader sort of people implicated in how systems are deployed and
developed, and along with the broad many more other researchers focusing on that question
specifically. I also think, like when we you know, is this a
user research problem? I mean I think this quite rightly. It's just pointing
out speaks to what is you valued by different sort of roles, different sort
of titles within orcs, Like software engineering has a particular history and a particular
culture from coming from that history of like what is valued as a software engineer,
like making working code, making it faster, making it more efficient,
making it more modular, the various like ways that people have written about this.
But code is good things, and we talked about in the paper house.
I think it was a technical lead was glad that that they were hoping
to soon modularize out the parts where they had to deal with people. We
have sales leads to do that, we have user experience researchers do that,
so sort of a conception of what is within scope for articular role or not
can make it harder to then reassemble the various sort of partial knowledges that are
necessary, the deep technical knowledges that are necessary to understand the technic, you
know, the implications of the technical choice on the wider world, and then
the concerns about the wider world that are then implicated or that have implications for
how we design the technical thing. So that's part of why we frame sort
of the paper as well. If we have dislocated accountabilities, how can we
actually reassemble partial knowledges using the broader frame of the supply chain of the values
and on the point of chat GPT and all that. And we see this
like modularization of concerns. It's not my problem of concerns into the way that
that has been talked about. Like maybe some folks at least listening are familiar
with the sparks of AGI paper, the paper that said that chat GPT is
demonstrating sparks of artificial intelligence and because it can draw unicorn well or something.
But one of the claims they made in the papers that since we don't have
access to the vast full details of GPT for its training data. We have
to assume it's seen every benchmark. So in this sense, it's sort of
saying, well, it's probably seen everything on the web, and so carefully
inspecting, you know, it's data set to make sure that it is appropriate
for use or doesn't have bias. And then, as Don says, disconnected
from any sense of what in particular it will be used for, for whom
it will be used, what problem is it trying to solve. We see
that disavowal as like, the data set is out there, the data is
not our problem. We are building this. I think I think it's a
super valuable point across both of what you're saying that it's easy, I mean,
upon reflection to see where and why some of those disconnects can take place,
right in terms of what is my responsibility, what's the scope of what
I'm producing and making based on my expertise of technical knowledge? You know that's
either a user problem or like an outsource a question of like how will it
be used? You're both approaching this. I think that's really important and challenging
question. That is taking stock of the existing epistemological framework of how developers do
what they do, Like how we make software today, right, we use
modula systems in order to understand them. Taking part at a part. Be
I'm in a barrow from this library. I'm going to hand my box off
to to David. He's gonna put his pieces in there, do his thing
to hand it off the dawn a babbabod. It's going to go on the
chain. At some point, some users are going to try it out.
But the police, example, say it was actually really great, Like when
you say users were actually like the police may be the users of the technology,
but actually the subjects of the data are an entirely different group of people
that have no input or you know, they probably didn't give permission to be
recorded in the ways that they were recorded, right, I mean what happens
with their data? So this is I think, I think one of the
big challenges. And also I appreciate why you kind of present three menu options,
right of like what could we do with this system in terms of do
we just burn it and get rid of modularity? Do we like actually increase
relationships between them? You know, what is the kind of the way the
way of finding our path forward? So I think I'm kind of coming away
from this feeling a lot more optimistic, but at least that the contours of
the problem feel clearer. So asking about that, you know, what do
we need to be paying attention to, you know going forward? What's like
caught your attention because of doing this paper, Like what do you now see
that you didn't see before? You know? I appreciate your pointing out that
part of the job of the researcher is to identify the contours of the problem.
And I will say that in the course of drafting and putting out preprint,
finally putting in proper publication, that just through time, the acknowledgement of
that this is a real issue is coming from a lot more quarters, and
I think it did when we started. So in that sense, on the
optimistic side, that is what I would be looking for, is people really
digging into what are the relationships between different institutions, between different developers. Then
if I had a crystal ball, there might be an elaboration of what developers
already do today. Right, So as an example, we found we found
cases where developers would slow walk dodging work. It would find other projects to
prioritize because there's always something else to prioritize, right, so we might see
more activity activity there. I mean certainly as regulation gets more serious, right,
the game changes again, and that that seems set to happen, so
it's not clear, and then you know, we'll also have to see what
the next flavor of the day is. Also true, yes, then that
is right yet, but it always inflects old issues, right, I mean,
you know we've always had you know, supposed AGI right, like that's
been a long term thing, So it'll surface things that are old instance,
to just to jump down, that's exactly. And this is where it's even
like AI feels new, but it's making us rethink the existing way that we
make software. Right, So it wasn't like oh, this is new how
we make software. It's more like, wait, let's think about why we're
making it the way that we're making it because there's new implications for us.
So I think that that's that's a great that's a great way to think about
that. Since Don and I did this work, I've done some follow up
pieces that ask a question like, you know, okay, so say we
have developers who want to ask the down the supply chain questions who ask like,
well, hang on, I'm uncomfortable that my software is being used down
the chain by the US military. What we did in that in the sort
of follow on work is survey a bunch of software developers with ethical concerns,
and when they try and raise those concerns or they try and act on those
concerns, I mean, they don't really have power to power is modularized itself.
People have power over particular part of the supply chain. So I think
that if I had to go from that to sort of thinking about the future,
we're going to think about different ways that we can build power, either
in part different parts of the supply chain, through collective action, through sort
of organizing efforts. We're going to think about ways that we can build power,
I mean, and part of that will be building power sort of by
connecting different parts of the supply chain, like having more relationships outside of the
chain that allow people to control more of it or to understand at least the
impact of what they're creating down the chain. Now, also, I'm thinking
about again how we conceive of what our job is, how we conceive of
what we're allowed to raise as concerns, and sort of the scope of those
so I'm in some very recent following work is thinking about different ways that we
can give people sort of social permission to think or thinking hypotheticals that don't speak
to what they're currently building, and allow people to talk about concerns or ethical
concerns that they have that in ways that aren't a part of their ordinary day
to day assigned duties or job roles. I mean, if I had to
sort of outline where I think we should go next, it's really starting to
interrogate questions of power and asking whether this is a you know, and a
lot of framings of a ethics, just a question of individual developers building software
better, or is there's more collective action and collective power that needs to be
built, and then also thinking about opportunities for action that we have within our
own armed rules to say and think about things that aren't necessarily within our particular
job mondual job role. And I'll just very briefly add if the future involves
more computer scientists using phrases like partial knowledges in a sentence, I will also
consider that to be a better future, more computer scientists talking about the how
power actually works. Better off we are signed me up to that that sounds
right on, you know, it's like, let's get some meta modules going
aside to the other modules or something. But I want to say Don and
David, thank you so much for joining me on the pot today. This
has been absolutely fascinating conversation and I'm really excited to dive in with our listeners
too and just get a sense of like where folks are at with this,
because I mean, AI is loud as we know, right, it's everywhere
in the news, pulling us away from our desks constantly to go look at
some other random story. And I think that the work that you're doing here
is really important of helping us actually take a moment to shine the light.
Let's actually reflect on the process of how this gets made in the real world
with real developers and ask is that process, this, this epistemological technical process
the best way forward if we're thinking about a broader scope of of you know,
well being in a bigger sense of users, a bigger sense of implications
and ethics and roles. So it's a big order, but y'all squeeze it
into like a sixteen page paper. So that's an excellent start, And excited
to see to see where we can build next. So thanks, thanks so
much for joining me on the on the pot today. Thank you wonderful,
be with you, Adam. And that's a wrap for this episode of This
Anthro Life. We've had a fascinating conversation with our today about responsible AI development
and the role of developers in creating ethical and beneficial software products. I want
to extend a huge thank you to our guest today David gray Witter and don
Nefis. Thanks again for sharing your insights and expertise for this. Now let's
recap our top three takeaways from today's episode. First, mondularity in software development
is a double edged sword. It allows for a more efficient creation and deployment
of technical systems, but potentially limits user experience and data privacy. Second,
user research and design must consider the broader stakeholders and people implicated in the deployment
and development of systems, beyond just the paying customer or knowingly engaged users.
And Third, dislocated accountabilities require the reassembly of partial knowledge using a broader supply
chain or value chain framework. And as always, I want to hear from
you. Do you think it's possible to give developers more power to act on
ethical concerns within their organizations and how could collective action and connection between different parts
of a supply chain play a role in this or at What steps could we
take to ensure that the broader stakeholders and people implicated in the deployment development of
algorithmic systems are considered in the research, design development efforts rather than just paying
customers or known users and as individuals, what actions can we take to ensure
responsible AI development? Can we also work collectively to address these challenges around AI
bias and accountability. Remember, you can get in touch with me over at
this anthrolife dot org on the contact page or reply to emails if you're on
a newsletter list, and if you don't get emails from us, I highly
recommend you subscribe over at this anthro life substack to get newsletters, blogs and
to be up to date with all the happenings in the anthro curious community.
And lastly, I want to ask you, my dear listeners, to help
spread the word about this anthro Life. If you find something of value in
this podcast or this episode, please share it with someone who needs to hear
it. Your support goes a long way and growing our community once again,
thank you for sharing your time and your energy with me and our guests.
I'm your host Adam Gamwell, and you're listening to this anthrow life. We'll
see you next time.