The following is a conversation between Jevin West, Co-author of Calling Bullshit: The Art of Skepticism in a Data-Driven World, and Denver Frederick, the host of The Business of Giving.
Denver: Misinformation, disinformation, and fake news abound, and it’s increasingly difficult to know what’s true and who to believe. What we could all use is a field guide and, thankfully, my next guest has provided us with one. He is Jevin West, Director of the Center for an Informed Public at the University of Washington, and Co-author of Calling Bullshit: The Art of Skepticism in a Data-Driven World.
Welcome to the Business of Giving, Jevin!
Jevin: Thanks for having me, Denver. I’m excited about the conversation.
Denver: “Calling Bullshit.” Now, that’s an outgrowth from a very popular undergraduate course that you teach at the University of Washington. Where did you and your co-author Carl Bergstrom ever come up with this idea?
Jevin: This happened far before 2016 when the term “fake news” really became common vernacular. We have worked together for many, many years. We exchanged emails of things we find in our professional world, things that we’ve read — or things even in our personal life — that just don’t sound right, and those kept piling up higher and higher. And we talked about creating a course like this. I was at the time teaching some of the earliest data science classes at the University of Washington. I helped design some of the initial ones, and it was during this big data craze that’s still here. And Carl says, “I want to do a course on calling bullshit on big data, ” and I said, “You’re right! We need to do that.”
So we got together and started putting our heads together on what this would mean, and it took us some time actually to develop. We took it very seriously — the content we developed, from the definition to the way that we deliver it in class. It’s probably the nearest and dearest class to us. So we take it very seriously and had a lot of fun with the students so far over the last several years.
We think that “bullshit” involves, of course, language and rhetoric. But it also involves statistical figures, data graphics, and other forms of presentation intended to persuade by impressing and overwhelming a reader or listener a lot of times with a blatant disregard for truth and logical coherence.
Denver: Let me stop you right there. What is the definition — the act of calling bullshit?
Jevin: When you make definitions in the economic world, you spend a lot of time on each word. You remove them and put them back, and it’s still evolving. But the way we define it is we think that bullshit involves, of course, language and rhetoric. But it also involves statistical figures, data graphics, and other forms of presentation intended to persuade by impressing and — here’s the key — overwhelming a reader or listener a lot of times with a blatant disregard for truth and logical coherence.
So bullshiters are different from liars. Liars know the truth and they are just pulling you away from the truth. Bullshiters? They don’t really care. They just want to impress you, get your attention, and they don’t really care necessarily about the truth or not. It can come up in many different forms, and we focus a lot on the data forms in which it comes.
Denver: You are teaching this to students at the University of Washington, and they are digital natives. How sophisticated are young people in detecting this bullshit?
Jevin: It’s an interesting question. We could have the entire podcast on that particular point because there are things that they are clearly better at than me in many ways. They’ve grown up in these digital environments. They know the kinds of things to look forward to there. They know what clickbait is. They know strange HTML formats to say “That site, it doesn’t look well.” But they still have a lot to learn when it comes to critical reasoning.
And so, I can actually compare them to the groups, like I’m working, for example, with the AARP on becoming better fact-checkers. Now, they don’t have quite the digital skills that the younger generation has, but they’re picking them up quite fast actually now. But what they do have is experience in context. So when they go about critical reasoning, they have a set of tools that maybe the younger generation doesn’t have.
I’ll give you an example of something that the younger generation really does struggle on. So, they claim to know what correlation causation is. So they’ll nod their head when we say, “Correlation doesn’t imply causation.” They’re not okay with it. “Can we move on to something else?” And then we say, “Let’s test you.” We do some tests and boy, do they struggle. And it’s not just them; we all struggle. But I would say it’s a problem from K1 to K99
Everyone can become better critical thinkers, especially in this new information world in which we live. We just need to integrate it in everything we do in education and certainly outside in the public as well.
Denver: Well, if we all do struggle with it — and you are absolutely right, we do — tell us the difference between correlation and causation.
Jevin: That’s another one we could talk about. We have an entire chapter devoted to that.
Denver: I know.
Jevin: Correlation causation — it’s so difficult because as humans, we look for patterns. It’s been an important part of our survival as humans. We look for patterns, and if we see, at one point in the day, a herd of animals coming into a field and we need that for food… well, we will return with that pattern. There seems to be some causative link.
But what happens when we see a graph on one axis that shows the number of people in various countries and another axis with a number of storks in the country. We might as humans — and we do this — we might then conclude that storks deliver babies because there’s a strong correlation: As you have more people in a country, there’s more storks. Of course, that’s a ridiculous example and it’s one that people certainly can recognize quickly, but that kind of thing happens all the time.
And so, what we try to do is we try to explain, first of all, how hard it is to really get at causation. And by having that appreciation, you’re a little more cautious before jumping to causation problems. But also because we in academia and the media, we make these mistakes all the time. We’ll hear of a new paper that’s published in science about some positive effect of wine now on our hearts or whatever, and then we jump to prescriptive headlines and things that really insinuate causation.
So, it’s really hard, partly because that’s how we’re built; so we have to overcome those through different kinds of exercises.
… the person that we need to be most careful about and look out for when it comes to bullshit creation is ourselves.
Denver: Jevin, BS is all around us, and it bombards us on a daily basis, an hourly basis, but how much do we BS ourselves?
Jevin: Oh, my goodness! This gets to one of my favorite bullshit laws and principles. It was from a sociologist by the name of Neil Postman. He was writing about this in the ’60s and actually talking about some of the problems of media at the time and its role in creating all this bullshit. And it’s one of my favorites — it’s his third law — and I don’t have him verbatim in my head, but the idea is that the person that we need to be most careful about and look out for when it comes to bullshit creation is ourselves.
And so, one of the goals that Carl and I talk about with our students is: we should all have the goal of being bullshit-neutral. We all talk about being carbon-neutral; we need to be bullshit-neutral. And that’s the point of the book, really, is to say, “We’re going to create it. We’re humans. We all bullshit to some extent, but we need to be spotting it and refuting it, too, and doing it with civics in mind and empathy in mind.”
We don’t just go out and call bullshit on anyone or on anything we might see. We have to be very careful about what we’re doing and to attack the argument rather than the person and various other things that we talk about. But I really, really think that it’s ourselves, it’s our confirmation bias that gets in the way. It’s our need for wanting to see things that fit our narrative of the world. So we have to be watching out for that, and that comes as much as it is in watching others as well.
What we have found in tracking these things at very large scale, at the billions of tweets scale, we find that the misleading information — misinformation that works its way out — sometimes it’s refuted, but the refutations almost never catch up with the lie itself.
Denver: I don’t know whether this is true or not, but is there a bit of a laziness in not wanting to go through that critical thinking? And I say that in the context that speed beats critical thinking. So if I see something that’s pretty sensational, and I know it’s going to get out there, I want to get it to all my friends first before checking it out or otherwise, it becomes old news. And people send the headlines on things that make a point. They don’t even read, correct? So I just was wondering about that balance between critical thinking and being first.
Jevin: You hit a bunch of really important points there, Denver. So first of all, is it lazy? Sometimes it is lazy, I admit to that fully, and partly, it’s because we have so much information coming at us all. We can’t refute everything, but what we can do is not necessarily spread it. But that’s easy to say in a world where there are strong incentives to be the first one out, even if you’re not even a journalist. But getting the first one out gives you social influence, and so we tend to push things out sooner than we should, and we push out things that really do capture people’s attention.
What we have found in tracking these things at very large scale, at the billions of tweets scale, we find that the misleading information — misinformation that works its way out — sometimes it’s refuted, but the refutations almost never catch up with the lie itself. It’s this sort of quote where the lie is taking off, and the truth is still just trying to pull his breeches on while he’s running after the truth that’s going around the world.
So I think — it’s Brandolini’s Asymmetry Bullshit Principle, which is one of my favorites, too — that it takes orders of magnitude more effort to clean up BS than it is to put it out. So we just need to be careful about not putting out as much, and that can be as simple as pausing a little bit more before we share. I know we really want to be the first one so we’d get that extra like and follower from it, but we’re the ones causing a lot of the spreading that’s causing a lot of this.
Denver: And just looking at the old, old news model of newspapers, the big scandal is on A1 and then the retraction is on C18, and no one ever sees it.
Jevin: Isn’t that… I love that! I hadn’t heard it described that way. But that’s exactly what it was in that world, in the subscription-based models, and that’s how it is now but even more extreme, I would say. You’re right. The scandal, you’re right, is in front to sell the newspaper, which is what we’re doing now. They put the headline to grab you in, get your eyeballs on there so you’ll pay… essentially those ads pay for the newspaper, and then the retraction maybe later on, or the corrections from really good fact-checkers and other journalists certainly get buried. So I love that description. It’s really what it was, even in the olden days.
Denver: Let’s pick up on that clickbait. That’s what you just described, I guess. Algorithms and the rest of it — how is that working and how is that keeping us glued to that platform?
Jevin: It all comes down to a very simple objective of these big tech platforms, and they won’t hide it either. They admit this fully. The one objective — its one objective — to keep our eyeballs on the screen because the longer we stay on this screen, the more likelihood you’re going to click on one of those ads.
And so, if that’s the objective, and you’re allowed to let all these algorithms run, run billions and gazillions of AB experiments every single day, those algorithms know more about human psychology than the most amazing human psychologists out there. They’ve peered in. They know what words attract us, what colors, what designs, what images. And so by pulling us in , they can get that click. We see this all the time. You go to CNN or Fox or any of these major news stations, or anywhere around the web; you can tell anyone that’s been on the internet for some time what clickbait is, but we still fall for it. We still fall for it.
Denver: But we still fall for it.
Jevin: They wouldn’t be putting it out if it didn’t work. It works, unfortunately. And that objective of sticking to the platform doesn’t necessarily have positive effects on society. And so that’s one area that we’re trying to think about in our centers. If you were to optimize something else on the social media platforms, what would it be?
Denver: The thing that scares me about that, too, is that now that they’ve been able to customize, they know me better than I know me. And I’m not too cool with that, but they really do.
Jevin: They absolutely do. I teach classes in machine learning where we talk about designing recommendation algorithms and how effective they can be at peering into you. The data broker industry that exists even outside, it’s kind of in the interstitial tissue of the big tech companies that moves from on-the-net credit card transactions, phone transactions, your sensors in your house, your car… You’re right. These algorithms know more about us than we do ourselves. It anticipates what we want to buy and anticipates what we’re going to click. And we live in this world. I don’t know that it’s going to change anytime soon.
There’s been some regulation in the European Union like the GDPR, which is starting to change things a little bit, at least pushing back a little bit on that invasiveness. But here’s the way I look at it: It’s here, and they know more about us, so we have to be even more vigilant as users. And sometimes just turning off these things and not being, I call it “tired drunk on the internet.” When you get so tired, then you’re more vulnerable, too. We’re all that because there’s just so much stuff coming in all the time.
Denver: You’re absolutely right. Let me just ask you about what I refer to as “military-grade fake news,” and that’s when that fake news is all wrapped up in data and in graphics and statistical analysis. Just let me know how much more open we are to receiving that information is true when it has all that wrapping.
Jevin: I love your description. I’ve never heard that description. Military-grade. I think that’s a great description of what is out there. Sometimes it’s done by honest mistake, but a lot of times, these are from sophisticated individuals, organizations that know that data carries a weight that words and rhetoric don’t carry.
And if I can give you a statistic or a number, like 2,139 DACA recipients have been convicted of crime — that sounds scary. That number is big. It’s weighty. It sounds precise, objective. Sometimes the number is wrong, but a lot of times, that number could even be right, and you can still carry an argument that’s misleading. And the only way that we can overcome that is to be able to ask questions of the data and graphics and statistics in the same way we ask questions of a car dealership or a car salesman or something that’s trying to sell us a car. You start asking, “Well, how does this car compare to others? Or ” Why is this car, the odometer down to zero. This looks like it’s from 1970 or something.”
These are the kinds of things that we want to be asking of data. We want to compare apples to apples. We want to see where there might be cherry picking. We want to see where there might be misrepresentations in percentages. We want to look at whether the axes have been manipulated. These are things that anyone can do. Actually, I’ve been teaching it to my 9-year-old and 6-year-old. Anyone can do it, but we need to create that habit of mind, and I don’t think the education system is doing enough to teach data reasoning the same way we teach other subjects. I think we need to integrate it in everything we do and so that we can fight back to this military-grade, misinformation disinformation.
Denver: I have a hunch your 9-year-old and 6-year-old are teaching you a lot of stuff, too.
Jevin: They’re calling BS on me all the time. Just like with my students, when my kids call BS on me, I absolutely love it! I want to create a culture where that’s OK because we all make mistakes. And there are some places where you just feel sometimes offended; we don’t want that. We want to do it to make us smarter. If we do it in a good way, it will make us all smarter.
Denver: Talk about science, the way it’s understood, the way it’s reported, and the role it plays in us being so skeptical. Don’t wear a mask; wear a mask. I think you mentioned before: red wine. That’s a classic. Good for my heart, bad for my heart, odd and even weeks. How does it all go? What don’t we understand about how science works and how it’s reported?
Jevin: I think it’s so critical that we do better in science in communicating the limits of science — what it can say and what it can’t say — and that it should be updating. It’s got a process set up that is meant to do self-correction, that is meant for us to update when new data comes in.
And so, this idea that mask, no mask — that can be confusing to the public, and it can reduce some of that trust. They can say, “Well, wait a minute. Two months ago, you were telling me “no mask.” Now, you’re saying masks.” And in that case, I think there really was a flood, both maybe on the science because the science hadn’t gotten in far enough, and also communication there, too. That was a pure communication issue.
But science in general, I would say it makes me more nervous actually when a scientist won’t change their opinion. You would hope that there is changing as new data comes in, and right now, in this pandemic, science has been politicized to a degree I’ve never seen. I’ve been studying what we call the “science of science” for many years, and the fact that you see this almost arbitrary split on certain topics by political parties just baffles me.
I can understand if before the pandemic, you said, “Predict, Jevin, whether Democrats or Republicans will be wanting to shut the economy down or expand the welfare state in response to the pandemic.” You could make predictions on how the different parties would react. But would you be able to have predicted the positions then on masks and hydroxychloroquine or all these other things that are not split by the science, but it’s split by party? And that to me is concerning.
But what we can do in science is do a better job of explaining how it works and why it is so effective and has created some of these great discoveries that we all depend on. And so, I think it’s really our fault really in science. We need to do a better job of explaining to the public, and I think we can use the media’s help in doing that as well and making sure that they don’t make those same mistakes.
But those are points that I could see the public sometimes having some issues, but fortunately, I will say this, despite some of the issues and this pandemic, science is one of the few institutions that still remains relatively high on the trust factor. A lot of other institutions have been going down, which is concerning to me.
Denver: Absolutely. I do wish that the media would invest a little bit more in science journalism.
Jevin: Yes!
Denver: They all seem to be gig workers. Scientists had never been that great at communicating things to begin with, and you need somebody to take that information and transfer it and translate it to people. Also, don’t make every one of these studies the final word. It’s just a process. It builds on each other, but they do it so definitively, which I think gets you back to your clickbait again. Like this is the answer. Nobody wants to know that this is just a trend or an approximation. They want a finale.
Jevin: That’s right. Can you imagine a headline that’s like “This was the 445th study that showed wine is good, but there are 538 that said…” The headline is “Wine Helps You.” So you got to drink to that…
Denver: Absolutely! Let’s drink tonight.
Jevin: Yes. They don’t know that science is sort of the crumbs being put together, and that is how science works. It’s not that science was wrong in those studies. Some of them maybe were better than others, but that’s how science is on most topics. It’s a handful of things that are this and a handful of that. We need to be careful when communicating that it is not the definitive answer, it’s one of many. We need to be able to show them the broader context when talking about it.
Denver: You’ve done a magnificent job in identifying so many of the problems. So let’s turn the tables here. How should an individual approach a piece of information so that they’re aware and that they’re looking at it critically?
Jevin: I think we need to be in a state of mind that allows us to do that critically, too. And with all the distractions on our devices and all the information we’re getting, we first have to put our self in a state of mind, one, to not just be ready to share and like or whatever reactionary behavior that the platforms want us to do.
But I think if it’s something you read on the internet and it looks like a piece of news, you could then start searching around and doing some triangulation and corroboration. You could check the source of the site by just putting — it’s a great trick. You take the URL of the site. So let’s say “crazynews.org,” and then space… Wiki or Wikipedia, it will take you most often to the Wikipedia, which is one of these rays of light.
I can’t believe I can actually say Wikipedia is one of the rays of light. But it works! It’s more reliable than most of the internet. You can do things like do reverse image checks if you’ve seen strange images. You can just ask basic questions: Who’s telling me this? What did they have to gain from it? And what makes them an expert? And then there are things like if it sounds too good to be true, it probably is. There’s just a whole set of little things that we can do, and those things you can do in less than a minute.
Denver: Getting back to Wiki, if you’d asked me 25 years ago that I could invest a thousand dollars in Wiki or the Encyclopedia Britannica, all my money is going to the Encyclopedia Britannica.
Jevin: Same!
Denver: Who would have ever guessed that Britannica would be dead, and Wiki would be thriving. You’re absolutely right. And you can count on it.
Jevin: Thriving! I wouldn’t have put a dollar. I would have put–you said a thousand? I would have put a thousand on Britannica…That this idea that it’s crowdsourced and actually educators, high school teachers still tell students, “Oh, you can’t really trust Wikipedia,” but they’re starting to come around and going, “Well, you certainly can trust it a lot more than Facebook.” So that is just bizarre to me, but at least it is a ray of hope.
And they built it because there’s a great, strong editorial army of fact-checkers out there that love what they’re doing. They’re free digital volunteers. They built a system that is robust, so these crazy things going on… it’s not perfect. You still get issues. There’s been huge editorial wars around misinformation around COVID, and I have some colleagues that study this online. But given all those things, I would tell students that it is a place to start. Not always a perfect place, but it is more reliable than many of the places on the internet.
Denver: And it’s certainly more comprehensive. So, look, I’m going to look at certain things that are going to reinforce my beliefs. What can I do to try to minimize my own confirmation bias?
Jevin: I think we need to have friends as much as we can that see things differently. They have their own filter. And so, do what we can to broaden both our friends in this space and to utilize those friends and to do that with real empathy and sincerity, I think. Of course, just broadening and diversifying what we read. They don’t need to be on the fringes. I don’t claim to…don’t do that sort of things. But also, just being aware of the fact that we do have this confirmation bias. We all have biases, and sometimes those biases are not always a bad thing. They’re built into us sometimes for good. Many times, it can be detrimental to how we filter information.
But I think just being aware that there tends to be a narrative, certainly in the hyperpartisan news world in which we live. And like you, Denver, I take this nonpartisan approach very seriously. So when we talk for example, about, in our class, a political example, we always try to balance it from left and right because it is from the left and right. And then as students sort of… if we force students, for example, to play in different sandboxes and on different teams, and then they have to create these narratives, they begin to be more vigilant themselves.
So I think just broadening both our actual friends and then also what we’re reading… it seems and sounds very simplistic, but it’s important that we do that as much as we can and look for some of these great projects that are starting to try to present both sides, like AllSides and other projects out there are doing , at least moving in the right direction.
Denver: Talk a little bit more about that because I think that’s what everybody’s looking for at the moment. The news model, I think, is broken, and the news model has become a locker room. People want to be in their locker room hearing their team — and essentially, it’s their team, not what they believe in — it’s more “We got to kill the other guy,” just like what you would do before a football game. Those people are evil. It’s not that they think differently than us. They have no morality. They’re evil, and we’re going to go out there, and we’re going to crush them.
The news model is broken. Unfortunately, we’ve become broken because that’s what people go to, and the advertisers go to. So where do you go for that objective information? You talked about AllSides. Any place else? And Wiki, probably, too.
Jevin: I’ll just real quick echo what you said so well, that we’re a reflection of the media, too. And so, one question we could ask is: This current hyperpartisan environment, is it causing this polarization? Or is the polarization and other things causing the hyperpartisan media? So we’re feeding into that. And it’s this interesting game-theoretic deadlock where maybe one of them could start to become more representative, but then the other team might think they’re getting an advantage.
Anyway, so there’s all sorts of interesting game theoretic dynamics there, but it is broken. And AllSides I just mentioned just sort of briefly… I don’t want to give them too much attention because I still am evaluating their approach. But I do think at this point, we need to go to more reliable local and national. One of the things that I would love to see over the next 10 years is for society to reinvent local journalism. Part of it’s coming through podcasts like yourself, and that to me is one way of pushing back at these news deserts that are occurring across the country.
To me, it’s very concerning to see many — and I wouldn’t say not even many — most local media around the country have just gone extinct, and they’re getting gobbled up by national conglomerates. Many times, people are writing stuff for these locals, but they don’t even live in the United States. It’s not local anymore. We need local journalism that would help sort of balance some of this hyperpartisanship in the United States and other parts of the country.
But what we can do as individuals is support that, support your local journalism and support other forms of journalism. And then just have this balanced diet. You know, sometimes I’ll go straight to the source instead of being opinionated. You can go to the AP directly, although the AP, some people will say things. It’s really to also look at news agencies outside. Of course, BBC and others that report on the same things might have a slightly different take. I think it’s that balance that helps.
And then, try to engage in content that sometimes… that maybe you wouldn’t… as long as it’s not offensive. I want to be careful when I say, “Don’t go to the fringes.” We’ve seen too many people… because this is what’s interesting about trust is that it doesn’t just go away; it just shifts. So when people say, “I don’t trust the national media.” So then they might move too far on one side or the other end of the fringes, and that’s not a good thing either. So, it’s a complicated thing, and we’ve got to fix this broken system that you’re noting.
Denver: I love your point on local. I don’t know if you’ve seen the work that The GroundTruth Project is doing, but they have Report To America, and they’re trying to fund a lot, a lot of local reporters. The funny thing is there’s absolutely no trust in this country between the left and right, but there’s actually a lot of trust at the local level. And if you’re going to build trust up again, don’t do it in Washington. Do it in your community and solve some problems, and it’s going to be a bottoms-up building of trust and not a top-down.
Jevin: I completely agree. I always, I almost overuse this, but if you gave me the one magic wand wish, it would be to really reinvigorate local journalism because the trust indicators that I’ve looked at, the polls and surveys that look at trust, almost across the board, people trust local journalists more, and that’s where you are seeing some bipartisan efforts. That’s where you see… like we have a politician in our state, Kim Wyman who’s on the Republican side, but is respected on both sides, and she’s involved in a lot of these discussions around voter integrity. It’s those things that are making progress.
National level, at this point, I don’t… I’m pretty depressed about what I’m seeing just like anyone else, and it’s going to take some serious time and some serious leadership to come in and try to deal with that. But in the meantime, let’s invigorate and refund and reinvent local journalism and support some of these projects like GroundTruth and others, because I’m so passionate about figuring out how to do this local journalism thing again.
Denver: Let me close with this, Jevin. If you were the manager of a bookstore, in what section would you put your book, and why?
Jevin: My publisher’s not going to like this answer, but I would put it in self-help. I’ll tell you why I’d put it in self-help. No one would put it in there. Amazon puts it in linguistic reference and some people put it in the political section. Some people…
I really think of this as an empowering book because right now, we’re all overwhelmed with the information that’s coming our way, and really — Carl and I — our goal was really to empower people to become better consumers of information in general, and specifically data information.
Denver: But you’re the manager of the bookstore now, so you can put it where you want.
Jevin: I can put it wherever I want it. My publisher probably wants it more in the data reasoning and somewhere near like a Freakonomics book or whatever it is. But you know what? I really think of this as an empowering book because right now, we’re all overwhelmed with the information that’s coming our way, and really — Carl and I — our goal was really to empower people to become better consumers of information in general, and specifically data information.
You don’t need a PhD or advanced degree in statistics or computer science to be able to call out BS, the majority of it. It’s very rarely in what we call the “black box” — this is how the algorithm works or the statistical procedure, how the data is collected. It’s just in looking at things like selection bias, and correlation causation, and manipulation of graphical malfeasance and things like that. This is really… it’s an empowerment book.
So if I was the book manager, I’d put it right in self-help, and I’d say, “If you want to become better information discerners and feel really strong about your ability, read our book.”
Denver: Well, you’re not going to be a bookstore manager because you write too well, and you teach too well. The book is Calling Bullshit: The Art of Skepticism in a Data-Driven World. We’re up against a lot of it incoming these days, and this book is the ultimate bullshit detector, and readers of it will never look at information in quite the same way again. It was a real pleasure to have you on the show, Jevin.
Jevin: Thank you, Denver. This was a lot of fun. Happy to chat anytime, and good luck with all the good work you’re doing as well.
Denver: Thank you.
Listen to more The Business of Giving episodes for free here. Subscribe to our podcast channel on Spotify to get notified of new episodes. You can also follow us on Twitter, Instagram, and on Facebook.