Error Term – Crowdid

The invisible hand in science: How ideological priors shape empirical outcomes

We found evidence that ideology impacts research results. What are the implications of this, and what should we do about it. Spoiler: not the least should we practice more open science!

The Experiment and Our Findings

In empirical social science, we are accustomed to substantial variation in reported outcomes. Research on the minimum wage’s impact on unemployment or immigration’s impact on voting for far-right parties, for example comes to a wide range of results. Metascience identifies several culprits for this dispersion: publication bias, confirmation bias, questionable research practices, and simply noise in the research process. However, one crucial factor has remained difficult to isolate experimentally: the political ideology of the scientists themselves.

In a study published in Science Advances, George J. Borjas and I exploited data from a unique crowdsourced experiment where 71 research teams analyzed the same data and hypothesis. The original study was designed by myself, Eike Mark Rinke, and Alexander Wuttke. The data given to the teams was from the International Social Survey Program spanning 1985-2016 across countries on all continents of the world. The hypothesis was: Does immigration reduce public support for social welfare programs? The teams estimated 1,253 alternative regression models, producing average marginal effects (AMEs) that ranged from strongly negative to strongly positive. Before conducting any data analysis, researchers in the teams answered a survey about their own immigration policy preferences, placed on a 7-point scale from making immigration laws “tougher” or more “relaxed”.

We found that research teams composed of pro-immigration researchers, those with higher scores on the immigration question, estimated more positive impacts of immigration on public support for social programs. In other words, they were more likely to find evidence that immigration supports social cohesion. Conversely, anti-immigration teams estimated more negative impacts, suggesting immigration reduces social cohesion. The raw differences in the teams’ reported AMEs were relatively small, but the differences in the distributions were stark, especially in the tails: anti-immigration teams were significantly more likely to report extreme negative and significant AMEs.

How did these differences arise? Our analysis revealed that research design is endogenous and serves as the primary mechanism through which ideology enters the parameter estimation process. There are no other ways possible as we checked that no results were mistakes or otherwise ‘faked’. Combinations of research design choices along five key dimensions accounted for a large portion of the differences in actual AMEs between the two extreme team types. These critical decisions were: (1) whether to aggregate the ISSP public attitude responses into a single composite dependent variable; (2) whether to measure immigration as a stock or a flow; (3) whether to employ multilevel modeling; (4) whether to use data for all available countries; and (5) whether to include the 2016 ISSP wave or not.

Furthermore, double-blind and random peer review scores of each team’s model specification showed that pro- and anti-immigration teams’ designs received significantly lower referee scores than moderate teams. This suggests that peers in the experiment recognized that highly ideological teams adopted “unconventional” or outlier specifications that were less suited to testing the hypothesis, yet yielded results consistent with their pre-existing ideological priors.

Why This Matters: Ideology as Scientific Uncertainty

These findings have profound implications for the credibility of empirical research. In science, we are trained to treat research as a systematic process of understanding, analyzing, and reducing uncertainty. In quantitative analyses, we carefully report statistical uncertainty, such as standard errors and confidence intervals, and occasionally explore methodological uncertainty through robustness checks. What we have identified, however, is a different form of uncertainty: the ideological priors of the researchers themselves.

If researchers subconsciously or consciously navigate the “garden of forking paths” to find specifications that confirm their ideological priors, science faces a challenge to its reliability and this risks that public trust in science will (further) erode. This concern is magnified in highly polarized policy-relevant areas where scientific consensus routinely collides with political ideology, such as climate change, vaccination, or immigration. In an era where populist rhetoric frequently questions the legitimacy of scientific progress, painting it as self-serving rather than socially useful, documenting that political beliefs bias empirical parameters is deeply concerning beyond the scientific arena.

Fortunately, because we can identify uncertainty arising from ideological bias, we can also design systemic barriers to reduce its impact. First, the adoption of open science practices is critical. Pre-registration of analytical specifications forces researchers to commit to their research design before observing the data, preventing checking many specifications until an appealing narrative emerges. Second, we should foster “adversarial collaborations”, where researchers with differing ideological priors and opinions work together to design and agree upon a shared modeling strategy. By co-authoring the research design in advance, we can systematically buffer against the individual biases that increase rather than reduce scientific uncertainty.

Limits, Rebuttals, and the Crisis of Reproducibility

Of course, our own estimates of ideological bias have limitations that must be transparently addressed. First, the original crowdsourced experiment was not designed to test the specific hypothesis that ideology influences findings; our results are exploratory not confirmatory and should serve as a prior for future confirmatory research. Second, although our estimated ideology effects are statistically significant, the confidence intervals are relatively large, making it difficult to draw definitive conclusions about the precise magnitude of the bias. Third, because very few participating researchers revealed strong anti-immigration sentiments, our analysis is underpowered to examine anti-immigration bias as robustly as pro-immigration bias. Fourth, potential social desirability bias in the initial surveys may have led some researchers to hide their anti-immigration views, contaminating the moderate and pro-immigration groups and expanding the confidence intervals. Fifth, the experimental data do not record the actual workflows or trial-and-error processes of how researchers experimented with and discarded models along the way, in case they did engage in so called “hacking”.

Our study has also sparked productive scientific debate. Katrin Auspurg and Josef Brüderl published a comment suggesting our evidence is fragile and hinges a team-level discipline variable that introduced statistical singularities, effectively excluding cases that happen to contradict our hypothesis. Although this critique has merit, in our original study and in a response to their criticism we demonstrated that our findings are highly robust by running hundreds of alternative models that do not suffer from this singularity problem. Running the analysis at the researcher-model dyad level for example completely removes team-level singularities and still provides evidence in support of an ideological bias effect.

Crucially, this vigorous academic debate was only possible because we adopted rigorous open science practices by sharing all our data and code online. Future vetting is absolutely essential not only for our work, but all scientific activities. The state of computational reproducibility in the social sciences is otherwise alarming. In a massive project published in Nature, Miske et al. (2026) estimated that computational reproducibility rates in some areas of the social and behavioral sciences are pathetic: less than 2% of sampled studies in education sciences including adult education science and less than 7% in sociology share their data and can be computationally reproduced. Scientists must commit to sharing code and data to allow rigorous peer vetting, and academic journals must adopt policies that support this. Currently none of the top journals in education science and sociology require transparent sharing of data and code. But this is a crucial need of science if we want to maintain public trust.

Subjectivity, Incentives, and an Honest Proposal

In reflecting on how to move forward, quantitative researchers working with data can learn a valuable lesson from qualitative methodologies. Ask any qualitative researcher, whether they are setting up a study based on participant observation, ethnography, or ethnomethodology, and they will tell you that subjectivity is not an obstacle to be ignored, but a reality to be managed and integrated. A rigorous qualitative study requires the researcher to actively reveal and integrate their subjectivity into the workflow, reflecting on how their own positioning influences their observations.

Quantitative and data-driven researchers should adopt a similar posture. Rather than clinging to the myth of the perfectly detached, value-free objective observer, we should reveal our subjective positions in advance and plan explicit methodological safeguards to protect our studies against them.

At a very basic level, we must acknowledge that we are all subjectively motivated by trying to pursue a career in scientific research within a highly competitive liberal capitalistic market for science. To secure employment, tenure, and funding, we need high-impact, publishable studies in highly ranked journals. Consequently, we are perversely incentivized by a “publish-or-perish” system to seek out and publish statistically significant, “clean” stories, which are often those that align with our preferred narratives.

If we were to explicitly state these pre-existing ideological priors and career incentives in a standardized “Conflict of Interest” statement, it would be an act of profound intellectual honesty. More importantly, it would serve as a powerful self-motivating mechanism. Acknowledging our subjective interests in black and white would compel us to integrate rigorous, pre-registered methodological buffers to protect the integrity of our empirical findings from our own career and ideological motivations.

Why we humans are not objective: English translation of the German National Radio show ‘Darum sind wir Menschen nicht Objektiv’.

German National Radio (Deutschlandfunk) produced a spot on objectivity and it included parts of an interview with me. The original German audio version is here: “Darum sind wir Menschen nicht Objektiv“, and this is a ChatGPT enhanced translation in English.

Let’s imagine this for a moment: a crowd of people, a real bustle. And everyone has a plank in front of their head—some a small one, others a bigger one. And these planks don’t lead to everyone bumping into each other or failing to find where they want to go. Instead, they lead to people simply not noticing certain things even though they’re there. “A plank in front of your head” is an expression for exactly this: that we humans mainly look at what we already know. Scientifically, that’s very well researched. The problem is: even though it’s so well established, we very often keep running around with our planks in front of our heads—and act as if they weren’t there.

If you look out into the world, we kind of act like everyone has their own way—according to that motto. Or at least everyone acts like it, or doesn’t reflect on it at all. Whether that’s concrete individual facts or broader “truths,” I think it’s reflected on too little.

Aileen Oeberst works on bias research—on exactly those planks in front of our heads and inside our heads. We’ll hear more from her in a moment. Because in these Deutschlandfunk “System Questions,” with me, Kathrin Kühn, we’re going to sort this all out: Why is objectivity so difficult for us humans? The occasion is a current case from academia.

“I don’t have the words for how extraordinary it is that someone who is regarded as a founder of a field at Harvard is recruited by an extreme-right populist presidential candidate. That is really remarkable. And that he worked in the White House last year…”

The person saying that is Michael Clemens, an economist from the United States. And the person he’s talking about is George Borjas, also an economist, who works on questions of when migration can have negative consequences for an economy. He is involved in a study that looked at this: do researchers’ own attitudes influence the results that come out—even in science? To better classify what exactly this study is about, we first need a few basics.

And Aileen Oeberst can provide those now. We already heard her briefly. She is Professor of Social Psychology at the University of Potsdam and has long researched biases in information processing—more precisely, confirmation bias, the “confirmation error.” I asked her: how much do we humans fool ourselves? Meaning: if I assume that what I see, how I evaluate it, is true.

Aileen Oeberst: In many ways. That doesn’t mean people can never be objective. But at the latest when they already have an idea of how things are, what is right in the world, then it’s often the case that we strongly tend to confirm that prior assumption again and again. And that’s what’s called the so-called confirmation error. It’s incredibly widespread. And we find it in very, very, very many contexts.

Kathrin Kühn: And I basically have such prior assumptions since the time when I was already a small child, when I learned the first things.

Aileen Oeberst: Exactly. About all sorts of things—about the world, but also about myself, of course. Am I good? Am I good at that? Is the world a place of dangers or one of possibilities? Etcetera. About everything. About other people. Stereotypes, prejudices, etc. Yes, all of that.

Kathrin Kühn: And if you explain that now a bit from the scientific perspective—how does it work exactly? How can it be bundled? This confirmation bias.

Aileen Oeberst: At its core, it’s basically about this: as soon as we have any prior assumption—or even just a suspicion, it doesn’t have to be a very firm conviction—then we tend to do a lot, so to speak, to confirm this prior assumption again and again. Very simply: it starts, for example, with where I even look for new information about the world. That already depends a lot on which political attitudes I have, for example. Likewise, on social media you might rather follow people with similar attitudes. Or otherwise we typically seek closeness to people who see things similarly.

But it’s not only information search, it’s also, for example, how we evaluate information we hear. A classic example: there was a study in the 1980s about a TV debate between presidential candidates. And it showed that people watched the same TV debate.

But those who were supporters of the Democratic Party clearly saw their candidate as the winner of the debate. And supporters of the Republican Party did the same. That means: they evaluated the same information differently.

Kathrin Kühn: And you’ve also worked on bundling the whole research on confirmation bias. And you arrive at a construct that we humans have “fundamental beliefs”—very basic assessments about ourselves. And then this biased information processing that we engage in as humans attaches itself to that. And together it has the consequence that we don’t really look at the world as it is.

Aileen Oeberst: Exactly. Maybe just one sentence beforehand: we basically proposed that about 17 different biases—each with its own name—are actually just variations of this confirmation error. And that they’re basically due to the fact that we have very basic convictions. We called these “fundamental beliefs,” simply because we wanted to make clear that they’re very basic convictions that probably the majority of people have. For example something like: I see the world correctly. My assessment of the world is correct.

Maybe my opinion is even the only correct one in this world. That would be an extreme form of it. And that this then repeatedly contributes—together with the tendency to confirm this presetting or assumption—that we very often perceive information in a distorted way, make distorted judgments, or that our memories are indeed distorted by our attitudes.

Kathrin Kühn: And does it make a difference whether I approach something rather openly and then these distortions happen to me more or less by chance, because I’m simply how I am? Or whether I have a clear goal in mind because, for example, I find something unjust and would like it to change. Does that make a difference for the goal, for the outcome?

Aileen Oeberst: Actually that’s a very hot debate in science right now. Typically, the view is that motivation—meaning I want some specific outcome—reinforces such distortions even further. So it rather leads to me being even more off. But that doesn’t mean I’m not off when I have no motivation.

A good example: there were studies where it was only about this. One person was asked to find out whether their counterpart is, for example, introverted. The other half of the participants were asked to find out whether their counterpart is extroverted.

The people had no interest at all in what comes out—whether their counterpart is introverted or extroverted. It doesn’t matter. So they had no motivation to achieve any particular result. But it led to—and it showed—that they ask very different questions. If the assumption of introversion is sort of given to them through the task, then they asked questions like: do you like to withdraw? Do you like to read books? Do you like being by yourself? Etcetera. Which of course also gives the other person fewer opportunities to present themselves in their entirety, and that then also shows in the outcome. That means: we find such distortions even without motivation. But if motivation is added, it usually gets worse, so to speak. Then the distortions become stronger.

Aileen Oeberst from the University of Potsdam. So the state of research is quite clear—from psychology but also from neuroscience. And what’s also exciting is how long this has already been a topic: that we humans do not walk through the world neutrally, objectively. Luca Rese-Knauf researched for this “System Questions” episode. We’re now here together in the studio. Luca, how far back does this whole discussion about our biases go?

Luca Rese-Knauf: Very far back. What today is fashionably called “bias” can already be found, for example, two-and-a-half thousand years ago with the philosopher Plato in the dialogues. There are figures who always confirm themselves. And a second old example: in the Bible it says, “Why do you see the speck in your brother’s eye but not the beam in your own?” So: I myself overlook the beam in my eye. That’s what it was called back then, and today it’s also called “a plank in front of your head.”

Kathrin Kühn: Yes, exactly. And your own perspective initially seems real or right. And it’s difficult or costly to question it—to move away from assumptions, even from simple opinions. And if that then hardens or even becomes ideological, meaning quasi immune to objections, into a closed worldview, then it’s of course even harder. So ideology here as the strongest form of bias.

And “biased,” “ideological”—that’s always the others.

Luca Rese-Knauf: Yes, noticing that in others seems very easy. That runs through it. And I also brought a couple of very current examples from politics:

“Ideology unfortunately has no place here. We have no ideology. We have open discussions about the best course.”

“At this point, unfortunately, that is not an ideological exaggeration. That is sober reality.”

“One sees: one could say ideology eats brains. You don’t know what you’re talking about. It is an… idea… chasing after this ideology.”

You can see: as hard as it is to notice something like that in yourself, it seems easy to accuse others of it. And this discussion about ideology has also existed for a long time—since the term was coined, really, at the time of the French Revolution. The “Ideologues” were a theoretical scholarly school more than 200 years ago. And Napoleon then used the term pejoratively against people who wanted to intervene in politics with philosophical ideas or also science.

Kathrin Kühn: And so ideology immediately became a political fighting term.

Luca Rese-Knauf: Which it still is today.

Kathrin Kühn: Yes. And this ambivalent relationship between science on the one hand and politics or ideology on the other runs through. And it makes the case of this current study so exciting. Because in science, striving for objectivity is a particularly high value. The point is precisely to isolate personal attitudes or opinions—and certainly ideology—through methods to achieve more objectivity. That’s the core idea behind scientific methods.

Which is not always easy. Not easy at all.

Luca Rese-Knauf: No. And that’s because research is also a process with many individual decisions. For example: how do I formulate a question, which period do I look at, or how exactly do I analyze data—with which model?

Kathrin Kühn: So a research question can be approached very differently, and then something different can come out.

Luca Rese-Knauf: Exactly. And then the question is: why? What factors in? And there are many studies on this—so-called many-analyst studies. That means: various research teams examine the same relationship or work on the same question with the same data.

And that’s exactly what this new study—also by George Borjas—does, which has triggered these debates. It’s a reanalysis of a study that already appeared in 2022 in the journal PNAS. There, 71 groups with over 150 scientists in parallel answered the same question:

Does immigration change support for welfare-state programs?

Kathrin Kühn: So in the end something like: how does migration influence a society?

Luca Rese-Knauf: Exactly—positively or negatively. That’s a question with a lot of influence, because politics also follows it and looks at such results. And in the first study, the results were very different. But the authors couldn’t explain exactly why.

I spoke with Nate Breznau, the other author of the new study, a sociologist and political scientist from the US. He works at the German Institute for Adult Education in Bonn, and he explained how this “new edition” came about.

Nate Breznau: The concrete research question emerged rather unexpectedly. A colleague, George, contacted me and had some questions about an original study by me and two colleagues. He reanalyzed our data at the time and found that there was a statistical relationship between the research teams’ pre-existing attitudes toward migration and their results.

Luca Rese-Knauf: Yes, and in the data, teams with a positive attitude toward migration more often found positive effects, and migration-critical teams rather negative. And the two, Nate Breznau and George Borjas, have now published this re-evaluation of the data in January in the journal Science Advances.

Kathrin Kühn: And that was then indeed a clear relationship between attitude and result.

Luca Rese-Knauf: Yes—but here comes the “but”: even if it’s labor-intensive to find so many research teams to do this, in the end 71 is still a small sample. Too small a sample to be able to say how large this influence actually is.

That means: the true influence—the effect—could be very small or medium-sized or large or very large. And we can say little about that here.

And one criticism of the study was that the starting question was too imprecise and each team, with its decisions, actually answered a different sub-question. And then, of course, a different answer came out. And the wording was criticized—that these attitudes were equated with ideology.

Kathrin Kühn: By attitudes, you mean now how someone stands on migration?

Luca Rese-Knauf: Exactly. The researchers who took part were asked whether they would rather loosen immigration in their own country or rather restrict it. But that doesn’t have to be ideology. It might also simply be that someone answers pro-migration because the government in their country promotes migration for economic reasons. And this is also criticized by the economist Michael Clemens from George Mason University—the one we heard at the beginning.

Michael Clemens: Ideological bias is not what is being measured here. If this is ideology, then every conviction would be ideological. And that’s not true. Ideology means that one is immune to facts. Convictions can arise from ideology, but also, for example, from scientific insights. That is the internal problem of the text. Beyond that, there is an additional external context which, in Borjas’s case, is very serious and clear.

Luca Rese-Knauf: Yes, and the last point is actually his bigger criticism. And now we’re at George Borjas, the co-author, who, according to Clemens, is himself ideologically biased.

So: it’s a study about how attitudes influence results in migration research. And the study itself could be influenced.

Kathrin Kühn: Yes, that’s what Michael Clemens says. For background: George Borjas is considered a pioneer of migration economics. He holds the position that migration can also have harmful consequences for a national economy. And he was last year an economic adviser in the White House—an office he resigned from in January on his own initiative. He hasn’t really distanced himself from Trump, at least not from what I found. And the Washington Post calls him “the man of Trump’s migration policy.”

Michael Clemens has known him for years from professional debates. And Borjas has also called him an activist or social engineer. So again an ideology accusation.

Michael Clemens: The context makes it very clear that Borjas reads this as an empirical confirmation of what he has long believed: that there is a kind of academic conspiracy against him. And that many scientists, when they have empirical results that differ strongly from his own, deliberately choose methods to produce exactly such results, due to their prior assumptions. But this work does not show that at all.

Kathrin Kühn: But that really is his interpretation. That’s not in the paper—we have to say that. And Clemens also says: the study itself is very well done. But the example shows how tangled the whole thing is, and how sensitive the relationship between science and ideology is in the end. We also contacted George Borjas, but got no answer. And Nate Breznau said this about the criticism:

Nate Breznau: It could be that he is right. It could be that ideology plays a role for George Borjas in his research. I think George and I do not have the same political positions—I don’t know exactly. But precisely for that reason, ideology was an additional reason for us to choose a particularly robust research approach.

Kathrin Kühn: By “robust research approach” he means: they recalculated everything with very different models. And the relationship showed up in around 90 percent of the cases.

And what follows from this now? Does it follow what Aileen Oeberst explained earlier—that one’s own attitudes always have an influence on what one does, how one sees it?

Luca Rese-Knauf: Yes, I think that is a universal phenomenon. And what also must be said: the sciences have means to reduce exactly that—the influence of one’s own attitudes on the path of analysis.

Michael Clemens: That’s one reason why recently there has been a very strong trend in research toward preregistered study designs. That means: before you begin evaluating data, you define exactly what you will do with the data, which variables will go into the statistical analysis, and what the analysis will look like. After that, your hands are bound in a certain way.

Kathrin Kühn: Michael Clemens says that. And Nate Breznau says: much more self-criticism would be possible and necessary in the sciences. That was also one reason why the paper was not accepted by some journals, he said. And, he says, you already learn in training to find “sexy” headlines—and that interesting outlier results can be marketed better than robust, dry results that in the end really are new knowledge. So all these incentives in the system can also already bias the situation.

Pretty similar to us in journalism: punchy headlines, interesting outlier results. And complexity doesn’t sell as well as a strong opinion I can get upset about.

Luca Rese-Knauf: Yes, exactly. And in science there are now already a few journals that promise publication before the results, so that these incentives don’t even arise in the first place. Or that people join forces who come from very different directions, with very different attitudes.

Michael Clemens: Researchers with different prior assumptions work together. And they set the research design in advance and agree on what evidence would support which position. That replaces competition with cooperation and places the gain of knowledge above career interests. And I see that as a very desirable path.

Kathrin Kühn: And in the end, a bias, a distortion can also be: where do you even look? Which questions get how much attention? The spirit of the times plays a role. And there are regularly formats like “The Forgotten News,” also here with us. And one could ask the same about research: what gets too little or too much attention?

A huge chunk. But in the end something where you might recognize the beam—or the plank—on your head after all. Luca, thanks for the research—not only on the Borjas case, but on this case: all of us with the plank, the beam in front of our heads, I’ll say.

Luca Rese-Knauf: Yes, thanks. Gladly. It was exciting.

Kathrin Kühn: And what to do about it—science, but also journalism? There are approaches. We’ve already heard what can help when it comes to you personally—when I say: I want to get closer to what the situation really is. I asked Aileen Oeberst that too. And she answered this:

Aileen Oeberst: As I said, the research would most strongly suggest that we should keep challenging ourselves again and again. And not in a pseudo way, but seriously. If I really have the desire to see things more objectively and not so distorted through my own personal lens, then it would be good to repeatedly ask myself consciously: to what extent could it be that I’m wrong here?

If you take a concrete example— a political topic on which one forms an opinion—then it would be good to consciously expose oneself to counterarguments.

But not to simply devalue them and say: you see it wrong anyway and I see it right. Rather to take it seriously, to truly challenge oneself and to look: does that really speak? Does everything really speak for my view of things? Or what actually speaks against it—and does that perhaps change something about my view of things?

Kathrin Kühn: But sometimes that’s pretty difficult. If I think back to the pandemic: I thought about certain points. But then very harsh judgments were made about groups. Then I also simultaneously detached myself from the group I perhaps identify with if I start thinking like that. That’s already exhausting.

Aileen Oeberst: Absolutely. Of course it is. And that contributes to exactly this confirmation bias and to why it is so widespread—because it has a lot of obstacles. Maybe I won’t be liked then in my group. Or maybe I even lose that group. Then I’m not socially so well embedded anymore; I have to see.

But also, this questioning of one’s own conviction—this is typically something people don’t like so much. Things like ambivalence, or for example, “so how is it now?” many people can’t tolerate very well. They want to keep things clear for themselves. And that, too, contributes to the fact that one perhaps rather does not question oneself, or does not sit down and think it through. And it’s also effortful. That’s also clear. We all have a full life, that’s understandable.

But if one had the goal of seeing things more objectively, then that would still be the recommendation.

Kathrin Kühn: What do I get out of it, if I see things more objectively?

Aileen Oeberst: Well: I am perhaps less influenced by something that I was either taught already or that I’m convinced of—something that maybe isn’t even true. And I think group beliefs are a good example. The belief that one’s own group is somehow better than other groups—if one systematically questions that, and if many would do that, that could clearly lead to very positive consequences. Namely, for example, to significantly fewer conflicts between groups.

Perhaps even on a large scale to fewer wars between groups—between nations, for example—because the belief isn’t so strong: we have the only correct perspective, we are better than the others, the others are the guilty ones, the evil ones, etc. I’m exaggerating a bit, but that’s the core.

Kathrin Kühn: But sometimes it can also be that you then become more exhausting for others at first. For example, I’m in science journalism and I notice: the more I deal with certain topics, the less I can, as a private person, say: yes, I have the following opinion. Instead I end up more often saying: oh, I used to think it was like this and now I don’t know. And that’s of course an ultimate extender for conversations.

Aileen Oeberst: Absolutely. I understand that and I see it exactly the same way. The more you deal scientifically with a topic, the more you always see: things are never that simple, never that clear, never that banal. It’s always differentiated. It’s always complex. You always have to consider: yes, it is like this if—and but sometimes it is like this. But at the same time it may really just be an illusion to believe that things are simple. And also that a stance on a topic can be banal and simple.

Surely there are topics where it can be quite clear. But often it’s more differentiated, isn’t it? And maybe that would help us—for example, if we take a more differentiated perspective—to understand counterpositions and other opinions better, or at least be able to accept them better.

Kathrin Kühn: Helping yourself more, challenging yourself—that’s the recommendation from Aileen Oeberst. And one can also say: the more often I do that, and the more people do it, the more normal the whole thing becomes. Which then creates new questions that end up scratching at how we, as a society, deal with information.

Science in survival mode

In this post I argue that unethical scientific behaviors like faking data are symptoms of triggered survival response. Scientists are exposed to repeated stimuli conditioning them to fear that without sexy results and ever increasing status, they will be rejected if not ejected from science.

Scientific research is unreliable. It comes with uncertainty. Whether launching a rocket or measuring racial prejudice, there is uncertainty. We use this uncertainty to make decisions. If the rocket has a 40% chance of exploding, best not to stick astronauts in it. If skin-tone bias of soccer referees is somewhere from none to a lot, it is irresponsible to conclude they are prejudiced.

Investigating uncertainty, is science. Truth and uncertainty are two sides of the same coin, they co-define each other. A problem for humans measuring uncertainty, is that humans are unreliable. Human scientists themselves add uncertainty to the measurement of uncertainty.

Recent studies suggest that somewhere between 25 and 60% of published statistical results cannot be recreated using the materials provided – these measures of uncertainty come with their own uncertainty. It turns out that scientific researchers are doing some really peculiar things to generate uncertainty. Some surveys suggest as many as 9% of scientists faked data at least once in their careers, and that more than half selectively reported findings – a behavior that makes the things they study to appear less uncertain than they actually are.

Why would scientists fake research?

I believe the answer lies in their humanness. Like all animals, they are genetically programmed for survival. They are capable of both rational, reflective decision-making and split second reaction without any thought. Given time to reflect, a human would generally conclude murder is unethical, but simultaneously would not hesitate to kill if it prevented their child from being killed. Murder remains wrong, but to not kill and let a child die is also wrong. It would be irresponsible parenting failing to ensure survival.

Murder is a profound act when a human perceives themselves to be in a situation of life or death. It is a symptom of subconsciously activated defense mechanisms in survival mode. But there are many other symptoms. In order to avoid death, humans, like other primates can engage in deception, disassociation, aggression, manipulation, submission, scapegoating, theft and hoarding. If someone held me at gun point and told me to prove the earth was flat, I would have no problem doing it. The math would work, I would just need to fake a little data

Why would scientist act as if their lives were threatened?

Most scientists have highly valued knowledge and competencies and thus live in situations where their lives are not under threat. At least not as a result of their scientific practice. For the sake of this thought experiment, lets just rule out regularly occurring life-threatening danger as a cause of scientists exhibiting survival mode behaviors. This leaves the perception that they are under threat, as a possible explanation.

It takes only a few stimuli to induce survival mode behavioral changes in animals. Like hearing a frightful noise when seeing an animal. This animal and anything that looks like it become automatic sources of anxiety and fear even without the sound. Imagine being told over and over and over that you have to have an exciting study with powerful results in order to get an academic job after graduate school, otherwise you wasted 3-8 years of your life and probably a large chunk of capital on getting a PhD. Could this alone, without introducing any actual shocks or physical pain, induce Pavlovian fear? I encourage you to go ask any graduate student to answer this that does not yet have such a study.

Now imagine that during graduate school a student invests all their time and resources into an experiment. After it is complete, they hypothesized result, the one sure to be exciting and publishable, is not there. Imagine the horror, the shame, the feeling of failure, the panic. Remember the poor soul who leaped to his death because he misunderstood futures trading and thought he owed three-quarters of a million dollars? That was a triggered survival response. Because dying felt like the only way to ‘survive’ the horror of facing that debt. Imagine if he could have just changed the futures market by adding his own numbers to the market. Would he have done it?

We should not be surprised at all then, when scientists acting out of fear-based survival strategies, fake data. Diederik Stapel faked an entire career of data before being caught. His behavior was self-described as an “addiction”. A common reaction of individuals placed under fear stimuli that are emotionally damaging if not traumatic. The intense pressures and expectations of the academic environment created a context in which he felt compelled to engage in unethical practices to maintain his status and success. It does not make murder or data-faking right, but to not take this as grounds for indicting the scientific rewards system is certainly wrong.

The incentive structures have to change before we can honestly expect the fear-driven pressure to fake, cheat, lie or steal – in order to avoid the experience of loss associated with the common null results that occur when conducting high quality scientific research on radically complex human brains and societies that are frustratingly difficult to measure things in – to go away.

Love is in the error term

A major segment of social science uses formal models and quantitative methods to explain social phenomena. That means they use mathematical symbols to define how they think the world works, and then find data and test it. For example, researchers want to explain social outcomes, like committing a crime, changing jobs, moving or having children. Why do some people do these things and not others? Researchers then speculate that other things cause these outcomes like getting married, having a job or losing a job, how much money a person makes, how old they are and all kinds of other stuff.

Next, researchers change their theoretical ideas into a formal model. This means an equation. Before you stop reading, maybe pause to consider that an equation is just a theory expressed with symbols instead of standard language. For example,

Y = X

and this might represent moving homes (“Y“) is caused by getting married (“X“). Another way of saying this is, Y depends on X or moving depends on getting married.

But seriously, getting married does not always cause moving. Sometimes it does. The correct claim is that moving is a function of getting married. Function means that marriage probably increases the likelihood of moving by some amount. Therefore, researchers add a modifier to X, like the letter b, so if b was 0.3 getting married would increase the likelihood of moving by 30%. People also move when they don’t get married, so researchers add a constant a to account for the likelihood of a person moving for any other reason. So if a was 0.3 then the average person would have a 30% chance of moving at any point.

Y = a + bX

Now if a and bX could be used to perfectly predict whether a person moves or not. The researcher would have made a monumental achievement here. Instead what happens is the researcher goes and observes moving and marriage behaviors. Tries to do this with a random sample of the population of a society. Then applies the above equation to the sample data. Does the equation fit perfectly to the data? No. Never. Researchers must admit that their theories come with uncertainty. Maybe moving depends on the type of home the person always lives in, maybe it depends on whether a couple can afford to move or if it is even a couple or a single parent. There are nearly unlimited things that could cause moving. Also, getting a perfectly random sample, mistakes when observing or coding data, etc. leads to uncertainty in the results. This means the formal equation has to have e, an error term.

Y = a + bX + e

Researchers spend their time trying to reduce what is in the error term. So they are basically tasked with the job of reducing uncertainty. If scientists wants to fly a spaceship to mars they have to reduce all possible forms of uncertainty in the take off, flight path, landing, electronics and so forth. But a social scientist who wants to predict how many people will move, where they will move or which people will move, cannot reduce uncertainty that much. Sure, they can explain moving with job changes, family members getting sick, weather, hobbies, wars, and so many observable things. But they will never fully explain moving. What I’m really saying here is that social scientists will never be able to perfectly explain human behaviors. Its like this:

An Enlightened Physicist describes Sociology

Why is it so hard? If we could just measure all the things that people are doing and thinking we could predict precisely what humans will do next, right? Maybe so, maybe not; but irrelevant because we cannot measure everything about humans. Even if we could measure the precise actions of the network of neurons in one human brain numbering greater than all the stars in the universe, we probably couldn’t measure love. But something that we might refer to as love seems to be a major force in guiding human actions. Both the effect of love:

“Love affects more than our thinking and our behavior toward those we love. It transforms our entire life”
– Thomas Merton

And the lack of love:

“Intense spiritual and emotional lack in our lives is the perfect breeding ground for material greed and overconsumption.”
– bell hooks
(both quotes from hooks 2000, pp. 187 & 105)

These suggest love is a prime mover of human decisions and behaviors. But we don’t know what it is exactly and we have made no attempts, that I am aware of, in any large population surveys to measure it. So,

When we add love as an X variable to our formal model we cannot test anything, because we have not developed instruments to measure it. So it remains in the void of all that variance in Y that we simply cannot explain. Maybe we should start focus some of our effort on that.

hooks, bell. 2000. All About Love: New Visions. William Morrow and Company, New York, NY. ISBN 0-688-16844-2.