Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Not Wrong: The Replication Crisis is a Metatheory Crisis?

What happens when you fill a conference room in Barcelona with social and behavioral scientists, a few policymakers and science administrators, and I lead a discussion about metatheoretical multiverse analysis with a 90-second timer? That’s what we found out at a recent Institute for Replication (I4R) workshop where I tried to flip the script from analytical robustness, coding discrepancies and replication packages, the usual I4R stuff, to something theoretically driving the reproducibility crisis that we often ignore.

Wanna know what that is? Its theory.

We talk about reproducibility as a methodological problem. We debate p-hacking, publication bias, and analytical forking paths. But the elephant just chillin in the corner is that our theories are weak. There are too many other plausible explanations for what we observe and measure. Sure, the things we measure are complex, but without more clarity, even Commisioner Odenthal isn’t gonna find us a clue to what we are testing. Meaning that we don’t really know what to make of our findings. Other than hoping they are publishable and packaging them neatly to increase those chances. The findings are not right or wrong, they are not really interpretable. At least not with weak theory.

Part of the problem is that unlike the ‘harder’ sciences, consciousness is in the equation. We are measuring phenomena far more complex than quantum physics. Another part is simply that we do not spend much time on theory. In one of my favorite esoteric and less-mainstream open science writings, Anne Scheel pointed out “Why Most Psychological Research Findings Are Not Even Wrong”.  Anne, I see your discipline specific arguments and raise you all social and behavioral science disciplines. We often don’t know what the estimand is. We don’t know what we are looking at. If our theories don’t explain the phenomenon in the first place, debating whether an empirical finding is statistically “right” or “wrong” is moot.

To confront this, I pitched a method I’m working on with my former doctoral student turned postdoc Hung H.V. Nguyen. It’s called Metatheoretical Multiverse Analysis (MMA) and I wanna kick some ‘theoric’ with it (cool huh? It rhymes with lyric and could mean theory in action…). What follows is a summary of the ensuing debate, full of interdisciplinary friction, economic curmudgeonism, and philosophy of science moments.

The Pitch: Metatheoretical Multiverse Analysis (MMA)

To understand why metatheoretical multiverse analysis is necessary, we have to look at how we currently treat plausible theoretical arguments about the data-generating model… Say what? I mean how we use and apply logic.

Imagine you are testing the effect of X1 on Y. But there is an unobserved confounder, X2, which causes both X1 and Y. If X2 is not in your test, your results are uninterpretable. You don’t know if X2 is confounding the results. And, don’t even get me started about X3. This might be a collider. But the theories are not clear in your subfield. So when we try to compare them we end up with several conflicts or unknown paths. These are all alternatively plausible theories and from them we have a multiverse of theory.

Now, imagine I have five equally plausible alternative theories explaining the same phenomenon. This means there’s a 20% chance the theory I am using is the correct one. But I cannot imagine any social or behavioral science where there are only five plausible theories. There are likely thousands. We only test five because it takes entire careers to write semantic theory. And since it takes 20-40 years for most of our discipline to forget those long-winded theories of the past maybe this is a sinking ship…. I digress. Let me re-gress instead (fancied opposite of digress which happens to be a statistically procedure too). 

This is where Metatheoretical Multiverse Analysis (MMA) comes punching (or wrestling or kicking) in. If a standard multiverse analysis runs all reasonable empirical specifications for a given dataset, a metatheoretical multiverse analysis would logically compare all these theories. Somehow…. That’s where the idea gets a little sticky. Metatheory is mostly something people write about, rather than try to formalize and analyze with math. But if they could, our method would then show them where they need to invest their theory-building efforts.

I opened the floor. The timer started.

The Economist’s Dilemma: HARKing and the Illusion of Theory

The pushback was immediate, insightful, and brutally honest. The first counterargument highlighted just how difficult theoretical work is. As one applied economist noted, reading James Heckman makes you realize how brilliant deep theory can be, but spending your career trying to come up with sufficient conditions to definitively disentangle two competing theories is a great way to find yourself out of academia before you get tenure. Remember the long-winded argument?

But a more damning critique came from the reality of how theory is actually utilized in modern economics. To commit a “statistical sin” and assign causality, one researcher pointed out that the lack of robustness in our fields might stem from the fact that HARKing (Hypothesizing After Results are Known) is practically the norm.

In economics, the workflow rarely starts with a pristine a priori theory. Instead, a researcher finds an intriguing empirical pattern in the data, and then builds a formal mathematical model to justify the findings. We only take the time to do the exhausting math if we already have a paper we want to publish. Often, the formal model is something requested by Reviewer 2 at a top journal ex-post. Because the theory is engineered to fit the data, it doesn’t improve our prior hypothesis in a meaningful way, nor does it guarantee robustness when exposed to new data.

Interestingly, someone from the I4R team chimed in with data to back this up. After checking roughly 15,000 robustness checks in the I4R database, they ran an AI classification to score papers from 0 to 100 on ‘how economic’ they were (i.e., true economic theory vs. a paper on TV habits published in an econ journal). The finding? There was absolutely no relationship between the “econ-ness” of the paper (the presence of formal modeling) and its robustness.

So, the means that even if we were to invest in theory development, we wouldn’t get anywhere because… basically, we suck at it.

The AI Revolution

These days there’s always gotta be something about AI. If not everything. The egomaniac AI that I asked to help me write this just couldn’t wait to point out how many people were talking about it.

Funnily we came to an AI discussion through a strong defense of the structural approach in economics (after the initial dust had settled and we could again breathe the cool Barcelona air-conditioned summer air). Economics, one participant noted, used to be purely theoretical because, prior to the 1970s, we simply didn’t have the capacity to handle large data. The empirical revolution brought an era of data-mining into play. P-hacking came into full swing. The academic industrial complex had taken over thanks to secondary data spewing forth from society.

But the tide now is actually flowing back toward structural modeling. Thanks to… AI? The massive amounts of data we have today, combined with AI’s unparalleled ability to explore patterns, is killing the human comparative advantage in pure empirical data mining. AI will always be better at finding patterns (at least an AI or data scientist tells us this, Gemini still can’t perform better regressions than I can IMO). The only place human researchers will retain a comparative advantage is in thinking, being creative, and structuring the data generating process. We don’t need one perfect theory to describe reality anymore (not that that ever worked for us) good approximations are good enough now – and good approximations means…. Drum roll and someone on the mic saying “Ya’ll ready for this?!”. PREDICTION. The better we can predict things, the less we will need to explain them, they will just become some sort of facts in our life worlds. Won’t they?

Metatheoretical analysis might just be the structural framework we need to survive the AI transition. I mean, I invented it, so I’d like to think so.

Bias, DAGs, and Kung-Fu Nancy Cartwright

As the 90-second buzzer kept interrupting and resetting, the conversation evolved into the philosophy and sociology of science itself.

From a legal and equality perspective, an important question was raised about those “five theories” we tend to rely on (you know that when equally plausible reduce the chances that any one is correct down to 20%). Where do they come from? Historically, they have been generated by a very specific demographic—predominantly white men from the Global North. By restricting our empirical tests to a handful of established theories, we inadvertently perpetuate biases and ignore alternative paradigms that might emerge from the Global South. A metatheoretical multiverse approach, by automatically generating and considering thousands of models, might offer a mechanical antidote to this historical bias by forcing us to acknowledge the vast space of un-theorized realities.

But are DAGs really capable of saving us? The room had its doubts. Hey Mister Jack… I’m talking to you. The world, as one researcher passionately argued, is not a clean DAG with X1, X2, and Y. It has X15, Y7562, and a zillion unobserved mediators. Furthermore, DAGs can’t handle cyclical relationships and feedback loops the most famous relationship in economics, price and quantity, is entirely cyclical, endogenous.

This brought us to Nancy Cartwright and the philosophy of science. If we are mapping out thousands of theories, we must remember the Popperian ideal: a model must be falsifiable. If a theoretical model cannot be thrown out under certain conditions, it ceases to be a model and becomes a religion. Metatheoretical multiverse analysis is only useful if we have the empirical tools to actually falsify the branches of the multiverse we generate.

The crow cheers with Nancy’s MMA kick to the face.

The Preregistration Battleground

You cannot talk about theory and open science without stumbling into the debate on preregistration. I posed the question: ‘Does preregistration inappropriately constrain our theories?’ because I want to seem smart and provocative. If there are hundreds of theories, forcing a researcher to write down a specific one beforehand essentially chokes the multiverse before it can breathe. Choking is forbidden in MMA by the way.

The responses were polarized:

Some argued that preregistration forces a hypothetico-deductive model onto fields that generate knowledge inductively. In economic history or sociology, research is exploratory. You learn from the data. Preregistration actively harms inductive discovery.

Others pointed out that preregistration is incredibly difficult if not overrated for secondary data analysis. Datasets are messy, collected for non-research purposes, and require deep exploration just to understand how missing variables are coded. Recently someone pointed out on LinkedIn that my praise of Neumark is overrated. An MMA sweep kick, totally permissible in the sport – touché.

Conversely, a third group (there’s always a third group otherwise it feels incomplete) advocated that preregistration is simply a record of where you started. It prevents the ex-post invention of stories and increases transparency. If ‘Mostly Harmless Econometrics’ had a chapter telling students to pre-specify their hypotheses before opening the dataset, it would have transformed the culture of economics entirely. To late, the historical institutionalists won.

Where Do We Go From Here?

As we wrapped up the session (and prepared to face the blistering heat outside in the name of finding Fideuà), the consensus was clear: our methodological tools have far outpaced our theoretical foundations. We have built incredibly sophisticated empirical engines, but we are putting them in theoretical chassis that are fundamentally flawed. I liked this shift. Oh wait I kinda led the discussion there.

Metatheoretical multiverse analysis is not a magic bullet. It will not solve the fact that social science involves the unpredictable chaos of human consciousness, nor will it easily map the cyclical, non-DAG-friendly feedback loops of the global economy. But it is a start. It is a way to stop pretending that our opportunistic, post-hoc theories are the only valid models of reality. It might trigger some epistemological soul searching if nothing else. By mapping the vast space of plausible theories and systematically testing where they align and where they conflict, we can begin to rebuild the credibility of our disciplines from the ground up – in theory (which is probably weak, so take it with salt).

Furthermore, as the discussion highlighted, we need to completely overhaul how we incentivize work. Open science has an image problem we often make it look boring, framing it as an adversarial compliance checklist rather than a thrilling pursuit of truth. We leave it up to early-career researchers to awkwardly teach their supervisors about open code and data. And shoulder them with the onus of spending hours making all their work open and reproducible, hours that their forefathers (yeah, they were 95% men) didn’t need to do. Heck these forefathers could publish like 2 papers per year and get tenure.

But since its my blog post, I get the last word: If we want to fix the replication crisis, we cannot just mandate better code and policies. We have to foster better theory. We need to acknowledge the sheer size of the theoretical multiverse, embrace the complexity, and start theorizing before regressionizing. In this case, sadly, the answer is not as simple as 42.

The invisible hand in science: How ideological priors shape empirical outcomes

We found evidence that ideology impacts research results. What are the implications of this, and what should we do about it. Spoiler: not the least should we practice more open science!

The Experiment and Our Findings

In empirical social science, we are accustomed to substantial variation in reported outcomes. Research on the minimum wage’s impact on unemployment or immigration’s impact on voting for far-right parties, for example comes to a wide range of results. Metascience identifies several culprits for this dispersion: publication bias, confirmation bias, questionable research practices, and simply noise in the research process. However, one crucial factor has remained difficult to isolate experimentally: the political ideology of the scientists themselves.

In a study published in Science Advances, George J. Borjas and I exploited data from a unique crowdsourced experiment where 71 research teams analyzed the same data and hypothesis. The original study was designed by myself, Eike Mark Rinke, and Alexander Wuttke. The data given to the teams was from the International Social Survey Program spanning 1985-2016 across countries on all continents of the world. The hypothesis was: Does immigration reduce public support for social welfare programs?  The teams estimated 1,253 alternative regression models, producing average marginal effects (AMEs) that ranged from strongly negative to strongly positive. Before conducting any data analysis, researchers in the teams answered a survey about their own immigration policy preferences, placed on a 7-point scale from making immigration laws “tougher” or more “relaxed”.

We found that research teams composed of pro-immigration researchers, those with higher scores on the immigration question, estimated more positive impacts of immigration on public support for social programs. In other words, they were more likely to find evidence that immigration supports social cohesion. Conversely, anti-immigration teams estimated more negative impacts, suggesting immigration reduces social cohesion. The raw differences in the teams’ reported AMEs were relatively small, but the differences in the distributions were stark, especially in the tails: anti-immigration teams were significantly more likely to report extreme negative and significant AMEs.

How did these differences arise? Our analysis revealed that research design is endogenous and serves as the primary mechanism through which ideology enters the parameter estimation process. There are no other ways possible as we checked that no results were mistakes or otherwise ‘faked’. Combinations of research design choices along five key dimensions accounted for a large portion of the differences in actual AMEs between the two extreme team types. These critical decisions were: (1) whether to aggregate the ISSP public attitude responses into a single composite dependent variable; (2) whether to measure immigration as a stock or a flow; (3) whether to employ multilevel modeling; (4) whether to use data for all available countries; and (5) whether to include the 2016 ISSP wave or not.

Furthermore, double-blind and random peer review scores of each team’s model specification showed that pro- and anti-immigration teams’ designs received significantly lower referee scores than moderate teams. This suggests that peers in the experiment recognized that highly ideological teams adopted “unconventional” or outlier specifications that were less suited to testing the hypothesis, yet yielded results consistent with their pre-existing ideological priors.

Why This Matters: Ideology as Scientific Uncertainty

These findings have profound implications for the credibility of empirical research. In science, we are trained to treat research as a systematic process of understanding, analyzing, and reducing uncertainty. In quantitative analyses, we carefully report statistical uncertainty, such as standard errors and confidence intervals, and occasionally explore methodological uncertainty through robustness checks. What we have identified, however, is a different form of uncertainty: the ideological priors of the researchers themselves.

If researchers subconsciously or consciously navigate the “garden of forking paths” to find specifications that confirm their ideological priors, science faces a challenge to its reliability and this risks that public trust in science will (further) erode. This concern is magnified in highly polarized policy-relevant areas where scientific consensus routinely collides with political ideology, such as climate change, vaccination, or immigration. In an era where populist rhetoric frequently questions the legitimacy of scientific progress, painting it as self-serving rather than socially useful, documenting that political beliefs bias empirical parameters is deeply concerning beyond the scientific arena.

Fortunately, because we can identify uncertainty arising from ideological bias, we can also design systemic barriers to reduce its impact. First, the adoption of open science practices is critical. Pre-registration of analytical specifications forces researchers to commit to their research design before observing the data, preventing checking many specifications until an appealing narrative emerges. Second, we should foster “adversarial collaborations”, where researchers with differing ideological priors and opinions work together to design and agree upon a shared modeling strategy. By co-authoring the research design in advance, we can systematically buffer against the individual biases that increase rather than reduce scientific uncertainty.

Limits, Rebuttals, and the Crisis of Reproducibility

Of course, our own estimates of ideological bias have limitations that must be transparently addressed. First, the original crowdsourced experiment was not designed to test the specific hypothesis that ideology influences findings; our results are exploratory not confirmatory and should serve as a prior for future confirmatory research. Second, although our estimated ideology effects are statistically significant, the confidence intervals are relatively large, making it difficult to draw definitive conclusions about the precise magnitude of the bias. Third, because very few participating researchers revealed strong anti-immigration sentiments, our analysis is underpowered to examine anti-immigration bias as robustly as pro-immigration bias. Fourth, potential social desirability bias in the initial surveys may have led some researchers to hide their anti-immigration views, contaminating the moderate and pro-immigration groups and expanding the confidence intervals. Fifth, the experimental data do not record the actual workflows or trial-and-error processes of how researchers experimented with and discarded models along the way, in case they did engage in so called “hacking”.

Our study has also sparked productive scientific debate. Katrin Auspurg and Josef Brüderl published a comment suggesting our evidence is fragile and hinges a team-level discipline variable that introduced statistical singularities, effectively excluding cases that happen to contradict our hypothesis. Although this critique has merit, in our original study and in a response to their criticism we demonstrated that our findings are highly robust by running hundreds of alternative models that do not suffer from this singularity problem. Running the analysis at the researcher-model dyad level for example completely removes team-level singularities and still provides evidence in support of an ideological bias effect.

Crucially, this vigorous academic debate was only possible because we adopted rigorous open science practices by sharing all our data and code online. Future vetting is absolutely essential not only for our work, but all scientific activities. The state of computational reproducibility in the social sciences is otherwise alarming. In a massive project published in Nature, Miske et al. (2026) estimated that computational reproducibility rates in some areas of the social and behavioral sciences are pathetic: less than 2% of sampled studies in education sciences including adult education science and less than 7% in sociology share their data and can be computationally reproduced. Scientists must commit to sharing code and data to allow rigorous peer vetting, and academic journals must adopt policies that support this. Currently none of the top journals in education science and sociology require transparent sharing of data and code. But this is a crucial need of science if we want to maintain public trust.

Subjectivity, Incentives, and an Honest Proposal

In reflecting on how to move forward, quantitative researchers working with data can learn a valuable lesson from qualitative methodologies. Ask any qualitative researcher, whether they are setting up a study based on participant observation, ethnography, or ethnomethodology, and they will tell you that subjectivity is not an obstacle to be ignored, but a reality to be managed and integrated. A rigorous qualitative study requires the researcher to actively reveal and integrate their subjectivity into the workflow, reflecting on how their own positioning influences their observations.

Quantitative and data-driven researchers should adopt a similar posture. Rather than clinging to the myth of the perfectly detached, value-free objective observer, we should reveal our subjective positions in advance and plan explicit methodological safeguards to protect our studies against them.

At a very basic level, we must acknowledge that we are all subjectively motivated by trying to pursue a career in scientific research within a highly competitive liberal capitalistic market for science. To secure employment, tenure, and funding, we need high-impact, publishable studies in highly ranked journals. Consequently, we are perversely incentivized by a “publish-or-perish” system to seek out and publish statistically significant, “clean” stories, which are often those that align with our preferred narratives.

If we were to explicitly state these pre-existing ideological priors and career incentives in a standardized “Conflict of Interest” statement, it would be an act of profound intellectual honesty. More importantly, it would serve as a powerful self-motivating mechanism. Acknowledging our subjective interests in black and white would compel us to integrate rigorous, pre-registered methodological buffers to protect the integrity of our empirical findings from our own career and ideological motivations.

What’s pop-music-in’? Thoughts on Post Malone’s “White Iverson”

Recently, I found myself listening to Post Malone’s album Stoney on repeat, captivated by one track in particular: “White Iverson.” Not the usual topic here, but it triggered something. This is my hot take on a catchy song that engages my sociological imagination.

Pop music serves as a fascinating window into society and culture; it tells about what is going through people’s minds statistically. Millions of people have listened to this song. While “White Iverson” sounds like, and for all practical purposes is, just another catchy pop tune. I found a deeper experience in it.

The Brand of Post Malone

The background is already interesting. Post Malone is racialized white. Yet, at the beginning of his career and on the cover of Stoney, he presented himself in a way that intentionally played with this perception. With braids and altered lighting, the album cover made him look almost black. It is ambiguous, and at first glance it appears he may be ‘light-skinned’.

I assume this was intentional – to engage racial stereotypes, trigger curiosity, and thus sell albums and ultimately himself as a product. To promote his brand. The music on Stoney is relatively simple, melodic sing-rap, which makes his later transition into becoming a massive country music singer interesting, if not uncomfortable. Rap and country music are not always compatible when it comes to racial and social group dynamics. Perhaps this is a form of healing. Perhaps cultural appropriation as a method to become a country star. Maybe just creativity or better yet creative marketing.

Being a Baller

The lyrics of “White Iverson” are creative but in the end deal heavily with standard pop-rap themes: having a massive ego, proving your greatness to others, and experiencing “rags to riches”. Post Malone is embodying the legendary basketball player Allen Iverson. He is the “White Iverson,” and that makes him a badass – at least in his own estimation.

The song manages builds a framework around basketball. He references competitive sport intensity with “Double OT,” Michael Jordan’s shoes (“the two threes”), and other basketball greats. In the mythos of urban culture and rap, a “baller” is someone who dominates the competition. It means you can dunk on people, juke them out, and sink the winning shot while they fail to stop you. Post Malone utilizes aggressive, alpha-masculine language to frame his musical career as a sport, boasting about how he has a “high average” and is “balling on these bastards.” He even pushes the narrative that he doesn’t need to practice hard, suggesting it is just “magic” so “fuck practice.” The concept of an effortless alpha domination is biologically and socially attractive. Some might argue however, that it is an immature and fear-based spewing of toxic masculinity – the sort that hinders social progress and understanding across social groups.

Bringing in Race

You cannot separate “White Iverson” from the concept of race. By calling himself “White Iverson,” Post Malone explicitly reminds us that the original Allen Iverson is Black, and that his Blackness and streetball style are part of what made him an icon. Rap as a genre stems from the African American experience and evolved out of an oppressive slave system that categorized people to enforce inequality. Rap is generally racialized as a Black genre of music.

From the perspective of race, one must recognize that Post Malone is using elements of Black culture to make a lot of money. There is a layer of exploitation. Like Eminem in an earlier era of rap, Post Malone’s success is partly driven by an audience of young white males who find a sense of racial satisfaction in seeing a white artist succeed in a Black musical genre. He is successfully leveraging his whiteness while borrowing Black cultural expressions to sell records, a dynamic that relies on the remnants of systemic racial disadvantages still present in modern society. In this case, the disadvantage is instead against those racialized as White. Like as in ‘white men can’t jump’ (play basketball as well) or ‘white people have no rhythm’ (can’t make jazz music) and certainly ‘can’t rap’.

The Tragic Beauty Hidden in the Music

Given the boastful, alpha-male lyrics, one might expect hard-hitting beats and a rough vocal delivery. Instead, the song is slow, featuring smooth vibrato and vibrating bass hits optimized for modern car stereos and bass-thumping headphones. The piano notes lingering in the background evoke a quiet sadness; they do not build tension, but rather wander downward, feeling almost neutral or part of building up only to be totally let down in life.

Because of this musical framing, a tragedy emerges in the mix of sounds, rap-singing and lyrics. When he says, “Balling on these bastards, it makes me happy… it’s tragic,” it carries a double meaning. Is it tragic for his defeated competitors, or is it tragic that this endless pursuit of domination is the peak of his (our) existence? The chorus reveals cracks in the baller facade. He sings with a mixed ego and sadness that “When I started balling I was young.” It is as if this alpha pressure early in life was exhausting or stressful, or maybe forced him to abandon what are otherwise healthy child social behaviors.

He also admits to or at least flexes that he is “spending all my fucking pay,” highlighting the hollow reality that maintaining the baller image burns through resources for their own superficial sake. Most poignantly, he sings, “I need that money like the ring I never won.” Despite his incredible talent, the real Allen Iverson never won an NBA championship title. Post Malone links this failure to his own fears of being forgotten, singing, “You won’t think about me when I’m gone.” It captures the fleeting nature of baller success and the tragedy of human existence when fueled only by the desire to dominate. Or is is a tragic beauty a la his song?

For Me Personally

I grew up watching basketball when Allen Iverson was at his peak. Art is experiential, and “White Iverson” artistically brings forth the memory of Iverson dribbling past defenders with pure swagger. The track completely captivated me because it holds these massive contradictions together in one space: the aggressive alpha nature of most rap and the rap genre in general, the complexities of race and racial roles, and a melancholic tragedy about the limits of success. It turned out to be an unexpectedly profound social text for me.

Appendix: Lyrics and Link to Song

Double OT
I’m a new three

I’m Saucin’, I’m saucin’, I’m saucin’ on you
I’m swaggin’, I’m swaggin’, I’m swaggin’, oh-ooh (Swaggin’)
I’m ballin’, I’m ballin’, Iverson on you (Swish, ooh, ayy)
Watch out, watch out, watch out, yeah
That’s my shot, that’s my shot, that’s my shot, yeah
I’m Spendin’, I’m spendin’ all my fuckin’ pay

I got me some braids and I got me some hoes
Started rockin’ the sleeve, I can’t ball with no Joes
You know how I do it, Concords on my toes
(This shit is hard) Oh
I ain’t rich yet, but you know I ain’t broke, ah
So if I see it, I like it, buy that from the store, ah (Store, ah)
I’m with some white girls and they love them the coca (Coca)
Like they OT
Double OT like I’m KG, smokin’ OG (Smokin’ OG)
And you know me, in my 2-3s and my gold teeth (And my gold teeth)
Bitch, I’m smiling, bet you see me from the nosebleeds (Nosebleeds)
I’m the new three and I change out to my new 3s (To my new 3s)

White Iverson
When I started ballin’, I was young
You gon’ think about me when I’m gone
I need that money like the ring I never won, I won

Ooh, Stoney
Cigarettes and a headband
Commas, commas in my head, man
Slumped over like a dead man
Red and black, ’bout my bread, man
I’m the answer, never question
Lace up, learn a lesson
Bitch, I’m saucin’ (Wow), I do this often, don’t do no talkin’ (No)
My options right when I walk in, jump all them Jordans (Ooh)
I’m ballin’, money jumpin’
Like I’m Davis from New Orleans
Or bitch, I’m Harden, I don’t miss nothin’
Fuck practice, this shit just happens, know y’all can’t stand it (Ayy)
I have it, I never pass it, I work my magic
High average, ball on these bastards, it makes me happy
It’s tragic, I make it happen, and all y’all Shaqtin’

White Iverson
When I started ballin’, I was young
You gon’ think about me when I’m gone
I need that money like the ring I never won, I won

Why we humans are not objective: English translation of the German National Radio show ‘Darum sind wir Menschen nicht Objektiv’.

German National Radio (Deutschlandfunk) produced a spot on objectivity and it included parts of an interview with me. The original German audio version is here: “Darum sind wir Menschen nicht Objektiv“, and this is a ChatGPT enhanced translation in English.

Let’s imagine this for a moment: a crowd of people, a real bustle. And everyone has a plank in front of their head—some a small one, others a bigger one. And these planks don’t lead to everyone bumping into each other or failing to find where they want to go. Instead, they lead to people simply not noticing certain things even though they’re there. “A plank in front of your head” is an expression for exactly this: that we humans mainly look at what we already know. Scientifically, that’s very well researched. The problem is: even though it’s so well established, we very often keep running around with our planks in front of our heads—and act as if they weren’t there.

If you look out into the world, we kind of act like everyone has their own way—according to that motto. Or at least everyone acts like it, or doesn’t reflect on it at all. Whether that’s concrete individual facts or broader “truths,” I think it’s reflected on too little.

Aileen Oeberst works on bias research—on exactly those planks in front of our heads and inside our heads. We’ll hear more from her in a moment. Because in these Deutschlandfunk “System Questions,” with me, Kathrin Kühn, we’re going to sort this all out: Why is objectivity so difficult for us humans? The occasion is a current case from academia.

“I don’t have the words for how extraordinary it is that someone who is regarded as a founder of a field at Harvard is recruited by an extreme-right populist presidential candidate. That is really remarkable. And that he worked in the White House last year…”

The person saying that is Michael Clemens, an economist from the United States. And the person he’s talking about is George Borjas, also an economist, who works on questions of when migration can have negative consequences for an economy. He is involved in a study that looked at this: do researchers’ own attitudes influence the results that come out—even in science? To better classify what exactly this study is about, we first need a few basics.

And Aileen Oeberst can provide those now. We already heard her briefly. She is Professor of Social Psychology at the University of Potsdam and has long researched biases in information processing—more precisely, confirmation bias, the “confirmation error.” I asked her: how much do we humans fool ourselves? Meaning: if I assume that what I see, how I evaluate it, is true.

Aileen Oeberst: In many ways. That doesn’t mean people can never be objective. But at the latest when they already have an idea of how things are, what is right in the world, then it’s often the case that we strongly tend to confirm that prior assumption again and again. And that’s what’s called the so-called confirmation error. It’s incredibly widespread. And we find it in very, very, very many contexts.

Kathrin Kühn: And I basically have such prior assumptions since the time when I was already a small child, when I learned the first things.

Aileen Oeberst: Exactly. About all sorts of things—about the world, but also about myself, of course. Am I good? Am I good at that? Is the world a place of dangers or one of possibilities? Etcetera. About everything. About other people. Stereotypes, prejudices, etc. Yes, all of that.

Kathrin Kühn: And if you explain that now a bit from the scientific perspective—how does it work exactly? How can it be bundled? This confirmation bias.

Aileen Oeberst: At its core, it’s basically about this: as soon as we have any prior assumption—or even just a suspicion, it doesn’t have to be a very firm conviction—then we tend to do a lot, so to speak, to confirm this prior assumption again and again. Very simply: it starts, for example, with where I even look for new information about the world. That already depends a lot on which political attitudes I have, for example. Likewise, on social media you might rather follow people with similar attitudes. Or otherwise we typically seek closeness to people who see things similarly.

But it’s not only information search, it’s also, for example, how we evaluate information we hear. A classic example: there was a study in the 1980s about a TV debate between presidential candidates. And it showed that people watched the same TV debate.

But those who were supporters of the Democratic Party clearly saw their candidate as the winner of the debate. And supporters of the Republican Party did the same. That means: they evaluated the same information differently.

Kathrin Kühn: And you’ve also worked on bundling the whole research on confirmation bias. And you arrive at a construct that we humans have “fundamental beliefs”—very basic assessments about ourselves. And then this biased information processing that we engage in as humans attaches itself to that. And together it has the consequence that we don’t really look at the world as it is.

Aileen Oeberst: Exactly. Maybe just one sentence beforehand: we basically proposed that about 17 different biases—each with its own name—are actually just variations of this confirmation error. And that they’re basically due to the fact that we have very basic convictions. We called these “fundamental beliefs,” simply because we wanted to make clear that they’re very basic convictions that probably the majority of people have. For example something like: I see the world correctly. My assessment of the world is correct.

Maybe my opinion is even the only correct one in this world. That would be an extreme form of it. And that this then repeatedly contributes—together with the tendency to confirm this presetting or assumption—that we very often perceive information in a distorted way, make distorted judgments, or that our memories are indeed distorted by our attitudes.

Kathrin Kühn: And does it make a difference whether I approach something rather openly and then these distortions happen to me more or less by chance, because I’m simply how I am? Or whether I have a clear goal in mind because, for example, I find something unjust and would like it to change. Does that make a difference for the goal, for the outcome?

Aileen Oeberst: Actually that’s a very hot debate in science right now. Typically, the view is that motivation—meaning I want some specific outcome—reinforces such distortions even further. So it rather leads to me being even more off. But that doesn’t mean I’m not off when I have no motivation.

A good example: there were studies where it was only about this. One person was asked to find out whether their counterpart is, for example, introverted. The other half of the participants were asked to find out whether their counterpart is extroverted.

The people had no interest at all in what comes out—whether their counterpart is introverted or extroverted. It doesn’t matter. So they had no motivation to achieve any particular result. But it led to—and it showed—that they ask very different questions. If the assumption of introversion is sort of given to them through the task, then they asked questions like: do you like to withdraw? Do you like to read books? Do you like being by yourself? Etcetera. Which of course also gives the other person fewer opportunities to present themselves in their entirety, and that then also shows in the outcome. That means: we find such distortions even without motivation. But if motivation is added, it usually gets worse, so to speak. Then the distortions become stronger.

Aileen Oeberst from the University of Potsdam. So the state of research is quite clear—from psychology but also from neuroscience. And what’s also exciting is how long this has already been a topic: that we humans do not walk through the world neutrally, objectively. Luca Rese-Knauf researched for this “System Questions” episode. We’re now here together in the studio. Luca, how far back does this whole discussion about our biases go?

Luca Rese-Knauf: Very far back. What today is fashionably called “bias” can already be found, for example, two-and-a-half thousand years ago with the philosopher Plato in the dialogues. There are figures who always confirm themselves. And a second old example: in the Bible it says, “Why do you see the speck in your brother’s eye but not the beam in your own?” So: I myself overlook the beam in my eye. That’s what it was called back then, and today it’s also called “a plank in front of your head.”

Kathrin Kühn: Yes, exactly. And your own perspective initially seems real or right. And it’s difficult or costly to question it—to move away from assumptions, even from simple opinions. And if that then hardens or even becomes ideological, meaning quasi immune to objections, into a closed worldview, then it’s of course even harder. So ideology here as the strongest form of bias.

And “biased,” “ideological”—that’s always the others.

Luca Rese-Knauf: Yes, noticing that in others seems very easy. That runs through it. And I also brought a couple of very current examples from politics:

“Ideology unfortunately has no place here. We have no ideology. We have open discussions about the best course.”

“At this point, unfortunately, that is not an ideological exaggeration. That is sober reality.”

“One sees: one could say ideology eats brains. You don’t know what you’re talking about. It is an… idea… chasing after this ideology.”

You can see: as hard as it is to notice something like that in yourself, it seems easy to accuse others of it. And this discussion about ideology has also existed for a long time—since the term was coined, really, at the time of the French Revolution. The “Ideologues” were a theoretical scholarly school more than 200 years ago. And Napoleon then used the term pejoratively against people who wanted to intervene in politics with philosophical ideas or also science.

Kathrin Kühn: And so ideology immediately became a political fighting term.

Luca Rese-Knauf: Which it still is today.

Kathrin Kühn: Yes. And this ambivalent relationship between science on the one hand and politics or ideology on the other runs through. And it makes the case of this current study so exciting. Because in science, striving for objectivity is a particularly high value. The point is precisely to isolate personal attitudes or opinions—and certainly ideology—through methods to achieve more objectivity. That’s the core idea behind scientific methods.

Which is not always easy. Not easy at all.

Luca Rese-Knauf: No. And that’s because research is also a process with many individual decisions. For example: how do I formulate a question, which period do I look at, or how exactly do I analyze data—with which model?

Kathrin Kühn: So a research question can be approached very differently, and then something different can come out.

Luca Rese-Knauf: Exactly. And then the question is: why? What factors in? And there are many studies on this—so-called many-analyst studies. That means: various research teams examine the same relationship or work on the same question with the same data.

And that’s exactly what this new study—also by George Borjas—does, which has triggered these debates. It’s a reanalysis of a study that already appeared in 2022 in the journal PNAS. There, 71 groups with over 150 scientists in parallel answered the same question:

Does immigration change support for welfare-state programs?

Kathrin Kühn: So in the end something like: how does migration influence a society?

Luca Rese-Knauf: Exactly—positively or negatively. That’s a question with a lot of influence, because politics also follows it and looks at such results. And in the first study, the results were very different. But the authors couldn’t explain exactly why.

I spoke with Nate Breznau, the other author of the new study, a sociologist and political scientist from the US. He works at the German Institute for Adult Education in Bonn, and he explained how this “new edition” came about.

Nate Breznau: The concrete research question emerged rather unexpectedly. A colleague, George, contacted me and had some questions about an original study by me and two colleagues. He reanalyzed our data at the time and found that there was a statistical relationship between the research teams’ pre-existing attitudes toward migration and their results.

Luca Rese-Knauf: Yes, and in the data, teams with a positive attitude toward migration more often found positive effects, and migration-critical teams rather negative. And the two, Nate Breznau and George Borjas, have now published this re-evaluation of the data in January in the journal Science Advances.

Kathrin Kühn: And that was then indeed a clear relationship between attitude and result.

Luca Rese-Knauf: Yes—but here comes the “but”: even if it’s labor-intensive to find so many research teams to do this, in the end 71 is still a small sample. Too small a sample to be able to say how large this influence actually is.

That means: the true influence—the effect—could be very small or medium-sized or large or very large. And we can say little about that here.

And one criticism of the study was that the starting question was too imprecise and each team, with its decisions, actually answered a different sub-question. And then, of course, a different answer came out. And the wording was criticized—that these attitudes were equated with ideology.

Kathrin Kühn: By attitudes, you mean now how someone stands on migration?

Luca Rese-Knauf: Exactly. The researchers who took part were asked whether they would rather loosen immigration in their own country or rather restrict it. But that doesn’t have to be ideology. It might also simply be that someone answers pro-migration because the government in their country promotes migration for economic reasons. And this is also criticized by the economist Michael Clemens from George Mason University—the one we heard at the beginning.

Michael Clemens: Ideological bias is not what is being measured here. If this is ideology, then every conviction would be ideological. And that’s not true. Ideology means that one is immune to facts. Convictions can arise from ideology, but also, for example, from scientific insights. That is the internal problem of the text. Beyond that, there is an additional external context which, in Borjas’s case, is very serious and clear.

Luca Rese-Knauf: Yes, and the last point is actually his bigger criticism. And now we’re at George Borjas, the co-author, who, according to Clemens, is himself ideologically biased.

So: it’s a study about how attitudes influence results in migration research. And the study itself could be influenced.

Kathrin Kühn: Yes, that’s what Michael Clemens says. For background: George Borjas is considered a pioneer of migration economics. He holds the position that migration can also have harmful consequences for a national economy. And he was last year an economic adviser in the White House—an office he resigned from in January on his own initiative. He hasn’t really distanced himself from Trump, at least not from what I found. And the Washington Post calls him “the man of Trump’s migration policy.”

Michael Clemens has known him for years from professional debates. And Borjas has also called him an activist or social engineer. So again an ideology accusation.

Michael Clemens: The context makes it very clear that Borjas reads this as an empirical confirmation of what he has long believed: that there is a kind of academic conspiracy against him. And that many scientists, when they have empirical results that differ strongly from his own, deliberately choose methods to produce exactly such results, due to their prior assumptions. But this work does not show that at all.

Kathrin Kühn: But that really is his interpretation. That’s not in the paper—we have to say that. And Clemens also says: the study itself is very well done. But the example shows how tangled the whole thing is, and how sensitive the relationship between science and ideology is in the end. We also contacted George Borjas, but got no answer. And Nate Breznau said this about the criticism:

Nate Breznau: It could be that he is right. It could be that ideology plays a role for George Borjas in his research. I think George and I do not have the same political positions—I don’t know exactly. But precisely for that reason, ideology was an additional reason for us to choose a particularly robust research approach.

Kathrin Kühn: By “robust research approach” he means: they recalculated everything with very different models. And the relationship showed up in around 90 percent of the cases.

And what follows from this now? Does it follow what Aileen Oeberst explained earlier—that one’s own attitudes always have an influence on what one does, how one sees it?

Luca Rese-Knauf: Yes, I think that is a universal phenomenon. And what also must be said: the sciences have means to reduce exactly that—the influence of one’s own attitudes on the path of analysis.

Michael Clemens: That’s one reason why recently there has been a very strong trend in research toward preregistered study designs. That means: before you begin evaluating data, you define exactly what you will do with the data, which variables will go into the statistical analysis, and what the analysis will look like. After that, your hands are bound in a certain way.

Kathrin Kühn: Michael Clemens says that. And Nate Breznau says: much more self-criticism would be possible and necessary in the sciences. That was also one reason why the paper was not accepted by some journals, he said. And, he says, you already learn in training to find “sexy” headlines—and that interesting outlier results can be marketed better than robust, dry results that in the end really are new knowledge. So all these incentives in the system can also already bias the situation.

Pretty similar to us in journalism: punchy headlines, interesting outlier results. And complexity doesn’t sell as well as a strong opinion I can get upset about.

Luca Rese-Knauf: Yes, exactly. And in science there are now already a few journals that promise publication before the results, so that these incentives don’t even arise in the first place. Or that people join forces who come from very different directions, with very different attitudes.

Michael Clemens: Researchers with different prior assumptions work together. And they set the research design in advance and agree on what evidence would support which position. That replaces competition with cooperation and places the gain of knowledge above career interests. And I see that as a very desirable path.

Kathrin Kühn: And in the end, a bias, a distortion can also be: where do you even look? Which questions get how much attention? The spirit of the times plays a role. And there are regularly formats like “The Forgotten News,” also here with us. And one could ask the same about research: what gets too little or too much attention?

A huge chunk. But in the end something where you might recognize the beam—or the plank—on your head after all. Luca, thanks for the research—not only on the Borjas case, but on this case: all of us with the plank, the beam in front of our heads, I’ll say.

Luca Rese-Knauf: Yes, thanks. Gladly. It was exciting.

Kathrin Kühn: And what to do about it—science, but also journalism? There are approaches. We’ve already heard what can help when it comes to you personally—when I say: I want to get closer to what the situation really is. I asked Aileen Oeberst that too. And she answered this:

Aileen Oeberst: As I said, the research would most strongly suggest that we should keep challenging ourselves again and again. And not in a pseudo way, but seriously. If I really have the desire to see things more objectively and not so distorted through my own personal lens, then it would be good to repeatedly ask myself consciously: to what extent could it be that I’m wrong here?

If you take a concrete example— a political topic on which one forms an opinion—then it would be good to consciously expose oneself to counterarguments.

But not to simply devalue them and say: you see it wrong anyway and I see it right. Rather to take it seriously, to truly challenge oneself and to look: does that really speak? Does everything really speak for my view of things? Or what actually speaks against it—and does that perhaps change something about my view of things?

Kathrin Kühn: But sometimes that’s pretty difficult. If I think back to the pandemic: I thought about certain points. But then very harsh judgments were made about groups. Then I also simultaneously detached myself from the group I perhaps identify with if I start thinking like that. That’s already exhausting.

Aileen Oeberst: Absolutely. Of course it is. And that contributes to exactly this confirmation bias and to why it is so widespread—because it has a lot of obstacles. Maybe I won’t be liked then in my group. Or maybe I even lose that group. Then I’m not socially so well embedded anymore; I have to see.

But also, this questioning of one’s own conviction—this is typically something people don’t like so much. Things like ambivalence, or for example, “so how is it now?” many people can’t tolerate very well. They want to keep things clear for themselves. And that, too, contributes to the fact that one perhaps rather does not question oneself, or does not sit down and think it through. And it’s also effortful. That’s also clear. We all have a full life, that’s understandable.

But if one had the goal of seeing things more objectively, then that would still be the recommendation.

Kathrin Kühn: What do I get out of it, if I see things more objectively?

Aileen Oeberst: Well: I am perhaps less influenced by something that I was either taught already or that I’m convinced of—something that maybe isn’t even true. And I think group beliefs are a good example. The belief that one’s own group is somehow better than other groups—if one systematically questions that, and if many would do that, that could clearly lead to very positive consequences. Namely, for example, to significantly fewer conflicts between groups.

Perhaps even on a large scale to fewer wars between groups—between nations, for example—because the belief isn’t so strong: we have the only correct perspective, we are better than the others, the others are the guilty ones, the evil ones, etc. I’m exaggerating a bit, but that’s the core.

Kathrin Kühn: But sometimes it can also be that you then become more exhausting for others at first. For example, I’m in science journalism and I notice: the more I deal with certain topics, the less I can, as a private person, say: yes, I have the following opinion. Instead I end up more often saying: oh, I used to think it was like this and now I don’t know. And that’s of course an ultimate extender for conversations.

Aileen Oeberst: Absolutely. I understand that and I see it exactly the same way. The more you deal scientifically with a topic, the more you always see: things are never that simple, never that clear, never that banal. It’s always differentiated. It’s always complex. You always have to consider: yes, it is like this if—and but sometimes it is like this. But at the same time it may really just be an illusion to believe that things are simple. And also that a stance on a topic can be banal and simple.

Surely there are topics where it can be quite clear. But often it’s more differentiated, isn’t it? And maybe that would help us—for example, if we take a more differentiated perspective—to understand counterpositions and other opinions better, or at least be able to accept them better.

Kathrin Kühn: Helping yourself more, challenging yourself—that’s the recommendation from Aileen Oeberst. And one can also say: the more often I do that, and the more people do it, the more normal the whole thing becomes. Which then creates new questions that end up scratching at how we, as a society, deal with information.

The path toward ethical science is paved with diamonds

I discuss why diamond open access is necessary to reduce perverse incentives in science, and to improve the outputs of Gen AI.

An editorial based on this blog post now appears in the Bulletin of Computer and Data Sciences 7(1):1-9. https://doi.org/10.71448/bcds2671-1

Over the last two decades awareness of a Reproducibility Crisis penetrated all scientific disciplines. Studies recently showed that 90% in STEM and 95% in psychology were aware of this Crisis1,2. At its core this is a crisis of trust. The findings scientists publish and tout as facts, are often not replicable by others. Moreover, a great many are not even computationally reproducible using the original data3,4.  The Open Science Movement developed partly in response to this Crisis5,6. Especially in the last two decades, researchers organized grassroots movements to make science more transparent, reliable and ethical.

One of the many tenets of the Open Science (OS) Movement is open access. Published research results must be accessible to everyone. This is not a new idea. In the 1940s Robert K. Merton developed normative goals necessary to ensure integrity in science7. One was that all scientific findings should be public property. He argued that effective and efficient progress of science depends on open public access. In data science this Movement led to the concept of Open Science by Design – a strategy where scientists plan in advance how they will make all metadata, data and algorithms available and easily re-usable by anyone8.

The OS Movement has been successful. Open access journals and articles increased rapidly. They outpaced the global increase in publications in other formats in the last two decades9. One problem is that these open access articles are overwhelmingly funded by the authors of the studies via Author Processing Charges (APCs). It costs over 10 thousand US$ to publish a study open access in the journal Nature, and most other journals charge at least two thousand. Except for elite institutes and projects with generous third-party fundings, these fees are not usually covered or coverable by universities. A naïve outsider might ask why scientists would pay astronomical fees to make their own hard work publicly available, when they can simply share it online in a free repository like the Open Science Framework, Github or any number of preprint servers based on the ArXiv model?

The answer is competition. The strongest norm governing the practice of science today is publish-or-perish. In every field and science at large, scientists are judged based on their publication record. Ask any academic what the top journals in their area are, and you will get relatively consistent answers by discipline, sub-field and science in general10. Look at the faculty of any ‘top’ university, and the faculty in any discipline will have one or more publications in these ‘top’ journals. Without publications in these journals, a scientist has no chance at a career in science. This is a collective cultural fact. One deeply institutionalized in the organizations, rules, norms, expectations and behaviors of scientists11,12.  

Thus, the founding of new open access journals with lower APCs has little impact on the scientific enterprise because they cannot compete with institutionalized legacy statuses of existing journals. The greatest success story is the non-profit publisher PLOS, which rose swiftly in the rankings, but could not crack into the very top tier. Although far cheaper than Nature, it is still not ‘cheap’, with APCs in their family of journals ranging from 2.5 to 3.2 thousand $US[1].

The most successful open access models are within existing paywalled journals where authors have the option to publish “gold” open access, rather than publish for free. The payment of somewhere between two and 10 thousand US$, gets authors the right to have their single article published open access inside of a closed access journal. This means that despite a massive shift toward open access, the fundamental structures of scientific publishing have not changed.

Ethical Implications

Competition in science supports motivation and innovation, but the publish-or-perish norm is a toxic externality. Scientists willingly prioritize subjective journal ranking, impact factor and increasing their citation counts to get ahead within the competitive scientific enterprise. They do this despite widespread skepticism and evidence that rankings and citations do not correlate strongly with the quality and reliability of published studies14–16. They do this because they must, or at least perceive that they must, in order to follow their scientific career aspirations. The pressure to publish thus motivates rent-seeking behaviors designed to increase publication chances, i.e., career chances. Conducting higher quality science is one of these behaviors. But there are many methods to increase publication chances that are science orthogonal, what are known today as questionable research practices (QRPs)17,18.

Some QRPs are unconscious and learned from supervisors in the process of converting research into a publishable paper. For example, researchers routinely run many statistical models but report only those that show the strongest support of their claims. Researchers who believe that their claim is true in the first place will gravitate toward models that support it, convincing themselves intrinsically that these models are the best tests of their claim.

Decisions based on confirming intrinsic beliefs or window dressing for peer reviewers reduce the replicability of science. They narrow down the multiverse of potential findings into a highly selected set of results. This selectivity is independent of the process that generated the data in the first place, in other words, it is science orthogonal. A classic example is a study published suggesting that hurricanes with feminine names cause more damage than those with masculine names. It turns out that using the available data, the original researchers selected a model that produced regression coefficients that were extremely far away from the central tendency among all other plausible models’ regression coefficients19. We do not know whether this was a conscious decision but their reporting hides the truth and simultaneously increases publication chances.

There are of course conscious and highly unethical behaviors leading to an entirely false representation of reality. Science is filled with scandals of hacking and data-faking. Rent-seeking alone can explain unethical learned unconscious and conscious behaviors of scientists. They seek status and money and job security, or in some cases seek results that support a particular worldview or policy outcome20,21. But rent-seeking is unambiguously the main reason22,23.

If we as a scientific community and science-interested public, want to eliminate the perverse incentive structures that bound and inform rent-seeking behaviors of scientists we need radical change. Status should not be assigned based on journal metrics that often have little to do with the quality of the research being conducted or published and more to do with legacy and embedded norms. The most radical proposal is to eliminate journals altogether. But this is an extremely unlikely outcome no matter how powerful the OS Movement becomes.

Journals became standard in science hundreds of years ago as a means for communicating scientific discoveries across time and space24. They enabled scientists to acquire knowledge without travelling to faraway universities. Publishing was not cheap, and with the dawn of digital media, it became even more expensive as publishers raced to provide their journal both in-print and online. The costs associated with publishing gave publishing firms a great deal of power over time.

The result is that companies like Springer Nature and Elsevier have enough power to dictate to scientists how they perform their research and communicate their results25. They shape academic careers, institutional priorities, governments’ science policies, and perceptions of journals and the publishing enterprise among scientists26. Their primary legal interest is their shareholders. Profit is their priority, and only second is to provide a service to science as their product. A simple mathematical proof confirms that profit is their priority: If they cannot make a profit they will no longer provide the scientific services but if they can make a product without scientific services they have no reason to stop.

Although big publishing firms have shown many draconian practices in their ‘service’ to science27–30, scientists still need a means to communicate their results with each other and the public. This can be done without for-profit publishing but probably not without a journal publication format, or something very similar. For example, we now have a plethora of ‘green’ open access preprint servers where scholars can deposit working papers, or prior versions of their published articles. These are a viable means to communicate science without the perverse incentives generated by big publishing or institutionalized journal rankings. This all still involves the journal article as the standard unit of science production.

Diamond Open Access

If we are ‘stuck’ with journals in science as our primary communication medium, then they should be as free from perversely incentivized bias as much as possible. The first step is thus to remove for-profit publishing from the equation. It is fine to use the services of for-profit publishers. They have shown the capacity to provide print on demand, marketing and scientific journalism. But when they control and direct the scientific enterprise when have an ethical conflict of interest – namely profit versus robust science.

The costs of publishing a journal are the lowest in history. There are publication kits that help associations, institutions and stand-alone journals to take publication into their own hands31. With minimal costs and self-governance, journals have no need to push institutions to purchase journal subscriptions and no need to charge authors astronomical publication fees. Thus, they themselves are not perversely incentivized to perpetually increase their status to make their product profitable.

The optimal existing solution is diamond open access, whereby a journal charges no APCs and is freely readable and downloadable online32. It is the most ethical and equitable by design, and is perceived as the ideal model by most academics33. The OS Movement is overwhelmingly in favor of diamond open access34, as are governance bodies – at least those free from the influence of big publishing like UNESCO35,36. Despite 13 thousand journals indexed in the Directory of Open Access Journals (DOAJ) with no fees as of February 16th, 2026, these journals are not on the radar of most indexing services, and do not belong to the mainstream of journals published by major scientific societies37. This means that successful diamond open access journals are very rare.

Because of such a saturated scientific ‘market’ for publication outlets, starting new diamond open access journals has had little impact on producing a more ethical science. They do not gain reputation. The reason PLOS was so successful, despite failing to break into the highest echelon of science, was because it has a huge cash flow and can use it for branding and promotion. From an ethical science perspective it is valuable, like gold, but it is not as valuable as diamond.

Flipping or Starting Over?

To have the highest ranking and most well-known journals diamond open access, societies need to cancel their contracts with for-profit publishers38. One problem with this is that many academic societies are themselves run like for-profit businesses. They seek to generate as much revenue as possible, and diamond open access would threaten this model. They have embedded relationships with publishers and agree to renew contracts together, mostly independent of the scientists they serve. I witnessed this first hand with the American Sociological Association and Sage39,40. Contracting a for-profit publisher provides a non-profit academic organization a scapegoat for amassing capital via subscription fees.

There are incredible exceptions; however, and they offer model success stories. Computational Linguistics is one of the earliest journals considered to be among the top in its field, to flip to diamond open access. The first step was a move by the Association for Computational Linguistics in 2002 to create the CL Anthology which made all articles published by association journals open access online after an embargo period. Then in 2009 the journal Computational Linguistics flipped to diamond open access. The journal Demography of the Population Association of America is another example.

The embeddedness of big publishing in science, and the capital that associations can raise through their journals when run by big publishers, are major barriers to flipping. Another major barrier is that some big publishers coerce scientific societies into signing away the rights to the titles of their journals in their publishing contracts. This tactic is most intensively deployed by Elsevier. They own the rights to the titles of nearly all the journals they publish. Therefore, when these societies want to change publishers, they cannot. They are trapped. They would have to start a new journal with a different title – what happened for example with the Journal of Infometrics41. There is otherwise no way around this problem because of copyright law.

Therefore, collective efforts to build the popularity of new diamond open access journals would greatly increase the movement toward a more ethical and effective scientific enterprise. If these are journals from societies, which are essentially the same journal but with a new (not copyrighted by Elsevier) name, it requires authors to support this journal and immediately abandon the other. Supporting new diamond open access journals, whether completely new or newly named, requires established scholars to put their status-seeking egos aside in the name of scientific progress and ethics. It is precisely those who have built major scientific reputations who need to engage in this change, because they can give them most clout to new journals by touting them and publishing in them.

Scholars and societies alone cannot carry the burden. Hiring committees need to reward these behaviors. Rather than seeing a publication in a new ‘unranked’ or ‘low ranked’ diamond open access journal as a sign that the article is ‘not high quality enough for top journals’, committees should judge publications only on their content. Then, if two publications are seen as equally scientifically rigorous and high quality, the one in a diamond open access journal should get a greater weight. Scientist who consistently publish high quality research in diamond open access journals should be favorites of hiring committees, all else equal.

A hiring committee would be unlikely to reward an applicant who shows sociopathic behaviors. Following this logic, they should not reward scientists who show behaviors that go against ethical science by practicing closed and profit-incentivized science. I need to be very clear here that I personally am still publishing regularly in journals published by for-profit publishers. I am not that famous, and I do not have tenure. This is a perfect example of why we need sweeping changes at the institutional level, and from the top of the scientific hierarchy.

With efforts to flip both  existing journals and efforts to reward new, ethical journals, we give science its greatest future chances for improvement and sustainability. There are many efforts underway and these should serve as guides, for example the Diamond Open Access Fund from the Dutch Research Council and MIT’s shift+OPEN initiative.

Diamond AI

Diamond open access has a second, equally important role for the future of science. It is necessary to inform Generative Artificial Intelligence (Gen AI). The LLMs that power popular Gen AI are trained heavily on corpora assembled from what is available via the Internet. Paywalled literature is missing from training corpora, not because it is unimportant, but because it is not accessible or legally usable42. Gen AI outputs are therefore based on a highly restricted sample of all scientific knowledge. Open access for all of science would ensure that anyone using Gen AI, would get the best possible information based on all that we know as humans. As essentially everyone is using Gen AI today43, this would mean that the public would be optimally informed.

It is not necessary for Gen AI development that all articles are diamond open access, they just need to be somewhere, e.g., green or gold open access. Yet, without diamond open access the entire knowledge enterprise will continue to favor the work of those with greater resources44. Resources are of course necessary to produce higher quality science because of research costs, but when it comes to publishing and dissemination, a resource advantage reproduces the already existing Global North-South disadvantages in science. This limits human capacity to tap resources in lower income societies. These are societies filled with potential contributions to science that could benefit both 1) their own societies’ development – because it enables them to gain status, resources and build stronger, more attractive and sustainable scientific institutions, and 2) all of scientific knowledge because there are brilliant minds waiting to be tapped that might otherwise give up on science because it is for them no sustainable in their region.

If the knowledge in Gen AI remains Global North and WEIRD biased (Western, educated, industrialized, rich and democratic), the result is that these countries remain culturally and economically advantaged beyond that which exists presently. This means that without diamond open access norms, we are willingly allowing Gen AI to increase cultural hegemony and economic domination of a minority. I am not directly arguing whether this is good or bad. I am in the Global North and profit from a stronger Global North science advantage. My argument is that science should be neutral, favoring the most optimal and reliable knowledge, rather than legacies or strategies that game the scientific system.

References

1.          Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016).

2.          Metskas, A. How Much Do Academic Psychologists Trust Academic Psychology, and Is There Still a Replication Crisis? Transparent Replications https://replications.clearerthinking.org/how-much-do-academic-psychologists-trust-academic-psychology-and-is-there-still-a-replication-crisis/#survey-demographics (2025).

3.          Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, (2015).

4.          Breznau, N. et al. The reliability of replications: a study in computational reproductions. Royal Society Open Science 12, 241038 (2025).

5.          Engzell, P. & Rohrer, J. M. Improving Social Science: Lessons from the Open Science Movement. PS: Political Science & Politics 1–4 (2021) doi:10.1017/S1049096520000967.

6.          Breznau, N. Legacy of Jon Tennant, “Open science is just good science”. Crowdid https://crowdid.hypotheses.org/548 (2022) doi:10.58079/ne8h.

7.          Merton, R. K. The Sociology of Science: Theoretical and Empirical Investigations. (University of Chicago press, 1973).

8.          Wittenburg, P. Open Science and Data Science. Data Intelligence 3, 95–105 (2021).

9.          NCSES. Publication Output by Region, Country, or Economy and by Scientific Field. https://ncses.nsf.gov/pubs/nsb202333/publication-output-by-region-country-or-economy-and-by-scientific-field#utm_source=chatgpt.com (2023).

10.        Serenko, A. & Bontis, N. A critical evaluation of expert survey‐based journal rankings: The role of personal research interests. Asso for Info Science & Tech 69, 749–752 (2018).

11.        Mancoridis, M., Sumers, T. & Griffiths, T. Publish or Perish: Simulating the Impact of Publication Policies on Science. Proceedings of the Annual Meeting of the Cognitive Science Society 46, (2024).

12.        Breznau, N. Questionable research practices from the practitioners’ perspectives. Crowdid https://crowdid.hypotheses.org/1666 (2025) doi:10.58079/14f3u.

13.        Lawrence, S. Free online availability substantially increases a paper’s impact. Nature 411, 521–521 (2001).

14.        Fleck, C. The Impact Factor Fetishism. European Journal of Sociology / Archives Européennes de Sociologie 54, 327–356 (2013).

15.        Rushforth, A. & De Rijcke, S. Practicing responsible research assessment: Qualitative study of faculty hiring, promotion, and tenure assessments in the United States. Res Eval 33, (2024).

16.        Dougherty, M. R. & Horne, Z. Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences. R Soc Open Sci. 9, 220334 (2022).

17.        Gopalakrishna, G. et al. Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands. PLOS ONE 17, e0263023 (2022).

18.        John, L. K., Loewenstein, G. & Prelec, D. Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychol Sci 23, 524–532 (2012).

19.        Muñoz, J. & Young, C. We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness. Sociological Methodology 48, 1–33 (2018).

20.        Borjas, G. J. & Breznau, N. Ideological bias in the production of research findings. Science Advances 12, eadz7173 (2026).

21.        Rainero, V., Stolz, J. & Luijkx, R. The Faith Factor. How Scholars’ Religiosity Biases Research Findings on Secularization. Sociological Science 13, 154–177 (2026).

22.        Aronson, J. K. When I use a word . . . “Publish or perish”: adverse effects. https://doi.org/10.1136/bmj.r1577 (2025) doi:10.1136/bmj.r1577.

23.        Paruzel-Czachura, M., Baran, L. & Spendel, Z. Publish or be ethical? Publishing pressure and scientific misconduct in research. Research Ethics 17, 375–397 (2021).

24.        Carey, J. Scientific Communication Before and After Networked Science. Information & Culture 48, 344–367 (2013).

25.        Larivière, V., Haustein, S. & Mongeon, P. The Oligopoly of Academic Publishers in the Digital Era. PLOS ONE 10, e0127502 (2015).

26.        Rossello, G. & Martinelli, A. The effect of lobbies’ narratives on academics’ perceptions of scientific publishing: A survey experiment. Information Economics and Policy 71, 101148 (2025).

27.        Butler, L.-A., Matthias, L., Simard, M.-A., Mongeon, P. & Haustein, S. The oligopoly’s shift to open access: How the big five academic publishers profit from article processing charges. Quantitative Science Studies 4, 778–799 (2023).

28.        Else, H. Dutch publishing giant cuts off researchers in Germany and Sweden. Nature 559, 454–455 (2018).

29.        Lancet, T. & Board, T. L. I. A. Reed Elsevier and the arms trade. The Lancet 366, 868 (2005).

30.        Jureidini, J. & Clothier, R. Elsevier should divest itself of either its medical publishing or pharmaceutical services division. The Lancet 374, 375 (2009).

31.        Scholastica. New fully-OA publishing toolkit and stakeholder reflections on 20 years of the BOAI. https://blog.scholasticahq.com/post/oa-publishing-toolkit-and-stakeholder-reflections-BOAI20/ (2022).

32.        Normand, S. Is Diamond Open Access the Future of Open Access? The iJournal: Student Journal of the Faculty of Information 3, (2018).

33.        Kumari, M. & A, S. Perceptions of open access publishing: A comparative study of gold and diamond models among global researchers. Alexandria 35, 55–73 (2025).

34.        Plan S. Working collectively towards an equitable, community-driven and academic-led scholarly publishing model. Action Plan for Diamond Open Access https://www.coalition-s.org/action-plan-for-diamond-open-access/ (2022).

35.        UNESCO. Diamond Open Access. Public Service Press Release vol. Online Report (2026).

36.        UNESCO. UNESCO Recommendation on Open Science. https://unesdoc.unesco.org/ark:/48223/pf0000379949.locale=en (2021).

37.        Simard, M.-A., Basson, I., Hare, M., Larivière, V. & Mongeon, P. The Value of a Diamond: Understanding Global Coverage of Diamond Open Access Journals in Web of Science, Scopus, and OpenAlex to Support an Open Future. Proceedings of the Annual Conference of CAIS / Actes du congrès annuel de l’ACSI https://doi.org/10.29173/cais1845 (2023) doi:10.29173/cais1845.

38.        Trueblood, J. S. et al. The misalignment of incentives in academic publishing and implications for journal reform. Proc. Natl. Acad. Sci. U.S.A. 122, e2401231121 (2025).

39.        Why I’m leaving the American Sociological Association. Family Inequality https://familyinequality.wordpress.com/2021/11/06/why-im-leaving-the-american-sociological-association/ (2021).

40.        Philip Cohen’s ASA Publications Committee platform. Family Inequality https://familyinequality.wordpress.com/2018/01/14/philip-cohens-asa-publications-committee-platform/ (2018).

41.        Singh Chawla, D. Open-access row prompts editorial board of Elsevier journal to resign. Nature https://doi.org/10.1038/d41586-019-00135-8 (2019) doi:10.1038/d41586-019-00135-8.

42.        Szkalej, K. The Paradox of Lawful Text and Data Mining? Some Experiences from the Research Sector and Where We (Should) Go from Here. GRUR Int 74, 307–319 (2025).

43.        Breznau, N. & Nguyen, H. H. V. An Introduction to Generative Artificial Intelligence for Academics. F1000 Research Preprint Status, (2025).

44.        Kwon, D. Open-access publishing fees deter researchers in the global south. Nature https://doi.org/10.1038/d41586-022-00342-w (2022) doi:10.1038/d41586-022-00342-w.

Ethical Statement

I have no conflict of interest to report. I used Google search which includes Gemini by default, ChatGPT, NotebookLM and Nano Banana to support my literature review, check arguments, suggest words or phrases and to extract facts. No writing was copied from Gen AI, it is all my own. Any mistakes or opinions are also my own.

[1] https://plos.org/fees/

Sociologists tend to be left politically. Is this a problem?

Someone recently asked me, ‘is it an epistemological problem if sociologists tend to be more left politically?’. And if it is, if I know how to solve this problem.

My thoughts: If it does represent an epistemological problem, I have no idea how to solve it. I can’t imagine trying to push right-leaning persons into sociology. At the same time, we should consider that the definition of what science is and should be varies, especially across disciplines. In sociology for example, there is a strong sense of communality. Robert Merton wrote about scientific norms that would be necessary to support the scientific method (test, re-test), because alone the scientific method does not guarantee the most efficient and effective knowledge production and/or can be used to maintain power over others. One of these norms he called “communism” which we today label as “communality” because he did not at all mean a system of governance or centralized economic planning or the Communist Party. What he meant was that everyone should have equal access to participate in science. If everyone does not have equal access to participate in society, then they by default do not have equal access to science. Why should everyone have access? Because this leads to the maximum efficiency in scientific gains and breakthroughs, because everyone can contribute consistent with their abilities. Therefore, many sociologists would define science in a way that demands social and political change in order to support scientific progress (scientific progress means also economic productivity and abundance to provide for everyone’s needs). But trying to change societies in ways that maximize science opens the door for political agendas within science, which can increase bias in findings. Therefore, I am not sure if this is an epistemological problem. It may be that epistemologies correlate with ideologies, but that otherwise the epistemological approaches in sociology and political science (and related disciplines) that are among the broadest in science (ethnography, participant observation, econometrics and psychometrics, survey data research, interviews, ethnomethodology, etc.) are beneficial to collective knowledge.

Regarding the photo above: In the Tuskegee Experiment, the scientific method was followed carefully. But the scientific method alone does not guarantee that humans are safe from, and/or have equal access to participate in science. The participants were deceived and treatment for their disease was withheld. Unethical and abusive, yet scientific if we take the narrowest possible epistemology of the scientific method.

Image source (public domain): https://www.flickr.com/photos/pingnews/441531333

Media outlet and Q&A for ‘Ideological bias in the production of research findings’ by Borjas and Breznau

George J. Borjas and I recently published our study with the same beginning title of this blog in Science Advances. There are starting to be media inquiries and social media commentary. Here are some results so far:

  1. Süddeutsche Zeitung (SZ) article by Sebastian Herrmann “Warum Forscher aus denselben Daten entgegengesetzte Schlüsse ziehen” (Why researchers reach different conclusions from the same data).
  2. Manhattan City Journal perspective piece written by George and I.
  3. A news report by Luca Rehse-Knauf for Deutschlandfunk (German radio) and their Forschung Aktuelle series “Migrationspolitik: Einstellungen können Forschungsergebnisse beeinflussen” (Migration policy: Attitudes can influence research results).
  4. A news report by Katrin Kühn and Luca Rehse-Knauf for Deutschlandfunk and their Fakten und Meinungen Series “Darum sind wir Menschen nicht objektiv” (Why we humans are not objective).
  5. A PsyPost report by Eric W. Dolan “158 scientists used the same data, but their politics predicted the results“.
  6. A podcast on The Last Show with David Cooper. Apple / Youtube.
  7. A podcast in Allegedly Does Not Replicate with the Institute for Replication (I4R) with Abel Brodeur and Juan Pablo Posada Aparicio.
  8. A substack post by Claudio Teixiera after an interview with us about how researchers conduct research and the ‘invisible’ paths they follow.
  9. A substack post by Laurenz Gunther describing our work and digging deeper into bias among researchers (here those working on the topic of immigration) – despite a relatively far out conclusion.
  10. There was a Neuer Züricher Zeitung article about the original ‘Hidden Universe’ study (paywalled) ‘Das Experiment: Wer bekommt die rote Karte?.

The following are a list of answers I prepared for interviews and inquiries to our article.

— How did you arrive at your research question, and what is the study about?

George emailed me and had a few questions about our original study. He was at that time analyzing our data and had found a statistical association between pre-existing preferences for more or less migration among the teams in our study, and their findings. I was very skeptical. I have now worked on replication and reproducibility themes for almost a decade. I am acutely aware of what we often refer to as ‘researcher degrees of freedom’, also known as ‘the garden of forking paths’. This refers to choices that researchers can make during the research process that can lead to different outcomes. I assumed that the statistical association he found would not hold under different but equally plausible model specifications. I began testing many different models. Basically, they all showed the same result. Therefore, I became convinced that this was more than a fluke.

Actually, George had already run most of the same models. We present all of our models in a multiverse analysis in our paper. Out of 883 models 88% showed a significant statistical effect suggesting that we should reject the null hypothesis that ‘ideology has zero effect on the teams’ research findings’. If we take the assumption that we should only trust models that control for researchers’ educational experiences – something we believe impacts their results – then we find that roughly 93% of the models show a significant statistical effect.

— Could you explain the experimental design and methodology?

This study is an exploratory secondary analysis of the data generated by the experiment of myself, Eike Mark Rinke and Alexander Wuttke. We gave 71 research teams the same data and hypothesis – that immigration reduces support for social welfare policies. We surveyed them on their backgrounds and research experience and asked them if they believed the hypothesis was true and what they thought about immigration policy. In the original study, again the one that I helped lead, we did not find any important impact of immigration preferences. We essentially found that the results went in all directions, and we could not easily explain the variation.

After working together, George and I agree that the statistical analysis in the original study was not a clean test of immigration on the research teams’ results, because it controlled for the statistical model specifications that the teams’ made. The original study was were searching for key decisions that might explain why results went in different directions, so this naturally made sense. But in hindsight, these decisions are the mechanism through which ideology gets transmitted into statistical results. Thus, the original study introduced what is known in statistics and causal analysis as a ‘confounder problem’. If someone has an ideological bias, they will choose statistical models that will lead to more desirable results. The original study was controlling for both the test variable (ideology) and its mechanism (the statistical models), and in doing so it suppressed the impact of the test variable we were trying to observe.

This time, George and I conducted regression analyses in which the research findings were the dependent variable and ideology the independent variable – without model specifications as control variables, but with further controls.

— What are the limitations?

A key limitation is that this study is exploratory, not confirmatory. It relies on secondary data from a study that was not specifically designed to test the impact of ideological bias. It cannot confirm that this bias exists, instead it demonstrates robustly that a statistical association exists between ideology and researchers’ findings. We are not aware of any other way to explain this association other than an ideological bias. But we can only confirm with confidence, that it is prudent to reject the null hypothesis that in this particular sample and study is that there is no association between preexisting preferences for immigration policy and research findings. More specifically, the likelihood of observing the data in this experiment if the null hypothesis were true is very low.

Another limitation is that the size of the effect we found is unclear. It points in a positive direction – more pro- immigration policy stances associate with findings that show immigration has a more positive effect on social policy preferences among the public, and vice-versa with more anti-immigration policy stances and a more negative effect. But because of the great variation in results and the small sample size, the standard errors of the estimated statistical effects are very large. This means that the true effect might be anywhere from miniscule and near-zero, to moderate, to very large. We simply cannot say much about this here. More research is necessary, although this is an implicitly difficult topic to study, because if we inform researchers that we are studying their ideological bias, they might behave differently and this would take away ecological validity.

— According to the study, ideology influences model specifications. Could you provide a concrete example to illustrate how a single design decision (or a combination thereof) can have an impact?

I cannot, and this is another limitation of the study. If I could, it would be something we would have found in the original experiment that collected the data. But we can only point at patterns here. There are certain model specifications, unique combinations of statistical modelling choices that produce more negative results. The teams with more anti-immigration ideologies were more likely to choose these. But there are far more model specifications than there are teams. This leads to a sparse data problem. There are many empty cells in the matrix of all possible model specification combinations that teams would plausibly make. This makes it roughly impossible to pinpoint exact specifications’ effects. and there are many different model specifications that can lead to a positive or negative statistical effect. The point is that the only thing that happened between the teams asked to test the hypothesis with the same data, was different modelling choices. Therefore, this is the only way they could arrive at different results. There was no cheating or result faking, we checked that their statistical code produced the results they reported to us.


— To what extent is this a problem, and to what extent is it normal that decisions, based on analytical decisions, depend on who you are and how you think?

This is nothing that our study answers. And it may not be fully possible to answer because we do not yet know the nature of consciousness. We also cannot measure what is happening inside a human neural network – a brain in other words. But it is clear to me that experience, ideology and preferences shape results. A simple example is statistical training. Many researchers have limited statistical training, and they build only those statistical models that they learned about in their studies. This impacts results.

But more generally idiosyncrasies of people, like ideology, shape what research questions that people are willing to pursue and how, and they shape the reporting of those results. Some could look at our study and think that the estimated statistical impact is large and highly concerning. Others, might look at it and think it is tiny and of no concern at all. Our study suggests that ideology can explain somewhere between 1 and 3% of the variance in the results. If scientific findings are on average 1 to 3% off of what they would be without bias, is that a big problem? I mean… what do you think?

— If I understand correctly, the experiment was originally intended to show how much the results diverged, not why. How did you arrive at ideology as a possible cause?

As I already mentioned, this is something that George noticed in our data. He already had this hypothesis in his mind. I cannot blame him for thinking this. The Open Science Movement and Metascience work reveals many so called ‘Questionable Research Practices’. These include everything from faking data, to tampering with statistical models or stopping the collection of data during an experiment to produce a desired result. These practices are designed to produce certain results in order to obtain a publication or support a pet hypothesis (confirmation bias). Obviously some of these studies were motivated by ideological goals.

— How could ideological bias be reduced? Is this even desirable, or should we simply be aware that it can exist?

The impact of ideology can be reduced by following some clear recommendations of the Open Science Movement. Studies should be pre-preregistered, they should provide all code and materials, they should not be conducted in isolation or in hiding, and researchers should cooperate. Some of us are engaging in so called ‚Adversarial Collaborations‘, where researchers who do not agree – those with different priors about a given hypothesis like the impact of immigration – collaborate. They lay out all the aspects of a study in advance, and they agree on what evidence would count as support of either of the positions. I highly recommend this form of science. It takes competition and turns it into collaboration with the goal of knowledge seeking prioritized above all else.


— You are investigating ideological bias in science using scientific methods. How do you deal with this tension in meta-scientific questions, where you are essentially also your own subject of investigation?

Similar to my last answer, one cannot fully understand or deal with one’s own bias, and therefore needs to build in checks into the process. Things that would reduce this bias, like preregistration. I am working currently on a project that is an Autoethnography of my own questionable research practices and the perverse incentives I encountered during my career in science. I hope that by doing this, and revealing my own behaviors, I can improve them. I also want to be a role model for others, to make it desirable to be highly critical of one’s own work. My goal is to get this study published in a high quality journal and thereby prove that self-criticism, something researchers mostly try to avoid to protect their theories, findings and careers, is something that can be used in a positive way in the scientific process.

— Can one’s own attitudes always play a role, even in this study?

Sure. Definitely. That is why it was important for me to take a so called ‘multiverse’ approach to this study, and many other studies I am working on recently. I want to ensure that I, or one of my colleagues, has not simply selected a statistical model that produces certain results. This practice, known as hacking, or p-hacking, is prevalent in science, especially in secondary data analysis. I essentially learned to do this during my graduate studies. We would find a result we liked and then develop convincing logical arguments why the model producing it must be the best model. So, the idea with multiverse analysis, is to run all or at least all plausible alternative models. This helps reveal if my model is an outlier. Whether it represents something very unique or unusual in the distribution of model results. If it does, this is a cause for great concern. If not, it is evidence of a robust statistical association.

— There have also been critical reactions, for example in this online Bluesky thread, which raise concerns about George Borjas’s views and background. How do you assess this criticism?

I have read the discussion thread by Michael Clemens, an economist at George Mason University. One line of criticism appears to concern the fact that George Borjas recently conducted research for the executive branch of government.

Another criticism seems to focus on the observation that different model specifications yield different results. This is precisely what our study confirms, and it is also a well-established fact in the history of empirical social science.

Both points are orthogonal to our study. Even if we were to assume, hypothetically, that George held some form of ideological bias and that this bias influenced his analytical choices, which I cannot confirm and do not claim, this would not undermine our findings. The reason is that we adopted a deliberately robust research design. We conducted a multiverse analysis comprising 883 regression models. The consistency of results across this large set of plausible specifications makes it implausible that our conclusions are driven by special highly selective model specifications – those that would be selected due to ideological bias.

It is also important to note that George and I do not share the same political views. Precisely for that reason, ideology was an additional motivation for us to adopt a highly robust analytical strategy. I explicitly advocated for the multiverse approach in order to minimize the influence of individual priors, including our own potential ideological orientations. I see this as a great strength of the study.

The purpose of our approach is to decouple empirical results from personal or ideological preferences as much as possible. And to estimate the robustness of our finding to any kind of bias, not just ideological. I would encourage critics to apply the same standards of robustness to their own work. To date, Mr. Clemens has not presented empirical evidence that contradicts our findings. Moreover, criticisms referring to modeling choices in studies conducted by George decades ago are not relevant to the validity of the present analysis.

— What does it mean when you say that only 3% of the variance can be explained by your results.

That has to do with the regression coefficient and the r-squared values. A coefficient of 0.03 shows that a one-point higher (more positive) ideology mathematically predicts a change in results of 0.03. This sounds meaningless, but we know that 0 would be none (and we can equate this with zero percent change) and that 1 would be 1-point on a standardized scale. This is 1 standard deviation in the distribution of the dependent variable. It would be possible to move the results more than 1 standard deviation, but this would be quite preposterous. There is nothing in the complex nature of social science, which lacks laws, that would do that. So I will set the upper bound of the largest possible effect at 1, meaning that 1 would equal 100% of the distance in the distribution of variance.  Therefore, 0.03 is like 3 percent of the distance. At the same time, the r-squared, which tells us how much of the error is reduced from this particular variable, is around 0.03 or less depending on how we measure this variable. This suggests that fitting the observed ideology values into the observed results from the teams, reduces the unexplained variance from 100% down to around 97%.

— Can you explain — very, very simply — what you did? 

In a study that I co-led starting in 2018 (Breznau, Rinke and Wuttke et al. 2022), we designed an experiment that allowed us to observe researchers doing research on the impact of immigration on social policy preferences. We gave them the same data and asked them to answer the same research question: whether immigration reduces support for social policies or not. We documented the researchers’ pre-existing methodological training, experience with and expectations about the topic, and their personal preferences for looser or tighter immigration laws in their own countries. We shared all of the data and documentation of our work publicly. Because we shared our data, a few years later George J. Borjas was able to reanalyze our data and find new evidence of a correlation between the researchers’ ideological positions on immigration and their findings. Those with more pro-immigration positions tended to find evidence that immigration had a positive impact on support for social policy, and those with more anti-immigration positions tended to find evidence that immigration had a negative impact on support for social policy. Here “social policy” means support for a more extensive welfare state providing social security via the government or not – many would call this support for social cohesion. I was skeptical of George’s initial work, and together we vetted George’s findings. We ran almost a thousand alternative statistical models to test George’s finding and of these 88% suggest that we should reject the null hypothesis. The null hypothesis is that if there is no impact of ideology on researchers’ findings, that we should not observe what we observed based on probability. But we did observe this association, and by rejecting the null hypothesis we have evidence that something more is going on, not just random luck in the data. Crucially, we should reject the idea that ideology has no impact on researchers’ results when analyzing the same data. 

— What motivated you to do this study?

As I said, George found this association between researchers’ ideological positions on immigration and their research findings. This is an important scientific observation. I was a bit more skeptical, in particular because my initial analysis of the data as part of the original experiment did not show this association. Together we became very motivated to tackle this problem, and in the end, we have relatively strong evidence of something. The exact nature of this should be subject to further research.

— What are the most striking findings? 

The main finding is that researchers’ own preferences for tighter or looser immigration predicts what they went on to find in their work. This appears striking, but when put into context it is maybe not so surprising. We know that there are many reasons that researchers may consciously or even unconsciously exert influence on their own findings. There are many cases of researchers engaging in questionable research practices to ‘fudge’ their data and results in ways that make them appear stronger than they really are. This has occurred frequently in biomedicine, for example in studies of products whose approval would net the researchers great personal profits. Consider also what we know from psychology, namely confirmation bias. People tend to seek out evidence of what they already believe is true. Researchers are people too. When presented with various forms of competing evidence, a researcher might gravitate toward that which supports their preexisting beliefs or preferences. 

— What are the implications for the social sciences, which are already reeling from the replication crisis? 

The implications reinforce what we already know. We should focus on refining two areas of science. The first is scientific training. We need to make transparent and open workflows and data sharing the norm. This includes researchers stating in advance what they plan to do and what they expect to find. When researchers do not do this, they can run several experiments or analyze hundreds of datasets with millions of different statistical models and simply choose one finding that looks exciting or sexy to them. Generations of researchers before us have learned to do exactly this in order to get published. They learned to ‘sell’ a single selected finding as confirmatory evidence of something in the real world, when in fact it is simply a highly selected, exploration of data leading to a unique event; one that probably does not generalize and is not reproducible (a.k.a. luck). I bring up the idea of getting published here because this is the second problem we must urgently address in science. A researcher’s worth or ranking as a scientist is judged almost entirely on their publication record. In particular, publications in journals that are considered higher status. These higher status journals should be publishing studies because the studies contain higher quality science. But there are ways to game this system. There are a a host of questionable research practices that makes findings look more exciting than they actually are. These practices often lead to irreproducible findings. Essentially fake science. Big publishing is a major profit industry and this increases the pressure to publish, as the publishers of the journals engage in questionable, sometimes unethical practices to sell more journals, as opposed to solid science which is often quite boring and tends to find that new drugs or treatments do not work and that our theories are wrong. Ideally we need to end big publishing’s control over science. Their role should be simply production and distribution, but they currently copyright much of the material and force universities to pay twice, once for the researchers’ salaries and again so that researchers can read what they are publishing. And this often done with public money. Its really wrong and generates perverse incentives among researchers and profit-seeking publishers. 

— It seems teams didn’t falsify data or cherry-pick numbers in any obvious way; instead, ideology appeared to influence judgement calls — is that a reasonable explanation or is it too charitable?

It is a reasonable explanation, but we must be very cautious with it. Ideology might explain about 3% of the variance in researchers’ findings. The rest has to do with other factors or random noise. Consider that many scientists have specific methodological training. Through this training they simply do what they know how to do. And this can influence results. For example, someone who only knows how to use a hammer, will treat things as a hammering problem, when it might be better solved with a different tool. This takes us back to better, broader methodological training, which scientists would have more time for if they weren’t under constant pressure to publish. 

— Are there ways to guard against the ideological bias highlighted in the study? It seems that peer review can spot poorly defined studies — but does it work well enough in the real world? 

Peer review is a poor solution. There are studies out there that show that peer review is not reliable. Just like giving the same researchers the same data, if you give different peer reviewers the same paper, they will come to a huge range of judgements about the paper. The publishing system is in some ways a lottery. One solution for this problem is what we call “adversarial collaboration”. This is a type of research where scientists who disagree about a topic work together. They design a study together and agree on all methods and on all criteria with which to judge the outcomes. Then after this is all agreed in advance, the study is conducted, ideally from a third-party, and then the results speak for themselves.

— What do you think the message here is to the public and to policy makers?

Everything we have in society that works is based on science. Smartphones, that open heart surgery that saved the life of a loved one, airbags, planes that don’t crash. We need science for every decision we make collectively. But the public and policymakers can be highly politicized, and this can influence science, we see this even in the scientists in our study whose own politics seemingly played a role. To cut through this, we need policymakers that support science conducted by scholars who have different political ideologies, who do not agree. For example, George and I are somewhat different in our own assessment of the impact of immigration on society and the labor market. This made us a very strong team. It meant that we could focus on the scientific process and try to get to the best, most reliable answer. Crucially, we need to never rely on single studies or single science teams. Before we declare evidence of anything, we need many studies. We should not just rely on single papers or scholars. We need dozens or hundreds of studies on a topic conducted by inter-disciplinary and inter-ideological teams.

Wishing you Christmas conflict [peace]

As a child, I often spoke of and wished for ‘world peace’. I get and give a lot of wishes of peace during Christmas. We constantly talk about peace as a goal. Sounds nice. But is it?

I don’t actually know what peace is, other than a lack of war. War is conflict. So peace is essentially a lack of conflict. This means that peace cannot be measured without conflict. We need conflict to understand and define peace.

If we flip this around and ask what the world is like without conflict, it is unimaginable. Let’s assume society has all the resources it needs and has the technology to handle any natural disasters – pandemics, earthquakes, typhoons, etc. There is no hunger and there is no lack of good paying jobs.

Now in this utopia people are still human. They have a desire for companionship, sex and in most cases to produce offspring. How many of these humans will have all their companionship, sex and offspring desires met? Probably many of them at times, but none of them all the time. Even with the ideal rate of companionship, sex and children, humans experience deficiencies.

If you give a human all the sex they could possibly want, they lose the desire for it. The dopamine that is pushing them hard to go get it, simply disappears. The adrenaline that fires during sex starts to wane – sex becomes so common that they build up a tolerance and/or simply produce less.

At a more basic, daily level, people experience highs and lows in emotion and experience. They make mistakes in judgements, they get tired. They let each other down, they piss each other off. They seek comfort in others, the look for more. Its a survival instinct.

Humans are programmed for interpersonal conflict.

They are not 100% trustworthy, because they can’t be. They make decisions to protect their family and children’s interests above the interests of society. This might mean lying or cheating, in case they feel their children are in danger. Humans are programmed this way for good reason. They are programmed for survival.

This survival programming leads them to prioritize their families. These families over time form larger communities who prioritize their own interests to collectively increase their chances of survival. It becomes a Pink Floyd song.

Us, and them
And after all, we’re only ordinary men
Me, and you
God only knows it’s not what we would choose to do

Humans form groups, prioritize those groups over others, defend those groups against others. They will try to extract resources from the other groups.

Christmas itself is a holiday named and celebrated widely based on certain human groups power and resources superseding others.

Humans are programmed for intergroup conflict.

Its who we are.

Peace then is a temporary state where these conflicts are not active. Respite from the human experience. An appreciation for what we have. A delusion to enjoy because conflict is actually constant in the background.

I wish you conflict in your life. In doing so, I am wishing you the peace that comes amidst the conflict inherent in our humanness. And in doing so I am wishing you the peace of mind that is possible through acceptance of our humanness.

To respectfully and ethically decline to support Elsevier

Just over 5 years ago, I decided I would no longer peer review for or publish with Elsevier. In addition, I would try not to support any of their products. This was a choice enabling me to be more consistent with my own ethical values and support of open science.

This is was a difficult decision to make, because for-profit publishing companies have all engaged in questionable behaviors in pursuing economic gains. Also, it means saying ‘no’ to colleagues who are editors and could really use my expertise for a peer review or book chapter contribution. Therefore, I have created a brief letter template to decline such offers. It provides some informative links, encourages the recipient to pursue their own opinion and avoids overreaction via defamation or fear-mongering. Elsevier is known to attack with lawyers, so it is also important that I protect myself and my associates when declining such offers.

Dear [person],

I apologize but I cannot in good faith [write a peer review / book chapter] for an Elsevier publication.

I realize that publishers are ‘for profit’ enterprises, and balance the needs of science with their own pursuit of economic success; however, in my experience Elsevier has consistently proven to be an exceptionally unethical company with practices antithetical to the pursuit of knowledge. Some of the more awful practices have been to sponsor arms fairs[1], advocate intensively against open access[2], create seemingly scientific journals and then sell space in those journals for companies to print bogus articles that support their products[3], purchase and copyright or paywall scientific tools[4], charge astronomical fees for journal subscriptions[5], and over-harvest my personal information[6]. Therefore, if I can help it, I do not peer review or publish with Elsevier anymore.

Please do not take this personally. This letter expresses my own personal opinion based on my own experiences. My opinion and resulting preferences do not reflect that of my employer or any colleagues with whom I have collaborated. I encourage you and each person to investigate for profit publishers on their own and make their own informed decisions.

Sicerely,

[Author]

[1] https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(07)60488-7/fulltext

[2] https://en.wikipedia.org/wiki/Elsevier#Criticism_of_academic_practices, https://www.theguardian.com/science/political-science/2018/jun/29/elsevier-are-corrupting-open-science-in-europe, https://www.lub.lu.se/en/find/lubsearch/epublications/new-agreement-elsevier/sweden-stands-open-access-cancels-agreement-elsevier (attacks on ResearchGate and Sci-Hub) https://www.science.org/content/article/publishers-take-researchgate-court-alleging-massive-copyright-infringement, https://www.nature.com/articles/nature.2017.22196

[3] https://www.the-scientist.com/elsevier-published-6-fake-journals-44160, https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(09)61404-5/fulltext

[4] Social Science Research Network – purchased and turned into a place for paywalls; Mendeley – purchased and converted into a product that is not (easily) interoperable with open source products like Zotero; tries to patent a peer review tool to a payment step into the scientific process https://www.eff.org/deeplinks/2016/08/stupid-patent-month-elsevier-patents-online-peer-review.

[5] https://www.science.org/content/blog-post/california-tells-elsevier-take-hike, https://www.thenation.com/article/society/neuroimage-elsevier-editorial-board-journal-profit

[6] https://eiko-fried.com/welcome-to-hotel-elsevier-you-can-check-out-any-time-you-like-not/

Open science. Back to basics

Are we really sharing all steps in our research? And are we making clear which steps we cannot share and why? There are always hidden steps that influence our research design and reporting.

Ask yourself this question: If someone else were to pick up one of your studies, most likely a published paper reporting the results of one of your studies,  would they be able to understand and reproduce everything that you did?

Are there things that can’t be exactly reproduced, like privacy-protected data, or subjective, interpretive, or ethnographic experiences in your study? If so, have you pointed this out for the reader?

These sound like trivial questions.

Anyone outside of science would think these things must be true. They would think that of course a scientist, a researcher, would report everything they did in their research. Isn’t that their job? How else should science look? What else are scientists doing other than doing research and reporting that research?!

Maybe your answer to this question is, ‘yes, I have reported everything I did’. You truly believe that people could follow everything you’ve done, but is this really the case?

I work in the area of statistical analysis, mostly, and in this area, there is code required to run statistical analyses, and that code is the basis of my research design, assuming I didn’t collect the data myself.

So, I have secondary data, I already have data, and then I analyze it, and my code is the basis for that analysis. Well, it turns out there’s a lot of things in my code that others would not be aware of without the code. So, this makes code sharing absolutely essential. And we’ve come a long way in this area.

Economics and political science especially, in general, have strong code-sharing norms for people who do statistical analysis. Sociology is a little further behind. Probably one of the main reasons for this is that journals have not adopted strong policies of code sharing in sociology, unlike in political science and economics. Either way, as a researcher, I should be responsible for making my results reproducible, but let’s think of an even more hidden case.

In your statistical analysis, did you do things that are not in the code? Now, if any researcher were to answer ‘no’ to this question, I would not believe them, including myself. There are always things that we do, that we then take out of the code because they seem redundant or unnecessary or, in some cases, don’t make our study look as good as we want it to look. One of those things is to run models that we don’t report.

Now, that would be fine if we just happened to run a model on accident or had some other glitch or mistake. But a lot of the models we run are intentional, and the reason we don’t report them is that they produce results that we don’t want. They produce results that don’t support what we think is true or what we want to appear to be true because it’s sexy, because it’s something that would be publishable.

But the question to ask is then: have any of those extra models that I’ve run or you’ve run been used to inform decisions that we make in terms of what models to run next, how to recode any variables in those models, and what to report on, in the paper and in the code? And if the answer is yes, then that’s part of the research design, and should be reported.

Now, this is not the norm at all. I don’t know any discipline where this is the norm, and I don’t know people who are trying to make this the norm. This is a hidden area of the open science movement for the most part, although we have research now that overwhelmingly suggests that the findings that we’re reading about, that people are reporting on, are probably selected (thanks to p- and z-curve analysis). They’re probably a subset of findings that are not just a random subset of the models that researchers intentionally ran, but a selected subset in the sense that they all point in a certain direction or they all have a larger size or there’s something about them that makes them specially selective – and this selection process reduces the reliability of science.

Therefore, when I say back to basics, I actually mean the basics of research, not the basics of the open science movement. I mean reporting everything in the research process that influences the results. That would truly be open science.

Image credit: Nate Breznau’s own photo

Trump is my president. Let’s talk

In a democracy, the democratically elected president is the president. Citizens who reject this, promote an agenda against democracy itself.

I am a U.S. citizen and therefore Trump is currently as of October 2025, my president. This is the reality whether I like it or not. Trump is the democratically elected president of the United States. He won an election unanimously that was relatively uncontested by observers, and for the first time ever he is a Republican candidate who would still have won even if all non-voters had voted.

A common slogan in recent anti-Trump protests is “Not my president“. The slogan “Not my president” is a symbolic rejection of democracy. The (mostly Democrat voting) protestors reject the results of a democratic process. The implications of this protest message, are that the protestors would suspend democracy to have a different outcome if they could. The reason that it is ‘OK’ in this case to suspend democracy would be because Trump is “wrong”, “evil” or  “unhinged“. These are the same protestors who bewail the decline of democracy BTW.

The mass January 6th protest against the 2020 election results, and insurrection by those who tried to or forcibly enter the Capitol, is basically the same. It was based on rejection by (the mostly Republican voting) protestors of the Democratic election result of Biden as president. It, like the “Not my president” slogan, was a rejection of the Democratic system itself. Because one group did not like the outcome. The implications are that we should throw out democracy to have a different outcome.

The two sides who are regularly protesting one another because they see democracy as under threat, engage in protest messages and actions that themselves threaten democracy. Protest is a foundational element of democracy – a human right in my opinion. But I a skeptical of these democratic protests that seem to hypocritically suggest we suspend democracy. 

We, the U.S., are in a state of historically unprecedented partisan division. Each side claims moral authority, and in doing so rules out the opinions and preferences of the other side as illegitimate and wrong.

This is not a formula for democracy nor a cohesive society. It eliminates the possibility of communication. And communication is one of two ways to resolve conflict – the other being domination (conflict, elimination of the opposing position).

A necessary condition for successful communication is that all opinions are valid. That each person has a legitimate right to have any opinion. That it is OK to disagree with each others’ opinions, but that it is not OK to claim moral authority that others’ opinions are wrong (nonsense, evil, unethical, stupid, insane, etc.). By ‘not OK’, I mean, not OK to be shared in a communicative setting, leading to communication breakdown and conflict (mobilization, domination, war, etc.).

If someone says that they think black people are dangerous, untrustworthy and criminals; or that women and men should fulfill certain roles in society and families as dictated by genes and chromosomes, or by God; or if they say that children should not be allowed to change gender, especially not take puberty-delaying drugs. I will only effectively communicate with them if recognize their opinions as valid and provide a safe space for them to share these opinions. To try and understand why they have them, and to seek a way forward that doesn’t require them to give up their opinion because I say so.

If someone says that black people have been systematically oppressed, that race is not an essential feature of human beings and is instead socially constructed; or that it is right for the government to promote equality; or that we should ignore gender in social life and treat men and women identically in all situations. I again will only make progress in communication when recognizing these opinions as valid and create a safe space for them to be shared.

If I am not willing to sit that my opinion has no ultimate truth to it, or that it is morally or intellectually superior to others’ opinions, I am not supporting democratic society, but instead my own vision of a society where I dictate opinion for others. That’s not democracy. So I question what we are doing with claiming when we claim that our position is right and others are wrong, and thinking that this is a democratic process.

Image AI credit

Questionable research practices from the practitioners’ perspectives

Discussions and narratives from four behavioral and social scientists about their experiences learning, observing, and doing things in the course of research, and communicating that research, that confound, and even contradict, the goals of science.

With Monica Gonzales-Marquez, Priya Silverstein and Eike Mark Rinke.

In preparation for Metascience 2025, I put together a panel with three other researchers called “Questionable Research Practices from the Perspective of the Researcher: Understanding Perverse Incentives using Autoethnography”. We used our own discussions in online meetings and individually as recorded narratives as content for this panel. The impetus was that much metascience points accusatory fingers at problems in science, thus supporting a culture of fear. Researchers comply to avoid scrutiny, and not necessarily out of an intrinsic motivation to do good science.

When successful and meaningful science is measured entirely by publication in a ‘high impact’ journal and high citation counts, the drive to do science is transmuted into a laser-like focus on publishing. The intrinsic motivation to contribute to the creation of scientific knowledge becomes confounded with publishing, while simultaneously deprioritising the robust, pedantic, methodical, humble labor involved in doing good research. The scientific method gets confounded with the mechanics of publishing and achieving “standing” in the scientific community.

We hope that by revealing our own experiences in the world of ‘publish-or-perish’, and how it has pushed us towards questionable research practices (QRPs), we might generate intrinsic motivation in others to dispassionately examine their own scientific practices.. By looking back through our histories in academic work and sharing them, we also expect to increase our own intrinsic motivations to be ever vigilant, and to learn to always privilege scientific integrity over publishing.

Confounding: Publication = Science

A major theme in our narratives is that science has been fully confounded with publishing. For some of us, the pressure to publish, and the toxic atmosphere and interactions pushed us out of academia entirely. For others, we internalized the publishing norm, convincing ourselves that we were doing impactful science, when we were actually just doing impactful publishing – which does almost nothing to alter collective human knowledge, promote social justice or solve other societal problems.

Some excerpts from our narratives:

“[When I started graduate school, science to me at first meant that] “the quality of knowledge produced is … the real value in science and what will be rewarded. [but] That got replaced with this publish thing … How do you know it’s good science? It’s a good publication”

“[Publishing papers is] serving God and the Devil. And researchers are in a position where this is what they’re supposed to do to pay the rent, to advance their careers and eat and all of those things. And the way the system is set up, it’s actually at cross purposes with creating reliable knowledge. Because if you try to do that [create reliable knowledge], well, you’re f#$!ed. Your career doesn’t advance.”

“Junior scholars who start out as idealists in most cases, I think, people who want to do science in the sense of finding about finding truth or understanding the world better, in the process of being socialized into the system [they] learn about publishing. The fetishism of long CVs and long publication lists. So they internalize this and learn … to understand science [as] an activity that’s… That resembles sports competition in which the goal is to … succeed and out outmaneuver your opponents who also sort of compete for those scarce resources, especially permanent positions.”

Questionable Behaviors: Hacking

The pragmatics of doing science in a publish or perish culture is that we either engage in some (often undeliberate) hacking and storytelling, or are sanctioned.

Some excerpts:

“That was my experience just being, being completely shocked when [my supervisor] told me, well, you know, just come up with a story to explain the data. And I’m like but, I have questions and these data go in a different way than this question. ‘Yeah, but otherwise you can’t get published. You have to tell a story.’  And this was just so jarring to me.”

“What comes to mind is the first time that I ever presented a poster… I was being encouraged to basically just think about all the analyses I’d done for my project and pick out the ones that told the best story, make sure that the poster told a really clear story, and it was explicit that it didn’t matter if this wasn’t everything. In fact, that’s normal. That’s why we do so many different things, is so that we can then pick out the ones that make the nice story. And it sounded kind of nice to me. I liked the idea because it, it was like a fun, creative exercise that I thought was part of science.”

“We would really just like put these variables in and adjust them, and certain variables might be highly correlated. I don’t know, let’s say like education and occupation. And so we, you know, we’d probably drop one [because of multicollinearity] if it wasn’t too important to our story. And which one would we drop? We would drop the one that had the least impact. You know, we would drop the one that would keep the X-Y relationship as strong as possible. And every decision was pretty much like that.”

Ego, Power and Personal Struggles

As human beings we are prone to seeking status, material security, community acknowledgement and different things depending on where we are at in life, and who we are personality-wise. Especially when in graduate school, we are in a position of little power in comparison to professors and our supervisors. People who  wield power over others, and push their own ego-centric agendas tend to be quite successful in science. This can lead to abuses of power, ego trips and other toxic behaviors that can diminish junior researchers’ ability to push back against pressures to engage in QRPs and have strongly demotivating effects.

“The way things are supposed to work. Right. Who am I to, to question that as a Ph.D. student?”

“I think that those who are humble, those who are keenly aware of their own limitations, the limitations of the work that not only they, but all of all scientists do, may not be the ones who have the highest chance of succeeding in this system.”

“I’d been told to my face that the only reason I was there, that the only reason I got an NRSA, the only reason I gotten a Sage fellowship was because I was a [minority]. I was told this to my face. All of my accomplishments meant nothing. The fact that I brought original ideas and raised important questions, none of that mattered. The only reason I’d gotten in was, according to these people, and these were my peers, other graduate students. This is what they said to my face and what they said behind my back. So it was an absolutely toxic environment.”

“So if it’s not a safe space within academia, we can’t expect the people in academia to behave well. Right. So people are… They’re not safe, they’re feeling unsafe. This creates conflict, fears, anxieties, and then there’s many different ways to deal with that. But I think some of those could be… questionable research practices.”

Intrinsic Motivation

We hope that by speaking out, and normalizing scrutiny of our own experiences of questionable research practices as something valuable, we can help others become intrinsically motivated to do the same. Moreover, we propose that ethnographic narrative may be an underexplored but powerful method to help uncover the causes and motivations of questionable research practices from researchers’ lived experience of “doing science”.

Part of the motivation for this panel is based on one of the authors’ (Nate Breznau’s) own authoethnographic research into my QRPs. He presented preliminary results at the Sociological Science Conference at Cornell (link to slides).

Readers can find the full poster for our presentation at Metascience 2025 at University College London here.

Our future plans are to seek a larger sample of researchers and invite them to a semi-structured narrative sharing process (via recording themselves) to further study QRPs. In particular we hope to target early career researchers to shed light on the current state of science training and supervision experiences. 

The ISA should not ban scientific associations for failing to take political stances.

Science itself is a humanitarian enterprise, because prioritizing science brings with it humanitarianism. But limiting science and its institutions for failing to take political sides in humanitarian crises is unfair, and opens up a Pandora’s box that is infinite in size. It also punishes scientists for not taking actions that could put their lives at risk.

On June 29th, 2025, just one week before its World Forum in Rabat, Morocco, the International Sociological Association (ISA) leadership banned the Israeli Sociological Society (ISS).

The reason is that the ISS did not take “a clear position condemning the dramatic situation in Gaza.” In other words, that they are not taking a position in direct opposition to and protest of their own government. Although there are reasons that some would want to take such a position, in particular the incredible humanitarian crisis in Gaza, it is not fair to punish a scientific organization for not taking this position. In other words, not fair to try and force a scientific organization to mix itself in with politics. I can think of three concrete reasons.

  1. Science itself should prioritize inquiry, not political action. We conduct research and develop theories, that is science. Sociology is the science of society and social interaction. Therefore, sociologists should (continue to) engage with the war and humanitarian crisis unfolding in Gaza. They should try to present facts to answer the questions: What is happening there? Who is affected? Are the actions of the Israeli military in violation of the Geneva Convention? Is this a genocide? To blackmail other scientists to force them to take political action or face consequences to their participation in science and scientific exchange, runs counter to prioritization of inquiry in science.
  2. Scientists taking political positions could be in danger. In particular in authoritarian, or martial law settings, scientists who stand up against their government put not only their scientific enterprise at risk, but also their lives and their families’ lives potentially at risk. The ISA is demanding that members of the ISS take actions that could lead to their Society facing funding cuts or being cancelled altogether. Even worse, the ISA is demanding that Israeli sociologists take a position counter to their government in the middle of one of a radically politically charged historical moments – it is asking them to potentially put their lives as risk.
  3. Where do we draw the lines of where to take political action? If a humanitarian crisis and what appears to some scholars as a genocide are conditions for taking political action on part of the ISA, where do we draw the line? Why does the ISA not ban all Myanmar sociologists from participating because of the situation of the Rohingya? Perhaps even more perplexing are the odd exceptions: The ISS is banned, but what if an Israeli sociologist is not a member, can they participate? Perhaps someone forgot to pay their membership dues this year but was a member before. Can they participate? What about the United States and Germany? These are the greatest allies of Israel, why are they not banned? What about Russia invading Ukraine? These questions are rhetorical, because this is a Pandora’s box that is infinite and quickly digresses into everyone getting banned.

In fairness, there is a humanitarian disaster going on in Gaza. It is tragic for all the innocent bystanders swept up in it. Those who have been kidnapped, killed, harmed by bombing. Its awful to watch. But I am not writing this in regards to any political movement or government. I am writing this to defend science, which I see as my responsibility.

Democracy needs harmony first. Other issues second.

In this post I discuss why sometimes social justice movements must take a backseat in order to prevent societal collapse.

We live in a time of intense division. At least most of us perceive this to be true. Media is smeared with divisive messages, warnings and threats suggesting imminent destruction of society. Why? Because those terrible other groups are destroying it we are told.

Disagreement, discussion and conflict are productive for collective decision-making up to a point. But when emotions and conflicts escalate too far, discussion ends and war begins. Once sides become entrenched, progress comes to a halt and society is at risk of breakdown. The scenario of an internal war breaking out in a rich democratic society might seem far fetched, but keep in mind that it almost happened on Jan. 6th in the United States.

Even if an armed civil war does not happen, there are social, cultural and political wars already raging, especially on social media where the enemy is not seen face-to-face but consumed in the form of media messages and then constructed in our minds. I remember the first time I was accused of being part of a cultural Marxist conspiracy, because I was “woke”. What a wake up call.

Social harmony and integration of different perspectives must be the ultimate

Is complete diversity, equity and inclusion really possible?

It would be ideal if we could heal racism, sexism and other -isms. We in the behavioral and social sciences know how. This would simply require people across all ages to repeatedly throughout the course of their life to take courses and personal time to reflect deeply on their biases, discuss these biases with others who have same and different biases, and actively listen to others’ struggles with oppressive and prejudiced treatments (including the stories of white people feeling mistreated).

Now as a reader I ask you to briefly set aside your ideas about being “woke” or not, what you think is the correct moral imperative for society, and what you know is true about social injustice in the world. I ask you to instead be “awake” with me for a moment. Do you think a scenario where we all just give up our personal priorities and go into some kind of collective harmonic oppression therapy on a regular basis, where we heal from all forms of trauma and overcome social biases? Its ridiculously unrealistic.

People like me who have pushed for social justice have taken this for granted. Just because this would be an effective method to heal racial and other intersectional prejudices and injustices, does not mean it is possible. The general agenda of movements pushing for diversity, equity and inclusion has proven to be too unrealistic for the society we live in.

Read the statement again:

This would simply require people across all ages to repeatedly throughout the course of their life to take courses and personal time to reflect deeply on their biases, discuss these biases with others who have same and different biases, and actively listen to others’ struggles with oppressive and prejudiced treatments (including the stories of white people feeling mistreated).

This is what it would really take to move past the historical oppressions and traumas that are embedded in modern democratic societies, which were mostly founded on certain land owning men having democratic rights and everyone else beholden to them, with some used for slave labor. In other words, these issues run deep, they are historical, they have collective trauma and they required violence to maintain.

No campaign, no curriculum, and no ideology will ever fully convince most people that they are (a) racist at times and (b) that they need to do something about it. And those that are openly racist, certainly will not be willing to take any actions or support spending public money on it. And if you force them, they are even less willing to learn or change. I’m sorry to say, but most people have other things on their agendas. I wish this were not the case, but that’s reality.

This is a critical juncture. The movements pushing for diversity, equity and inclusion wanted to reduce group-based prejudice and discrimination. Not a bad goal in my opinion. These movements took many steps in that direction that could have helped achieve this goal, but it upset many segments of society. They went too far. There was a backlash.

There are still many people in the U.S. who believe that white families should own black slaves. And although such people are rare, it is likely that a majority of people racialized as white believe that persons with dark skin are genetically inferior. That they are essentially a single race with features that they are born with that make them less reliable and potentially dangerous. White people who think this way – the average white person I would guess whether they admit it or not – are triggered by social justice movements, especially those targeting race or gender identity. Especially if they themselves are struggling to make ends meet, telling them they are “the problem” is certain to piss them off.

I wish these things were not true, but this is reality. Such people are triggered and fueled by media fear-mongering playing up the threat of social justice movements. One outcome of this fear was a monumental swing in votes so that the Presidency, House and Senate all went Republican in 2024; and not just Republican Party but a certain brand of politics that is radically opposed to social justice movements that push for diversity, equity and inclusion. The first order of business was then not surprisingly to remove two decades worth of policies to push for social justice.

I say “critical juncture” , because society should go on. We must live together somehow, despite this reality. I say “must” because any other option would require removing certain groups from society. Trying to remove racist white men from society is just as racist as trying to oppress those with dark skin. We cannot remove racism through voting or arguing. It is a feature of society that nobody controls, it just exists.

Most of what I read online and hear from people is how awful things are. Those opposed to the social justice movements for diversity and inclusion see them as awful, and those who support the movements see those opposed as awful. Hate, hate and more hate. I cannot tell you how many of my close colleagues and friends, many with PhDs, constantly state that anyone that voted for the current U.S. president is disgusting, morally vacuous, and fucked in the head. They claim absolute moral superiority without discussion.

This reaction is so strong, that it appears to me that there is a complete lack of respect for other people, or understanding of why others voted the way they did. There is an unwillingness to discuss politics with such people on equal ground in a manner that respects their right to opine and vote for whomever they want. The groups tend to see the other as sick and/or morally inferior. If this remains or deepens in intensity, I will not be surprised if armed conflicts ensue, or perhaps worse, that we democratically vote ourselves into a dictatorship. Some kind of dystopian Hitler 2.0.

Some issues need to take a backseat

Social harmony and integration of different perspectives must be the ultimate priority in a democracy, and especially right now. The key decision-rule when deciding how to proceed in any policymaking or political discussions has to be to first accept and embrace others’ right to think and say how they feel, and vote for what they want. Otherwise, the only other option is war until one side is defeated and removed.

If we are to co-exist, it has become logically obvious to me that some issues that people deeply care about like abortion, public education, public health care, racism, sexism, transgender issues and more – many with profound moral stakes – have to take a backseat to the stability of the system as a whole. Without a stable system, it no longer matters whether abortion is legal because the end of society and democracy means mob rule. Mob rule is the end of the rule of law, where anything goes and people will become slaves of the most powerful gangs. Governance and democracy is there to prevent mob rule in the first place.

To suggest that the intensity of fighting for certain issues needs to be toned down, or to suggest that they might need to be shifted down the priority list is painful to write for me. I have spent my life deeply concerned about race and social justice. I studied African-American Studies and Sociology in college and Sociology and Political Science in graduate school and my focus has been on social inequality. I had not until recently considered that my ideas for society and policies might be simply impossible. If so, it is absurd and potentially dangerous to push for them.

What to do. I need to work on accepting and integrating the fact that people are racist and I (we) cannot change that much into my plans for what would be ideal policies and behaviors in society.

That does not mean prejudice and discrimination are not important nor morally wrong in my opinion, nor that abortion and other issues should not be discussed in society. It means instead that if pushing these issues too hard reduces social stability and harmony so much that society collapses, then they need to be set aside or at least reduced in intensity in the name of harmony.

Above: remains of a once vast and impressive society, governed by an early form of Democracy. Long laid to waste due to a lack of harmony.

Democracy is a container, not a cure

Democracy is not based on agreement. It is also not based on cleansing society of those with opposing viewpoints. It works because people accept that others have different viewpoints, and that it is okay to disagree. It is a system designed to contain conflict, not to eliminate it. That means it only survives when tolerance and harmony are treated as priorities above ideological battles, moral righteousness, and sometimes social justice. We need to admit we disagree but that we are still going to work together despite that.

I personally would prefer to prioritize social justice, but if the cost is the end of democracy, I need to admit that it sometimes is not an option.

Racism exists, religious hatred exists, ridiculous fighting across political parties exists. But no solution to these problems is possible in a society that has collapsed under the weight of its own infighting. There is no justice in war. No equality in tyranny.

Putting harmony first is not a way of ignoring injustice, but as a way of preserving the conditions in which justice, tolerance and knowledge can grow and spread.

Democracy only exists in countries, not all human society

While global governance and international cooperation matter, we don’t have global governance that has any real power. The nation-state remains the most powerful unit in our world. It is where laws are made, where rights are protected or denied, where armies are mobilized, and where media and education systems are anchored.

If a country falls into civil war or dictatorship, nothing else matters. That’s why harmony must be cultivated within each country first. Stable nations become better neighbors. They are more predictable, more cooperative, and less aggressive. So harmony within nations becomes a foundation for peace between them.

Here is another case of certain priorities taking a back seat. It would be ideal if humans acted as one species, one society. If there was a system to enforce equal rights across the globe. But this doesn’t exist. And fighting for it in the modern world system is ineffectual and may lead one to be branded a terrorist. Then it is stupid to push for that in the first place. Impossible policy goals are a waste of time and dangerous. This is a reminder that we need to play the hand we were dealt. This is how things are. What can I do today to support harmony and co-existence. Me asking myself that led to this essay. Call it a manifesto for harmony as a solution if you like. Call it anti-woke. Call it selling out. I accept your criticism openly and would like to talk discuss it with you. And would like to do this as equals interested in co-existence.

Science in survival mode

In this post I argue that unethical scientific behaviors like faking data are symptoms of triggered survival response. Scientists are exposed to repeated stimuli conditioning them to fear that without sexy results and ever increasing status, they will be rejected if not ejected from science.

Scientific research is unreliable. It comes with uncertainty. Whether launching a rocket or measuring racial prejudice, there is uncertainty. We use this uncertainty to make decisions. If the rocket has a 40% chance of exploding, best not to stick astronauts in it. If skin-tone bias of soccer referees is somewhere from none to a lot, it is irresponsible to conclude they are prejudiced.

Investigating uncertainty, is science. Truth and uncertainty are two sides of the same coin, they co-define each other. A problem for humans measuring uncertainty, is that humans are unreliable. Human scientists themselves add uncertainty to the measurement of uncertainty.

Recent studies suggest that somewhere between 25 and 60% of published statistical results cannot be recreated using the materials provided – these measures of uncertainty come with their own uncertainty. It turns out that scientific researchers are doing some really peculiar things to generate uncertainty. Some surveys suggest as many as 9% of scientists faked data at least once in their careers, and that more than half selectively reported findings – a behavior that makes the things they study to appear less uncertain than they actually are.

Why would scientists fake research?

I believe the answer lies in their humanness. Like all animals, they are genetically programmed for survival. They are capable of both rational, reflective decision-making and split second reaction without any thought. Given time to reflect, a human would generally conclude murder is unethical, but simultaneously would not hesitate to kill if it prevented their child from being killed. Murder remains wrong, but to not kill and let a child die is also wrong. It would be irresponsible parenting failing to ensure survival.

Murder is a profound act when a human perceives themselves to be in a situation of life or death. It is a symptom of subconsciously activated defense mechanisms in survival mode. But there are many other symptoms. In order to avoid death, humans, like other primates can engage in deception, disassociation, aggression, manipulation, submission, scapegoating, theft and hoarding. If someone held me at gun point and told me to prove the earth was flat, I would have no problem doing it. The math would work, I would just need to fake a little data

Why would scientist act as if their lives were threatened?

Most scientists have highly valued knowledge and competencies and thus live in situations where their lives are not under threat. At least not as a result of their scientific practice. For the sake of this thought experiment, lets just rule out regularly occurring life-threatening danger as a cause of scientists exhibiting survival mode behaviors. This leaves the perception that they are under threat, as a possible explanation.

It takes only a few stimuli to induce survival mode behavioral changes in animals. Like hearing a frightful noise when seeing an animal. This animal and anything that looks like it become automatic sources of anxiety and fear even without the sound. Imagine being told over and over and over that you have to have an exciting study with powerful results in order to get an academic job after graduate school, otherwise you wasted 3-8 years of your life and probably a large chunk of capital on getting a PhD. Could this alone, without introducing any actual shocks or physical pain, induce Pavlovian fear? I encourage you to go ask any graduate student to answer this that does not yet have such a study.

Now imagine that during graduate school a student invests all their time and resources into an experiment. After it is complete, they hypothesized result, the one sure to be exciting and publishable, is not there. Imagine the horror, the shame, the feeling of failure, the panic. Remember the poor soul who leaped to his death because he misunderstood futures trading and thought he owed three-quarters of a million dollars? That was a triggered survival response. Because dying felt like the only way to ‘survive’ the horror of facing that debt. Imagine if he could have just changed the futures market by adding his own numbers to the market. Would he have done it?

We should not be surprised at all then, when scientists acting out of fear-based survival strategies, fake data. Diederik Stapel faked an entire career of data before being caught. His behavior was self-described as an “addiction”. A common reaction of individuals placed under fear stimuli that are emotionally damaging if not traumatic. The intense pressures and expectations of the academic environment created a context in which he felt compelled to engage in unethical practices to maintain his status and success. It does not make murder or data-faking right, but to not take this as grounds for indicting the scientific rewards system is certainly wrong.

The incentive structures have to change before we can honestly expect the fear-driven pressure to fake, cheat, lie or steal – in order to avoid the experience of loss associated with the common null results that occur when conducting high quality scientific research on radically complex human brains and societies that are frustratingly difficult to measure things in – to go away.