Meta-constructing social theory

Certain hypotheses are constantly tested in social science. The impact of income inequality on health, racial bias on police brutality and public opinion on elections, just to name a few. At some point more tests of the same hypothesis stop contributing to scientific knowledge, and may even harm it by introducing more ‘noise’ into the scientific discourse.

I study social policy preferences and the impact immigration has on them. In this area there has been sustained efforts to test the hypothesis that immigration has a negative impact on support for policies of the welfare state; things related to protecting against risks of aging, unemployment and health. To justify this hypothesis, scholars construct theoretical variations of group dynamics arguments, often drawing on resource competition, nationalism and social identity. Despite claiming to test the hypothesis, the formal models applied to data suggest any number of data-generating processes. They often have little in common other than some measure of immigration and some measure of policy preferences. The results of their tests go in all directions, i.e., a positive, negative or nil effect of immigration. It would appear that the topic is at a standstill, new analyses of the same handful of cross-national survey data sink in the mire. How to break through such a scientific impasse?

In designing the Crowdsourced Replication Initiative (CRI) with co-PIs, Alexander Wuttke and Eike Mark Rinke, we asked researchers to to do research; and we gave them semi-structured tasks and observed them. Specifically they were supposed to come up with the best possible way to test the immigration hypothesis given the same International Social Survey Data source. Although we are currently meta-analyzing the hypothesis test-results (see our virtual APSA poster) to determine which modelling decisions impact the outcomes, we also have a second goal in mind: to discover what is behind the specification curve.

Each research team had to design a best possible test. This is at once a statistical question and a theoretical question. They needed to think carefully about the data-generating process and attempt to recover it in a model. We asked them to write down their research designs after doing this thought exercise, but before analyzing any data. From their researcher choices we can identify where key consensus and disagreements exist about the data-generating model, thus is not only evident in their designs but also in a structured deliberation and voting procedure. This process offers a major advantage over ‘normal’ theoretical discussion and debate among academics, because we have the results that go along with the different modeling choices; and, let’s be honest, when else do over 150 researchers get together and focus on a single hypothesis? By observing this process we can identify where data-generating theories differ and how important these differences are for the results. This will allow us to map where immigration and social policy scholars should focus their theoretical efforts in the future to reduce the most uncertainty, i.e., the largest gains in knowledge.

We have a sound piece of scientific research from Brady and Finnigan (2014) from which we draw our working hypothesis for the CRI crowdsourced researchers: That immigration undermines support for social policies. Brady and Finnigan found little or no support of this hypothesis, at least not in a generalizable macro-comparative sense. This was the launching point for the research of the 77 teams who by now managed to submit replicable results (yes there are still a few out there we are hoping will submit a final model or fix issues we identified in our replication of their models).

Although we are in the process of analyzing the ocean of data generated by this project; a sneak preview offers exciting evidence of the possibility for meta-construction of theory.

Here are two glimpses of what’s to come. One are the deliberation and voting results summarized (Figure 1). The other are differences in definitions of ‘immigration’ (Table 1). We used Kialo, an online structured deliberation platform, to allow participants to discuss the data-generating model after they proposed their own ideas for how to best test the hypothesis. Readers can observe how this deliberation unfolded as we divided the participants into two groups: here and here. Later (after they had the possibility to update their models based on the deliberation) they were given other teams’ models or our own variations on those models to vote on and rank in terms of their appropriateness for testing the hypothesis without having seen the results of those models. Figure 1 quantifies both the Kialo veracity scoring and survey-based voting into one overall scale and then plots the average score of models by their features. Each different color is a discrete set of model features with the zero (y-axis) set to the average support of models choosing an OLS estimator (among the least preferred).

Figure 1. Researcher Preferences for Recovering the Data-Generating Model
“Model” is the hypothesized general impact of immigration on support for social policy. Data and code still being prepared for online sharing, stay tuned.

In Figure 1, it becomes clear looking at the longest bars in each color category that models that incorporate all 5 waves of the ISSP data, include countries of Eastern Europe, include heterogeneous error variation by country-year and year (like a cross-classified model), and incorporate survey sampling weights are preferred over the others. Some of this runs counter to the state of the art. For example, most research follows a logic that major immigrant destination societies – the “Rich 13” and “Rich 17” advanced democracies – should be where “public opinion is likely most influential for the politics of social policy” (Brady and Finnigan 2014:24).

To summarize the motivation for looking across all possible countries, especially Eastern Europe, one crowdsourced researcher put it like this: “Either there is an effect of ‘immigration stock (increase)’ or not“.

Another followed up on this point stating: “To test the general hypothesis we should use as many countries as available and account for variations in GDP and social welfare expenditures in the models.”

These comments demonstrate the majority voice in the CRI that if immigration has a an impact on social policy preferences we should see it across all countries of the globe, not restricting our analysis to only very rich, strong welfare states.

Although Brady and Finnigan and all other research in this area comes to no consensus on whether there is a negative impact of immigration on support for social policy preferences, we should remain skeptical of results if we do not trust the data-generating model. In other words, if our tests do not match what most researchers see as the appropriate theoretical perspective, results are inconclusive and thus uninformative. The deliberation and voting offer us clues where to focus theoretical effort, namely specifying why more countries of the world should (or should not) show a causal effect of immigration on social policy preferences and whether this should (or should not) appear across several decades or only certain times. I am not aware of extensive theory that attempts to tackle these issues. Now is the time to write it!

Even more productive for the possibility of meta-construction of theory is the correspondence between the actual decisions made by the researchers and the subjective and objective outcomes of those decisions. Again, our results are in progress, but we offer a snapshot in Table 1 of different ways the researchers chose to measure immigration as their main hypothesis test variable (1 out of dozens of model decisions to compare). In the first row, 67 out of 77 teams used a “Stock of Foreign-Born” measure in at least one of their models, and 27% of their models using the “Stock” variable showed support of immigration having a negative and significant statistical impact on support for social policy at p<0.05.

Table 1. Crowdsourced Researcher Decisions, Deliberations and Results.
Five different measurement strategies for the immigration test variable.

In the column ‘Positive Test Result Rate’, we see that the ‘Difference’ between “Stock” models (referenced as [1] in Table 1) and those instead using “Flow” to measure immigration models (referenced as [2]) is 3.6. In other words, “Stock” models arrive at support of the hypothesis 3.6 percentage points more than “Flow” models, all else equal. “Stock” models were not more or less popular than “Flow” models, with the average vote score of 0.43 on a scale of 0 (worst) to 1 (best equipped to test the hypothesis) versus 0.45 for “Flow”.

The values in bold indicate that “Change in Flow” models (those measuring derivatives of “Flow”) were among the most popular in the voting process. So the rate of change of the flow of immigrants is seen as an important component in testing this hypothesis. Interestingly, these models were 4 percentage points more likely than “Stock” and “Flow” models to support the hypothesis. When measuring immigration as specific to certain outgroups (from Muslim-majority countries, non-Western countries or refugees), the “Flow” of these various ‘Outgroups’ was seen as more popular than “Stock” of ‘Outgroups’ by a large margin, but the results were over 10 percentage points less supportive of the hypothesis.

What can we learn from this. We argue that a full analysis of the massive range of modeling decisions will give us a guide to move this entire research area forward. Some other decisions for example were different social policy domains, whether ethnic and fractionalization is the ‘real’ cause of the ‘immigration’ effect, construction of latent social policy preference measures, whether or not GDP and unemployment are part of the data-generating assumptions just to name a few out of hundreds. We are only scratching the surface here, but it seems that observing researchers make research decisions, deliberating them, voting and making final choices, we will gain immense knowledge as to where better theory is necessary. As such we see meta-constructing of social theory as a promising avenue for social science. This would be the concept of theory designed replication writ large.

Die ‚Novel Coronavirus‘ Pandemie und die Grenzen von Open Science

Deutsche Übersetzung von Novel coronavirus pandemic and the limits of open science (8. April 2020).

Am 30. Januar 2019 erklärte die WHO einen ‚Global Health Emergency‘ basierend auf Hinweisen auf ein Virus, das sich schnell verbreitet. Das Coronavirus aus der SARS-Familie (Sars-CoV-2, und die Krankheit Covid-19). Die Beweise, mit denen die WHO diesen Notfall erklärte, stammten fast ausschließlich aus chinesischen Daten.

Die chinesischen Daten zeigten im Januar eine alarmierende Ausbreitungsrate, wie in Abbildung 1 dargestellt. Ohne die chinesischen Daten hätte die WHO wenig Anlass zur Sorge gehabt, da in allen anderen Ländern zusammen kaum 90 Fälle bekannt waren und kein einziger Todesfall.

Abbildung 1. Die Verbreitung von Covid-19, die zur Notstandserklärung der WHO am 30. Januar führte. Johns Hopkins Daten.

Anfang Januar ergriff die chinesische Regierung Maßnahmen, um Nachrichten und Daten1 im Zusammenhang mit dem Virus zu blockieren. Trotzdem gelang es chinesischen Wissenschaftler*innen, offenen wissenschaftlichen Praktiken zu folgen, einschließlich des Teilens partiell-genetischer Sequenzdaten mit der Welt. Dies ermöglichte der WHO, geeignete Maßnahmen zu ergreifen, und befähigte  Wissenschaftler*innen in Deutschland, Tests zur Identifizierung des ,novel Coronavirus‘ zu entwickeln. Das deutsche Team veröffentlichte seine Methoden am 13. Januar auf der WHO-Website. Technologie und globale Kommunikation haben sich zu einem Punkt entwickelt, an dem Regierungen den freien Informationsfluss verlangsamen, aber nicht stoppen können.

Die gemeinsame Nutzung aller Daten und Erkenntnisse ist die beste Form der Wissenschaft, wird aber nicht immer praktiziert. Die Open Science Movement hat das Ziel, dies zu ändern. Wenn jeder auf der Welt gleichermaßen Zugang zu Theorie, Methoden, Daten und Ergebnissen aller anderen wissenschaftlichen Forschung hat, steigen Qualität und Effizienz exponentiell an. Dies zeigt sich in den offenen wissenschaftlichen Praktiken hinter dem globalen Kampf gegen Covid-19, die Leben retten und retten werden, möglicherweise Millionen von Leben.

Abbildung 2 ist eine Simulation, die vorhersagt, wie viele Menschen in einem bestimmten Land als Ergebnis des Zeitpunkts der staatlichen Intervention an dem Virus sterben würden. Intervention heißt wann die Regierungen die von der WHO empfohlenen Vorgehensweisen befolgen, wie z. B.: Anweisungen für den Aufenthalt zu Hause, Durchführung umfassender Tests und Quarantäne für diejenigen, die positiv auf das Virus getestet wurden und die, mit denen sie in Kontakt waren. “Tag 0” in Abbildung 2 ist der Moment, in dem mindestens 3 symptomatische Fälle pro Million Menschen auftreten, normalerweise etwa 2 Monate nach dem ersten Fall in einem Land, aber natürlich viel schneller, wenn mehrere Fälle gleichzeitig auftreten.

Abbildung 2. Die Auswirkungen staatlicher Interventionen auf die Reduzierung der Todesfälle durch Covid-19. Quelle: Gabriel Goh, und eigene Berechnungen des Autors (* vorhergesagte Todesfälle)

Der Leser sollte bedenken, dass Abbildung 2 eine vereinfachte Simulation ist. Die Realität ist äußerst komplex. Insbesondere gehen die Regierungen nicht an einem Tag vom normalen Betrieb zur vollständigen Stilllegung der Gesellschaft über, dies geschieht normalerweise schrittweise. Diese Simulation basiert jedoch auf den bekanntesten Modellen der prädiktiven Epidemiologie und zeigt, wie selbst ein Tag der Unentschlossenheit Tausende von Menschenleben kosten kann.

Als Reaktion auf diesen Ausbruch in China und das rasche Auftreten von Covid-19 weltweit folgte Südkorea den standardisierten „Emergency Operating Procedures“ der WHO. Das heißt: möglichst viele Personen testen, alle Fälle isolieren, Reisen und Versammlungen beschränken, nicht notwendige Geschäfte schließen. Das Virus war eingedämmt und nur 200 Menschen starben. Natürlich haben frühere Virusausbrüche in Südkorea die Bereitschaft verbessert. Ebenso war Deutschland gut vorbereitet, weil es schnell Tests entwickelt hatte  und weil es aus den Erfahrungen Italiens als Europas „Ground Zero“ gelernt hatte.

Grob gesagt hat Italien um den 15. Februar herum die Schwelle für „Tag 0“ in Abbildung 2 überschritten. Als Land war es am wenigsten vorbereitet, weil es das erste in Europa war und ein Ort ist, zu dem Menschen aus der ganzen Welt als Touristen, wenn nicht als Fußballfans, strömen. Somit ist der Fall Italiens keine Geschichte eines großen Versagens der Regierung, auch da es Gründe gab, dem chinesischen Fall misstrauisch gegenüberzustehen.

Die Schwelle für “Tag 0” lag in Deutschland um den 2. März herum, und “Tag 0” war um den 8. März herum in New York, zumindest auf dem Papier. New York begann jedoch erst am 1. März mit dem erfolgreichen Testen von Personen, da die Anfang Februar veröffentlichten CDC-eigenen Testkits fehlschlugen. ‘Tag 0’ in New York war wahrscheinlich Mitte Februar oder früher. Dennoch hätte New York aus „pandemischer Sicht“ noch viel Zeit gehabt, Maßnahmen zu ergreifen. Der Rest der Welt hatte seit Ende Januar, dank des offenen Datenaustauschs auf der WHO-Website, genaue Tests durchgeführt. Dies geschah jedoch weder in New York noch in den USA als Ganzes. So wurde New York völlig unvorbereitet getroffen, aber nicht, weil das Virus überraschend aufgetaucht  war.

In Kombination mit den Daten aus China, Südkorea und mehreren anderen Ländern erklärte die WHO am 12. März, dass der globale Notfall nun eine „Global Pandemic“ sei. New York hatte den Ausnahmezustand verhängt, aber erst ab dem 20. März Anordnungen für den Aufenthalt zu Hause erteilt. Erst eine Woche später wurden die meisten Schulen geschlossen und die Polizei autorisiert, diese Anweisungen durchzusetzen (der blaue Pfeil um „Tag 31“ in Abbildung 2). Trotz massiver offener wissenschaftlicher Bemühungen, die durch die WHO kanalisiert wurden, haben New York und ein Großteil der USA offensichtliche wissenschaftliche Beweise und Vorhersagen einfach nicht beachtet. Dies ist umso schockierender, als Seattle und nicht New York in den USA „Ground Zero“ war. Der gesamte Bundesstaat Washington hatte frühzeitig und erfolgreich Sofortmaßnahmen ergriffen.

Die Überprüfung des Versagens von Ländern, Staaten oder Städten, vor oder am 30. Januar (globaler Notfall) oder 12. März (Pandemie) sofort drastische Notfallmaßnahmen zu ergreifen, ist nicht Gegenstand dieses Blogposts. Dank Open Science Praktiken, der WHO und mehrerer Partnerorganisationen und Websites hatte die Welt Zugang zu denselben Daten und Kenntnissen darüber, wie man auf das Virus testet.

Die Botschaft, die ich vermitteln möchte, ist, dass Open Science nicht ausreicht. Ihre Grenzen liegen in den Regierungen. In vielen Ländern hat die Wissenschaft wenig Platz in der Entscheidungsfindung der Regierung. Dies ist vielleicht in einem dysfunktionalen autoritären Regime verständlich, in dem fast alle politischen Entscheidungen getroffen werden, um die Macht aufrechtzuerhalten und zu konzentrieren. Dies ist sicherlich ein Grund dafür, dass die schlimmsten Schrecken des Virus in Afrika südlich der Sahara und in Zentralasien noch bevorstehen. Aber es ist schockierend in Demokratien, in denen es eine Schar von Wissenschaftler*innen und Agenturen gibt, die die Regierung dabei beaufsichtigen und beraten sollen, was zu tun ist, um ihre Bevölkerung zu schützen.

Die Vereinigten Staaten hatten reichlich Informationen darüber, dass sich Covid-19 in den USA befand und sich schnell verbreitete, wie man wirksame Tests entwirft und was genau zu tun ist, um die Ausbreitung des Virus und die Zahl der Todesopfer zu verringern, Monate vor dem Ergreifen größerer Maßnahmen – dieselben Informationen, die der Staat Washington zur Eindämmung der Ausbreitung nutzte. Aber diese wissenschaftlichen Informationen, die in einem Umfang und einer Geschwindigkeit geteilt wurden, die in der Weltgeschichte noch nie zuvor gesehen wurden, reichten einfach nicht aus.

Die Open Science Movement hat ethische Grundsätze, die ihrem offenen Zugang, den offenen Daten, den offenen Methoden und Empfehlungen zum Austausch von zugrunde liegen. Es ist nicht nur so, dass offene wissenschaftliche Praktiken die Wissenschaft zuverlässiger und effektiver machen. Sie fördern soziale Gerechtigkeit oder wissenschaftliche Gerechtigkeit, wenn Sie so wollen. Wenn jede*r Wissenschaftler*in auf der Welt auf alle Informationen zugreifen kann, über die jede*r andere Wissenschaftler*in auf der Welt verfügt, besteht wissenschaftliche Gleichheit. Während reiche Universitäten Elsevier boykottieren, können sich ärmere Universitäten nicht einmal ein Abonnement leisten. Open Access würde also der Welt eine globale Nord-Süd und eine dotierte vs. nicht dotierte Universitätsgleichheit bringen. Aber es kann denjenigen, die potenzielle Virusopfer sind, keine Gerechtigkeit bringen.

Im Fall der Covid-19-Pandemie schien die offene Wissenschaft zunächst das Gezänke und die Tiraden der Regierungen zu untergraben, konnte aber nur an der Tür klingeln. Einige Regierungen weigerten sich einfach, die Tür zu öffnen und Maßnahmen zu ergreifen. Dies wirft die Frage auf, ob die Open Science Bewegung politische Handlungsprinzipien verabschieden muss, die über Maßnahmen zur Förderung von Transparenz und Reproduzierbarkeit hinausgehen. Muss die Open Science Bewegung die Regierungen dazu drängen, administrative, wenn nicht verfassungsrechtliche Verfahren einzuführen, die die Regierungen bei einer Naturkatastrophe oder einem Notfall wie einem Hurrikan oder einer Pandemie den Wissenschaftler*innen gegenüber rechenschaftspflichtig machen?

Ich sage ja aus ethischer Sicht. Aber es ist nicht so einfach. Sobald wir anfangen, Dinge wie Verfahrensreformen voranzutreiben, werden tiefsitzende Sonderinteressen einbezogen und es wird hässlich. Als Wissenschaftler*innen sind wir wahrscheinlich nicht für Schlammschlachten und politisches Manövrieren geeignet. Ganz zu schweigen davon, dass wir umso weniger Zeit für die Wissenschaft haben, je mehr Zeit wir für Lobbying aufwenden. Einige von uns haben die Fähigkeit, die Bewegung zu führen und Regierungen zu beeinflussen, aber die meisten von uns sind schlecht gerüstet, um die Mächte zu bekämpfen, die hinter der Politik stehen.

Das wirft die Frage nach dem Endspiel auf: Reicht es aus, den Regierungen die richtigen Antworten zu geben, auch wenn sie sie ignorieren? Haben wir unsere Pflicht als Wissenschaftler*innen erfüllt, wenn wir nur vor der Haustür auftauchen und Regierungsbeamte entscheiden lassen, ob wir eintreten dürfen?

1 Der ursprüngliche Nachrichtenartikel wurde von der Website der chinesischen Nachrichtenagenturen gelöscht, kann aber im Internet Archive gefunden werden.

2 Quelle: Goh, Gabriel. „COVID Epidemic Calculator“. Tag 0 ist mindestens 3 symptomatische Fälle pro Million Menschen, was bedeutet, dass aufgrund der Inkubationszeit möglicherweise Hunderte infiziert sind. Für Vorhersagen verwendete Parameter: 106 mio. Bevölkerung, ein einziger Erstfall, Ansteckungsgefahr pro Person von 2,2, Übertragungsrate 0,73, Inkubationszeit 5,2 Tage und Sterblichkeitsrate 2%.

Ein Hinweis von mir: Ich habe versucht, die empirischen Beweise und den historischen Zeitplan so genau wie möglich zu erfassen, aber alle Fehler in diesem Blog-Beitrag sind meine eigenen. Ich bin dankbar für die Kommentare von Lisa Heukamp.

Novel coronavirus pandemic and the limits of open science

German version available.

On January 30th, 2019, the WHO declared a global health emergency based on scientific evidence of a rapidly spreading coronavirus from the SARS family (Sars-CoV-2, and the disease Covid-19). The evidence the WHO used to declare this emergency came almost entirely from Chinese data.

The Chinese data demonstrated an alarming spread rate in January, as shown in Figure 1. Without the Chinese data, there would have been little cause for alarm as all other countries combined had barely 90 known cases in that period, and not a single death.

Figure 1. The spread of Covid-19 leading to the WHO emergency declaration Janurary 30th. Johns Hopkins data.

In early January, the Chinese government took measures to block news and data1 related to the virus; however, Chinese scientists still managed to follow open science practices (updated news on this here) including sharing partial-gene sequence data with the world. This allowed the WHO to take appropriate measures and enabled scientists in Germany to develop tests to identify the novel coronavirus. The German team shared publicly their methods on the WHO website on January 13th. Technology and global communications have evolved to the point where governments can slow but not stop the free flow of information.

Sharing all data and findings is the best form of science, but not always practiced. The Open Science Movement has the goal of changing this. If everyone in the world has equal access to the theory, methods, data and results of all other scientific research, quality and efficiency increases exponentially. This is evidenced in the open science practices behind the global fight against Covid-19 that saved and will save lives, potentially millions of them.

Figure 2 is a simulation predicting how many people would die of the virus in any given country depending on when governments follow WHO recommended operating procedures, as in: issue stay-at-home orders, engage in widespread testing and quarantine both individuals with the virus and those they were in contact with. ‘Day 0’ in Figure 2 is the moment when there are at least 3 symptomatic cases per million people, usually about 2 months after the first case in a country but much faster if several cases arrive at once.

Figure 2. The impact of government intervention in reducing deaths from Covid-19. Source: Gabriel Goh & author’s calculations
(*indicates a predicted death toll)

The reader should keep in mind that Figure 2 is a simplified simulation. The reality of the situation is extremely complex. In particular, governments do not go from normal operations to full lockdown of society in one day, this usually proceeds in stages. Nonetheless, this simulation comes from the best known predictive epidemiology models and helps demonstrate how even one day of indecision can cost thousands of lives.

In response to this outbreak in China and rapid appearance of Covid-19 globally, South Korea followed the WHO’s standard emergency operating procedures. Meaning: Test everyone possible, isolate all cases, restrict travel and gatherings, close non-essential businesses. The virus was contained and only 200 people died. Of course, previous virus outbreaks heightened their preparedness level. Germany was also well prepared given its rapid development of tests, and because they learned from the experience of Italy as Europe’s ‘ground zero’.

Roughly speaking, Italy crossed the ‘Day 0’ threshold in Figure 2 around February 15th. It was the least prepared as a country because it was the first in Europe, and a place where people from around the globe flock as tourists if not football fans. Thus, Italy’s case is not a story of major government failure, also given that there were reasons to be suspicious of the Chinese case.

The ‘Day 0’ threshold came around March 2nd in Germany, and ‘Day 0’ was around March 8th in New York at least on paper. But New York only started testing people with success around March 1st because the CDC’s own test kits released in early February failed. This left plenty of time, in ‘pandemic terms’, to source the accurate tests being deployed in the rest of the world since January. This did not happen in New York or the US as a whole. Thus, New York was caught completely unprepared but not because the virus was a surprise arrival.

When combined with the data from China, South Korea and several other countries, the WHO upgraded the global emergency to a global pandemic on March 12th. New York had issued a state of emergency but only gave stay at home orders as of March 20th. It was not until a week later that most schools were closed and police authorized to enforce these orders (the blue arrow around ‘Day 31’ in Figure 2). Despite massive open science efforts channeled through the WHO, New York and much of the US simply failed to heed obvious scientific evidence and predictions. This is even more shocking because Seattle, not New York, was ‘ground zero’ in the US. Washington State as a whole implemented early and successful emergency measures.

Reviewing the failures of countries, states or cities to immediately take drastic emergency measures before or on January 30th (global emergency) or March 12th (pandemic) is not the subject of this blog post. The world had access to all the same data and knowledge of how to test for the virus thanks to open science practices, the WHO and several partner organizations and websites.

The message I want to convey is that open science is not enough. Its limits are found in governments. In many countries, science has little place in government decision making. This is perhaps understandable in a dysfunctional authoritarian regime where nearly all political decisions are made to maintain and concentrate power. This is certainly a reason that the worst horrors of the virus are yet to come in sub-Saharan Africa and Central Asia. But it is shocking in democracies where there are throngs of scientists and agencies tasked with monitoring and advising the government on what to do to protect its people.

The United States had ample information that Covid-19 was in the US and spreading rapidly, how to design effective tests, and exactly what to do to reduce the spread of the virus and its death toll months before any major actions were taken – the same information Washington State used to stem the spread. But this scientific information, shared at a scale and speed not seen before in world history, was simply not enough.

The Open Science Movement has ethical principles underlying its open access, data, methods and sharing recommendations. It is not just that open science practices make science more reliable and effective; they promote social justice, or scientific justice if you will. When every scientist in the world can access all the information that every other scientist in the world has, there is scientific equality. While rich universities boycott Elsevier, poorer universities cannot even afford a subscription. Thus, open access would bring a global North-South and a endowed v. not-endowed university equality to the world. But it can’t bring justice to those who are potential virus victims.

In the case of the Covid-19 pandemic, open science looked to undermine the bickering and buffonery of governments at first, but it could only ring the doorbell. Some governments simply refused to open the door, to take action. This begs the question if the Open Science Movement needs to adopt principles of political action that extend beyond policies promoting transparency and reproducibility. Does the Open Science Movement need to push governments to adopt administrative, if not constitutional procedures that make governments accountable to scientists in a natural disaster or emergency like a hurricane or pandemic?

I say yes from an ethical stand point. But its not so simple. As soon as we start pushing things like procedural reform, deep-pocketed special interests get involved and it gets ugly. As scientists we are not likely suited to mudslinging and political maneuvering. Not to mention that the more time we spend on lobbying, the less time we have for science. There are some of us with the ability to lead the Movement and influence governments, but most of us are ill-equipped to combat the powers-that-be behind politics.

That brings up the end game question: Is it enough to give the right answers to governments even if they ignore them? Have we done our duty as scientists if we just ‘show up at the doorstep’ and let government officials decide if we get to come in?

1 The original news article was deleted from the Chinese News Agencies’ website but can be found in the Internet Archive.

2 Source: Goh, Gabriel. “COVID Epidemic Calculator“. Day 0 is at least 3 symptomatic cases per million people, meaning there are potentially hundreds infected given the incubation period. Parameters used for predictions: 106 mil. Population, a single initial case, contagiousness per person of 2.2, rate of transmission 0.73, incubation period 5.2 days and mortality rate 2%.

A note from me: I have sought to capture the empirical evidence and historical timeline as accurate as possible, but any errors in this blog post are my own.

.

Behind the specification curve

{I was asked to write something for the American Sociological Association‘s (ASA) ‘Inequality, Poverty and Mobility’ Section Newsletter; I reprint it here with a kudos to this section for supporting open science despite the ASA’s general closed mindedness on the subject}

Analytical errors are a normal part of social science, as David Brady pointed out in the December 2019 IPM Newsletter. Whether conscious, unconscious or simply mistakes, we take certain paths in our research processes. All possible paths constitute researcher degrees of freedom, and they can lead to different findings. In fact, we can intentionally exploit paths to the results we want. Replications can work against this by detecting errors or identifying questionable decisions. We have few replications in sociology, those working in IPM being no exception. Thus just one replication of any given published should be a scientific improvement.

But is it?

Let’s imagine a secondary data study that a replicator wants to reproduce. We now know from a major social science journal, that even when researchers provide their code it rarely runs ‘right out of the box’; replicators often need additional information or materials from the authors (Janz 2015). Moreover, consider an R user trying to replicate Stata code. This requires writing brand new code. Replications with the simple goal of reproduction are not as straightforward as we think! Now let’s imagine a replicator has more complicated goals of testing generalizability or scrutinizing the original study. A replication is as prone to researcher degrees of freedom problems as the original study. This means replications might not be as useful as we hoped, and alone cannot alleviate the ‘crisis of science’.

To be more effective, original studies and replications alike need to focus on model specification. As Muñoz and Young (2018) demonstrate, the cost of running billions of models is roughly zero. Unless they are super-complicated, a computer can do this for us very quickly. Given that researchers accidentally or intentionally report only models supporting their claims, it is a useful exercise to check how alternative specifications might affect the findings. This does not mean running all possible models. Doing so throws models into the results pool that are impossible causally speaking. We need to carefully, logically select models based on plausible research decisions. For example, if a researcher does not weight their survey data, we should ask, ‘why not?’. If a researcher measures poverty as half the median income, we should explore alternative measures.

This figure demonstrates effect sizes (top) and model specifications (bottom) of analyses of the outcome variable “positive reciprocity” (a tendency to pay back favors). This figure was produced by Julia Rohrer (2018) and demonstrates that specification curves are both visually appealing and scientifically useful

Putting together all combinations of plausible decisions we come to a set of models that are theoretically defensible. A major gain is that we can see which decisions have an impact on the results. This gives an indication of where we need to focus our future research. One of the newest ways to do this is p-curve analysis, also known as “specification curve” modeling. These methods were originally introduced as ways to detect p-hacking and publication bias. But we can extend them to standard research and replications, leading to gains in theory. These methods caught on in psychology recently (Rohrer 2018), and it is time for sociology to get behind the curve and move ahead our discipline’s reliability.

References:

Janz, Nicole. 2015. “Leading Journal Verifies Articles before Publication – So Far, All Replications Failed.” Political Science Replication Blog. Retrieved July 22, 2019.

Muñoz, John and Cristobal Young. 2018. “We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness.” Sociological Methodology 48(1):1–33.

Rohrer, Julia M. 2018. “Run All the Models! Dealing With Data Analytic Flexibility.” APS Observer 31(3).

Better than a Computer. A Crowd of Researchers.

Crowdsourcing researchers. A new use for an old tactic. When Silberzahn and colleagues asked if football referees are skin-color biased in their assignment of red cards, they brought a new meta to social research. Rather than the typical one research team, one project; they got together twenty-nine teams. All got the same research question and same data. What can we learn?

Hold on. A single academic strapped with programming skills can run just about every possible statistical model configuration. Level up this academic and they can program machine learning routines to tell us almost anything we need to know. So why do we need a crowd of researchers to analyze the same data?

The answer: theory.

Computers do not do theory. Computers crunch data. They can’t tell us where the data came from. Running every possible in an effort to ‘test robustness’ means testing for things that do not exist. Even worse, it means taking the results of tests for things that cannot possibly exist and using them to draw conclusions on the robustness of a test for something that could exist. Throwing all possible variables into a model or ordering variables in every possible configuration will maximize statistical predictions, yes. But what good is predicting something that cannot exist?

An example: Policymakers decide they want to increase the number of females in society. A computer determines that bearing children is a good predictor of being biologically female. Now, imagine that in a statistical model, being pregnant or ever having had biological children explains 75% of the variance in the sex of a given population (its probably about 100% at this point in human history, but lets allow room for error). The policymakers thanks to the computer, conclude that if 1,000,000 more people were pregnant, a predicted 750,000 of them should be female and only 250,000 male, plus a margin of error. Thus, they conclude that getting 1,000,000 people pregnant is likely to increase the number of females by 250,000 in their society, assuming the society is currently 50% female.

Epic fail.

If we want to develop causal theories that provide useful knowledge for societies, human logic is necessary. Rather than having one computer report 8.8 million false positives after running 9 billion different regression models, humans can identify correct, or at least ‘better’, model specifications. In fact, we would not even know they are ‘false positives’ without a human to understand that getting someone pregnant does not turn them into a female, for example.

But relying on the logic of one human, or even one team of humans, is risky. There are researcher degrees of freedom that make their research unreliable. Their prior beliefs, knowledge, experience and context lead to variation in results. These priors lead them along different paths through the garden of research. With the same research question and even the same data researchers often come to different results as demonstrated by the Silberzahn study and our Crowdsourced Replication Initiative (CRI).

Sounds like a meta-problem. So what good is crowdsourcing if we cannot rely on the crowd?

Answer: social interaction.

Crowdsourcing when done with careful planning and central organization, allows participants to comment on, if not deliberate, each others research choices. Suddenly meta-uncertainty turns into the power of meta-logic. Not just one team and their narrow ideas, but a communal debate with diverse inputs. Both the Silberzahn et al study and our CRI involved deliberation and voting on research designs. Combined with the growing area of specification curve analysis, crowdsourcing increases credibility for social research at the level of the population under study and the meta-level of the researchers themselves.

Finally, the relevance of crowdsourcing for collaborative theory construction is an untapped but promising avenue for the future. Crowd research departs from the current system that favors individualism by rewarding novelty of individual researchers. Crowdsourcing instead isa system of consensus building and direct responsiveness to theoretical claims. It could resolve the perpetual problem of scholars, areas and disciplines talking ‘past’ each other. If not consensus, it can identify critical unresolved questions to guide future research.

Crowdsourcing can move us toward open science in the Mertonian sense of a communalistic endeavor. To achieve this, all participants should be co-authors on the project, get to discuss each others’ models and theory, and get to update their own results during the process. We need machines to facilitate this kind of large scale research, but they cannot produce communal, logical exchanges. For that we need to stick with the crowd.

It’s Crowdid

Meta-science, social inequality, methods, open science.

At inception this blog catalogs the wracking process of organizing, administering and evaluating the project and its massive amount of generated data in the Crowdsourced Replication Initiative (CRI).

A technical blog, looking at the ‘nitty-gritty’ of the methods necessary to carry out and present the findings of a project involving 88 research teams, almost 200 researchers, an online deliberation, four survey waves during the process, an experimental condition, a replication, an original research condition, and analysis and meta-analysis of the results.

A research question blog, asking bigger questions about meta-science, researcher reliability, reproducibility, social inequality and crowdsourcing.

In the future…. this blog will address more.

Why blog? The future of science may involve a more interactive, hyperlinked and faster disseminated format. Blogs can facilitate this. In the social science the turnaround time from research findings to published papers is somewhere around 3 to 4 years (counting rejections and R&R’s). Moreover, blogs offer space for discussing things that simply don’t fit in our awfully restrictive 6-12,000 word journal articles.

Open science calls for blogging. Its free, fast and owes no debts. It circumvents the institutional problems of science. As an Open Science Fellow as part of the Freies Wissen program of Wikimedia Germany, a Catalyst for the Berkeley Initiative for Transparency in the Social Sciences (BITSS) and a concerned sociologist, this blog is a contribution to the open science movement.