Novel coronavirus pandemic and the limits of open science

German version available.

On January 30th, 2020, the WHO declared a global health emergency based on scientific evidence of a rapidly spreading coronavirus from the SARS family (Sars-CoV-2, and the disease Covid-19). The evidence the WHO used to declare this emergency came almost entirely from Chinese data.

The Chinese data demonstrated an alarming spread rate in January, as shown in Figure 1. Without the Chinese data, there would have been little cause for alarm as all other countries combined had barely 90 known cases in that period, and not a single death.

Figure 1. The spread of Covid-19 leading to the WHO emergency declaration Janurary 30th. Johns Hopkins data.

In early January, the Chinese government took measures to block news and data1 related to the virus; however, Chinese scientists still managed to follow open science practices (updated news on this here) including sharing partial-gene sequence data with the world. This allowed the WHO to take appropriate measures and enabled scientists in Germany to develop tests to identify the novel coronavirus. The German team shared publicly their methods on the WHO website on January 13th. Technology and global communications have evolved to the point where governments can slow but not stop the free flow of information.

Sharing all data and findings is the best form of science, but not always practiced. The Open Science Movement has the goal of changing this. If everyone in the world has equal access to the theory, methods, data and results of all other scientific research, quality and efficiency increases exponentially. This is evidenced in the open science practices behind the global fight against Covid-19 that saved and will save lives, potentially millions of them.

Figure 2 is a simulation predicting how many people would die of the virus in any given country depending on when governments follow WHO recommended operating procedures, as in: issue stay-at-home orders, engage in widespread testing and quarantine both individuals with the virus and those they were in contact with. ‘Day 0’ in Figure 2 is the moment when there are at least 3 symptomatic cases per million people, usually about 2 months after the first case in a country but much faster if several cases arrive at once.

Figure 2. The impact of government intervention in reducing deaths from Covid-19. Source: Gabriel Goh & author’s calculations
(*indicates a predicted death toll)

The reader should keep in mind that Figure 2 is a simplified simulation. The reality of the situation is extremely complex. In particular, governments do not go from normal operations to full lockdown of society in one day, this usually proceeds in stages. Nonetheless, this simulation comes from the best known predictive epidemiology models and helps demonstrate how even one day of indecision can cost thousands of lives.

In response to this outbreak in China and rapid appearance of Covid-19 globally, South Korea followed the WHO’s standard emergency operating procedures. Meaning: Test everyone possible, isolate all cases, restrict travel and gatherings, close non-essential businesses. The virus was contained and only 200 people died. Of course, previous virus outbreaks heightened their preparedness level. Germany was also well prepared given its rapid development of tests, and because they learned from the experience of Italy as Europe’s ‘ground zero’.

Roughly speaking, Italy crossed the ‘Day 0’ threshold in Figure 2 around February 15th. It was the least prepared as a country because it was the first in Europe, and a place where people from around the globe flock as tourists if not football fans. Thus, Italy’s case is not a story of major government failure, also given that there were reasons to be suspicious of the Chinese case.

The ‘Day 0’ threshold came around March 2nd in Germany, and ‘Day 0’ was around March 8th in New York at least on paper. But New York only started testing people with success around March 1st because the CDC’s own test kits released in early February failed. This left plenty of time, in ‘pandemic terms’, to source the accurate tests being deployed in the rest of the world since January. This did not happen in New York or the US as a whole. Thus, New York was caught completely unprepared but not because the virus was a surprise arrival.

When combined with the data from China, South Korea and several other countries, the WHO upgraded the global emergency to a global pandemic on March 12th. New York had issued a state of emergency but only gave stay at home orders as of March 20th. It was not until a week later that most schools were closed and police authorized to enforce these orders (the blue arrow around ‘Day 31’ in Figure 2). Despite massive open science efforts channeled through the WHO, New York and much of the US simply failed to heed obvious scientific evidence and predictions. This is even more shocking because Seattle, not New York, was ‘ground zero’ in the US. Washington State as a whole implemented early and successful emergency measures.

Reviewing the failures of countries, states or cities to immediately take drastic emergency measures before or on January 30th (global emergency) or March 12th (pandemic) is not the subject of this blog post. The world had access to all the same data and knowledge of how to test for the virus thanks to open science practices, the WHO and several partner organizations and websites.

The message I want to convey is that open science is not enough. Its limits are found in governments. In many countries, science has little place in government decision making. This is perhaps understandable in a dysfunctional authoritarian regime where nearly all political decisions are made to maintain and concentrate power. This is certainly a reason that the worst horrors of the virus are yet to come in sub-Saharan Africa and Central Asia. But it is shocking in democracies where there are throngs of scientists and agencies tasked with monitoring and advising the government on what to do to protect its people.

The United States had ample information that Covid-19 was in the US and spreading rapidly, how to design effective tests, and exactly what to do to reduce the spread of the virus and its death toll months before any major actions were taken – the same information Washington State used to stem the spread. But this scientific information, shared at a scale and speed not seen before in world history, was simply not enough.

The Open Science Movement has ethical principles underlying its open access, data, methods and sharing recommendations. It is not just that open science practices make science more reliable and effective; they promote social justice, or scientific justice if you will. When every scientist in the world can access all the information that every other scientist in the world has, there is scientific equality. While rich universities boycott Elsevier, poorer universities cannot even afford a subscription. Thus, open access would bring a global North-South and a endowed v. not-endowed university equality to the world. But it can’t bring justice to those who are potential virus victims.

In the case of the Covid-19 pandemic, open science looked to undermine the bickering and buffonery of governments at first, but it could only ring the doorbell. Some governments simply refused to open the door, to take action. This begs the question if the Open Science Movement needs to adopt principles of political action that extend beyond policies promoting transparency and reproducibility. Does the Open Science Movement need to push governments to adopt administrative, if not constitutional procedures that make governments accountable to scientists in a natural disaster or emergency like a hurricane or pandemic?

I say yes from an ethical stand point. But its not so simple. As soon as we start pushing things like procedural reform, deep-pocketed special interests get involved and it gets ugly. As scientists we are not likely suited to mudslinging and political maneuvering. Not to mention that the more time we spend on lobbying, the less time we have for science. There are some of us with the ability to lead the Movement and influence governments, but most of us are ill-equipped to combat the powers-that-be behind politics.

That brings up the end game question: Is it enough to give the right answers to governments even if they ignore them? Have we done our duty as scientists if we just ‘show up at the doorstep’ and let government officials decide if we get to come in?

1 The original news article was deleted from the Chinese News Agencies’ website but can be found in the Internet Archive.

2 Source: Goh, Gabriel. “COVID Epidemic Calculator“. Day 0 is at least 3 symptomatic cases per million people, meaning there are potentially hundreds infected given the incubation period. Parameters used for predictions: 106 mil. Population, a single initial case, contagiousness per person of 2.2, rate of transmission 0.73, incubation period 5.2 days and mortality rate 2%.

A note from me: I have sought to capture the empirical evidence and historical timeline as accurate as possible, but any errors in this blog post are my own.

.

Behind the specification curve

Analytical errors are a normal part of social science, as David Brady pointed out in the December 2019 IPM Newsletter. Whether conscious, unconscious or simply mistakes, we take certain paths in our research processes. All possible paths constitute researcher degrees of freedom, and they can lead to different findings. In fact, we can intentionally exploit paths to the results we want. Replications can work against this by detecting errors or identifying questionable decisions. We have few replications in sociology, those working in IPM being no exception. Thus just one replication of any given published should be a scientific improvement.

But is it?

Let’s imagine a secondary data study that a replicator wants to reproduce. We now know from a major social science journal, that even when researchers provide their code it rarely runs ‘right out of the box’; replicators often need additional information or materials from the authors (Janz 2015). Moreover, consider an R user trying to replicate Stata code. This requires writing brand new code. Replications with the simple goal of reproduction are not as straightforward as we think! Now let’s imagine a replicator has more complicated goals of testing generalizability or scrutinizing the original study. A replication is as prone to researcher degrees of freedom problems as the original study. This means replications might not be as useful as we hoped, and alone cannot alleviate the ‘crisis of science’.

To be more effective, original studies and replications alike need to focus on model specification. As Muñoz and Young (2018) demonstrate, the cost of running billions of models is roughly zero. Unless they are super-complicated, a computer can do this for us very quickly. Given that researchers accidentally or intentionally report only models supporting their claims, it is a useful exercise to check how alternative specifications might affect the findings. This does not mean running all possible models. Doing so throws models into the results pool that are impossible causally speaking. We need to carefully, logically select models based on plausible research decisions. For example, if a researcher does not weight their survey data, we should ask, ‘why not?’. If a researcher measures poverty as half the median income, we should explore alternative measures.

This figure demonstrates effect sizes (top) and model specifications (bottom) of analyses of the outcome variable “positive reciprocity” (a tendency to pay back favors). This figure was produced by Julia Rohrer (2018) and demonstrates that specification curves are both visually appealing and scientifically useful

Putting together all combinations of plausible decisions we come to a set of models that are theoretically defensible. A major gain is that we can see which decisions have an impact on the results. This gives an indication of where we need to focus our future research. One of the newest ways to do this is p-curve analysis, also known as “specification curve” modeling. These methods were originally introduced as ways to detect p-hacking and publication bias. But we can extend them to standard research and replications, leading to gains in theory. These methods caught on in psychology recently (Rohrer 2018), and it is time for sociology to get behind the curve and move ahead our discipline’s reliability.

References:

Janz, Nicole. 2015. “Leading Journal Verifies Articles before Publication – So Far, All Replications Failed.” Political Science Replication Blog. Retrieved July 22, 2019.

Muñoz, John and Cristobal Young. 2018. “We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness.” Sociological Methodology 48(1):1–33.

Rohrer, Julia M. 2018. “Run All the Models! Dealing With Data Analytic Flexibility.” APS Observer 31(3).

Love is in the error term

A major segment of social science uses formal models and quantitative methods to explain social phenomena. That means they use mathematical symbols to define how they think the world works, and then find data and test it. For example, researchers want to explain social outcomes, like committing a crime, changing jobs, moving or having children. Why do some people do these things and not others? Researchers then speculate that other things cause these outcomes like getting married, having a job or losing a job, how much money a person makes, how old they are and all kinds of other stuff.

Next, researchers change their theoretical ideas into a formal model. This means an equation. Before you stop reading, maybe pause to consider that an equation is just a theory expressed with symbols instead of standard language. For example,

Y = X

and this might represent moving homes (“Y“) is caused by getting married (“X“). Another way of saying this is, Y depends on X or moving depends on getting married.

But seriously, getting married does not always cause moving. Sometimes it does. The correct claim is that moving is a function of getting married. Function means that marriage probably increases the likelihood of moving by some amount. Therefore, researchers add a modifier to X, like the letter b, so if b was 0.3 getting married would increase the likelihood of moving by 30%. People also move when they don’t get married, so researchers add a constant a to account for the likelihood of a person moving for any other reason. So if a was 0.3 then the average person would have a 30% chance of moving at any point.

Y = a + bX

Now if a and bX could be used to perfectly predict whether a person moves or not. The researcher would have made a monumental achievement here. Instead what happens is the researcher goes and observes moving and marriage behaviors. Tries to do this with a random sample of the population of a society. Then applies the above equation to the sample data. Does the equation fit perfectly to the data? No. Never. Researchers must admit that their theories come with uncertainty. Maybe moving depends on the type of home the person always lives in, maybe it depends on whether a couple can afford to move or if it is even a couple or a single parent. There are nearly unlimited things that could cause moving. Also, getting a perfectly random sample, mistakes when observing or coding data, etc. leads to uncertainty in the results. This means the formal equation has to have e, an error term.

Y = a + bX + e

Researchers spend their time trying to reduce what is in the error term. So they are basically tasked with the job of reducing uncertainty. If scientists wants to fly a spaceship to mars they have to reduce all possible forms of uncertainty in the take off, flight path, landing, electronics and so forth. But a social scientist who wants to predict how many people will move, where they will move or which people will move, cannot reduce uncertainty that much. Sure, they can explain moving with job changes, family members getting sick, weather, hobbies, wars, and so many observable things. But they will never fully explain moving. What I’m really saying here is that social scientists will never be able to perfectly explain human behaviors. Its like this:

An Enlightened Physicist describes Sociology

Why is it so hard? If we could just measure all the things that people are doing and thinking we could predict precisely what humans will do next, right? Maybe so, maybe not; but irrelevant because we cannot measure everything about humans. Even if we could measure the precise actions of the network of neurons in one human brain numbering greater than all the stars in the universe, we probably couldn’t measure love. But something that we might refer to as love seems to be a major force in guiding human actions. Both the effect of love:

“Love affects more than our thinking and our behavior toward those we love. It transforms our entire life”

– Thomas Merton

And the lack of love:

“Intense spiritual and emotional lack in our lives is the perfect breeding ground for material greed and overconsumption.”

– bell hooks

(both quotes from hooks 2000, pp. 187 & 105)

These suggest love is a prime mover of human decisions and behaviors. But we don’t know what it is exactly and we have made no attempts, that I am aware of, in any large population surveys to measure it. So,

When we add love as an X variable to our formal model we cannot test anything, because we have not developed instruments to measure it. So it remains in the void of all that variance in Y that we simply cannot explain. Maybe we should start focus some of our effort on that.

hooks, bell. 2000. All About Love: New Visions. William Morrow and Company, New York, NY. ISBN 0-688-16844-2.

Help. We can’t know the data-generating model.

Lack of data-generating models. A problem ransacking macro-comparative research efforts. We got strong data-generating theories. Institutions, politics, materials, procedures, conflicts and symbols help explain macro-level events, like the introduction of social security or the onset of war. But we can’t model these theories as causes of the data we observe. There is far more theoretical complexity than there are countries.

There just aren’t enough countries to provide reliable measures of central tendency given theoretical complexities. Even if our theory were as simple as Y will be significantly different from some value (e.g., zero) given X, we should have at least 10 countries. If we were to actually conduct a power analysis it would tell us we need more like 200 countries depending on how big of a difference we expect in Y given X. I’m just gonna leave aside that countries are not a population in the typical sense (what is a country again exactly? exactly.).

What should we do?

  • Option A: stick to qualitative case studies.
  • Option B: continue running various country-level regressions and ignore the problem.
  • Option C: wait until data from 180+ countries covering a period of 50 years becomes available.

Maybe we just ask a better question. Why do we need to do macro-comparative analysis if we already got good theory?

Answer: Reliability and utility.

Without systematic evidence we don’t actually have theories, only ideas or conjectures. Logic and in-depth case study help define data-generating theories for one country in one time period. So we want to test if this might apply to most countries in most time periods. In testing this we get Pomeranian.

Even with the same variables, regressions on country data over time go all over the place given only tiny changes to the estimation procedure. That was a finding of the CRI. The effect of immigration on social policy preferences could be anything.

To the right: Mr. Summerbottom showcases the size and direction of effects from macro-comparative regressions using the same data. Outliers cropped.

Even if we run all possible model configurations, and even if we consider Bayesian posterior distributions. We still don’t know if we have a correct model specification, because we cannot truly test it. We can only make sweeping statements like, ‘in all possible model combinations, variable X was not significant so it probably does not have an effect’. Too bad we might have some kind of suppression from an unobserved variable!

Don’t drink the water. There is no method that can substitute for human logic. Zombie modeling where the researcher lumbers aimlessly through thousands of models looking for blood doesn’t work. The answer is therefore Answer D, none of the above.

We need new school meta. We need to logically analyze researcher’s decisions. We can’t do much with the limited data that are out there for macro-comparative research. Thus, we need to get down with specifications. By comparing specification curves and researcher decision trees, we can identify the difference between critical and benign. Some programmers offer us R packages for this (thanks Joachim Gassen for rdfanalysis!).

Some models come to the same results regardless of whether the researchers apply weighting, use latent variables or correct for autocorrelation; others not. When we identify the decisions a researcher made that carry the potential to influence the results to the greatest degree, we simultaneously identify where to focus our theoretical work. These decisions cannot be improved by running more models, instead they require, as Andrew Abbott once put it, ‘more sitting in our offices and staring at the wall’. We need to dig in and use logic, reflection and mental energy to improve our theories in small-N macro-comparative research.

Better than a computer. A crowd of researchers.

Crowdsourcing researchers. A new use for an old tactic. When Silberzahn and colleagues asked if football referees are skin-color biased in their assignment of red cards, they brought a new meta to social research. Rather than the typical one research team, one project; they got together twenty-nine teams. All got the same research question and same data. What can we learn?

Hold on. A single academic strapped with programming skills can run just about every possible statistical model configuration. Level up this academic and they can program machine learning routines to tell us almost anything we need to know. So why do we need a crowd of researchers to analyze the same data?

The answer: theory.

Computers do not do theory. Computers crunch data. They can’t tell us where the data came from. Running every possible in an effort to ‘test robustness’ means testing for things that do not exist. Even worse, it means taking the results of tests for things that cannot possibly exist and using them to draw conclusions on the robustness of a test for something that could exist. Throwing all possible variables into a model or ordering variables in every possible configuration will maximize statistical predictions, yes. But what good is predicting something that cannot exist?

An example: Policymakers decide they want to increase the number of females in society. A computer determines that bearing children is a good predictor of being biologically female. Now, imagine that in a statistical model, being pregnant or ever having had biological children explains 75% of the variance in the sex of a given population (its probably about 100% at this point in human history, but lets allow room for error). The policymakers thanks to the computer, conclude that if 1,000,000 more people were pregnant, a predicted 750,000 of them should be female and only 250,000 male, plus a margin of error. Thus, they conclude that getting 1,000,000 people pregnant is likely to increase the number of females by 250,000 in their society, assuming the society is currently 50% female.

Epic fail.

If we want to develop causal theories that provide useful knowledge for societies, human logic is necessary. Rather than having one computer report 8.8 million false positives after running 9 billion different regression models, humans can identify correct, or at least ‘better’, model specifications. In fact, we would not even know they are ‘false positives’ without a human to understand that getting someone pregnant does not turn them into a female, for example.

But relying on the logic of one human, or even one team of humans, is risky. There are researcher degrees of freedom that make their research unreliable. Their prior beliefs, knowledge, experience and context lead to variation in results. These priors lead them along different paths through the garden of research. With the same research question and even the same data researchers often come to different results as demonstrated by the Silberzahn study and our Crowdsourced Replication Initiative (CRI).

Sounds like a meta-problem. So what good is crowdsourcing if we cannot rely on the crowd?

Answer: social interaction.

Crowdsourcing when done with careful planning and central organization, allows participants to comment on, if not deliberate, each others research choices. Suddenly meta-uncertainty turns into the power of meta-logic. Not just one team and their narrow ideas, but a communal debate with diverse inputs. Both the Silberzahn et al study and our CRI involved deliberation and voting on research designs. Combined with the growing area of specification curve analysis, crowdsourcing increases credibility for social research at the level of the population under study and the meta-level of the researchers themselves.

Finally, the relevance of crowdsourcing for collaborative theory construction is an untapped but promising avenue for the future. Crowd research departs from the current system that favors individualism by rewarding novelty of individual researchers. Crowdsourcing instead isa system of consensus building and direct responsiveness to theoretical claims. It could resolve the perpetual problem of scholars, areas and disciplines talking ‘past’ each other. If not consensus, it can identify critical unresolved questions to guide future research.

Crowdsourcing can move us toward open science in the Mertonian sense of a communalistic endeavor. To achieve this, all participants should be co-authors on the project, get to discuss each others’ models and theory, and get to update their own results during the process. We need machines to facilitate this kind of large scale research, but they cannot produce communal, logical exchanges. For that we need to stick with the crowd.

It’s Crowdid

Meta-science, social inequality, methods, open science.

At inception this blog catalogs the wracking process of organizing, administering and evaluating the project and its massive amount of generated data in the Crowdsourced Replication Initiative (CRI).

A technical blog, looking at the ‘nitty-gritty’ of the methods necessary to carry out and present the findings of a project involving 88 research teams, almost 200 researchers, an online deliberation, four survey waves during the process, an experimental condition, a replication, an original research condition, and analysis and meta-analysis of the results.

A research question blog, asking bigger questions about meta-science, researcher reliability, reproducibility, social inequality and crowdsourcing.

In the future…. this blog will address more.

Why blog? The future of science may involve a more interactive, hyperlinked and faster disseminated format. Blogs can facilitate this. In the social science the turnaround time from research findings to published papers is somewhere around 3 to 4 years (counting rejections and R&R’s). Moreover, blogs offer space for discussing things that simply don’t fit in our awfully restrictive 6-12,000 word journal articles.

Open science calls for blogging. Its free, fast and owes no debts. It circumvents the institutional problems of science. As an Open Science Fellow as part of the Freies Wissen program of Wikimedia Germany, a Catalyst for the Berkeley Initiative for Transparency in the Social Sciences (BITSS) and a concerned sociologist, this blog is a contribution to the open science movement.