When Pilot Projects Go Off the Rails
Calls for evidence-informed policymaking have grown louder in recent decades. Advocates argue that the systematic use of the best available scientific evidence can help us avoid harm and achieve social policy goals while avoiding the deliberate manipulation of scientific evidence to achieve political ends. The Basic Income experiments taking place in Finland, Ontario (Canada), Barcelona, the Netherlands and elsewhere seem an ideal way to generate solid evidence for policymaking.
However, public policymaking is qualitatively different from technical decision making. While clinical decisions in, for example, medicine can be based on well-gathered evidence about the effects of interventions, policymaking usually involves tradeoffs between competing ideas and social values — questions that science alone cannot address.
For example, the decision about whether to license a new childhood vaccine rests on the answers to a set of well-understood questions: Is the vaccine effective at lessening the number and severity of cases of the disease in question? Do the benefits outweigh the potential harms? And (sometimes) is the new vaccine cost effective? These are technical questions with clear answers.
If, however, we want to decide whether or not to make the new vaccination mandatory, an entirely new set of questions must be addressed: Does the state have the right to mandate vaccination or should parents be able to decide for their young children? At what age should a child be asked for assent? Do the rights of vulnerable people who might be infected by non-vaccinated children take precedence over the rights of individual families to refuse vaccination? Should public health take precedence over the rights of religious or cultural minorities? These questions reflect social values and, in a pluralistic society, there is no unambiguous set of answers.
With this distinction in mind, it should come as no surprise that the current crop of basic income experiments has set the stage for some monumental misunderstandings between researchers and policymakers. These issues came to a head when the planned three-year Basic Income Guarantee Experiment in Ontario was cancelled after the election of a new Progressive Conservative provincial government, barely three months after the sampling was complete.
Academic researchers and political actors see policymaking in fundamentally different ways. Trained to focus on answerable questions, academics attempt to depoliticize the science so that the evidence they gather will not be misused or misrepresented. They focus on questions of appropriate sampling and experimental design, and the limits to generalization.
But the scientific evidence that an academic finds persuasive holds little sway with policymakers. Policymakers worry that the experiments themselves privilege certain kinds of evidence and shift the political debate away from the social values and political issues that should be central.
The cancellation of the Ontario experiment reveals the gap between the perspectives of researchers and politicians. On July 31, Minister Lisa MacLeod said the “broken” program isn’t working.
Since no evidence had been collected beyond the baseline survey, baffled researchers pressed her to explain. She complied: “When you’re encouraging people to accept money without strings attached, it really doesn’t send the message that I think our ministry and our government wants to send. We want to get people back on track and be productive members of society where that’s possible.”
That is, whatever evidence might have been collected by the experiment is beside the point; the experiment, by its nature, “sent the wrong message” according to a government convinced that “the best social program is a job.” It appears the government was simply not interested in whether the pilot improved the physical and mental health of recipients, or food insecurity declined, or whether families were equipped to make better financial decisions, whether educational attainment increased or people found better quality jobs.
It seems the very idea of “money for nothing” offended against deeply held political and social values of the new government, which included being seen to enforce employment no matter what the costs or consequences of such a policy might be.
The Ontario experiment had been approved by an ethics board that required participants to give informed consent, so 6,000 participants were enrolled and 4,000 had signed letters promising them a basic income for three years. Then came the media reports that the Minister was cancelling the project, with no mention of how participants would be transitioned off the funding. For a full month, participants had no idea what would happen next.
Researchers, used to the oversight of research ethics boards, expected the Ministry to communicate with participants and to seek the advice of the research ethics board that approved the experiment. The Ministry, then, found itself caught between the expectations of elected politicians used to turning off the funding tap on demand, and a research infrastructure concerned with minimizing harm to human subjects of research experiments.
When academic researchers attempt to engage in policy-relevant research, the most common outcome is that the political establishment pays no attention to what researchers consider persuasive evidence, simply because it doesn’t address the social values that dominate policymaking from the point of view of politicians. The Ontario cancellation shows that, however well-intended, efforts to generate policy-relevant evidence in partnership between academics and policymakers rarely survives an election that brings a new government with a new set of social values to power.
Evidence-informed policy is a laudable goal, but our experience suggests that academic researchers must be wary of partnering with government researchers when the outcomes are politically sensitive.
Governments should explicitly acknowledge their obligations as researchers under the ethics guidelines that govern academic researchers. Politicians are ultimately answerable to the diverse population that they serve. However, civil servants and politicians who engage in research have additional and specific obligations to vulnerable human research subjects.
And finally, researchers should hesitate before they undertake partnered research in which the funding is under political control. An arms-length body designed to oversee the experiment, disperse the funds and publish the results is a much preferred alternative.
Evelyn L. Forget is the author of Basic Income for Canadians: the Key to a Healthier, Happier, More Secure Life for All (James Lorimer and Co.). She is a health economist at the University of Manitoba and a Contributor with EvidenceNetwork.ca based at the University of Winnipeg.