New Blog

I switched my blogging focus to discussing local public policy in Austin, Texas at Keep Austin Wonky.

Foreign Policy’s Think Tank Index is Flawed

Foreign Policy magazine just put out a ranking of the world’s think tanks.   The methodology that FP used was a survey of “hundreds of scholars and experts.”

The problem with this type of ranking is that it is more of a popularity measure than a measure of resource efficiency.  A think tank with a lot of money is going to do a lot of work, some of it presumably of high quality and some that will in turn have impact on its agenda.  But how do we determine which of these institutions is good at using resources to turn them into influence?  David Roodman of the 15th-ranked Center for Global Development points out that if the rankings were based on productivity, his outfit would do even better.

So are the higher ranked outfits more productive or is it just name recognition?

Using the data from Foreign Policy I created a scatterplot with the Y-axis as the rank and the X-axis as the last annual budget in millions.  IF (big IF!) we accept that the survey of the experts is a legitimate way of ranking policy impact AND that the difference in impact between each rank is linear, then we can measure resource efficiency in generating impact.

In this first scatterplot we include RAND which has a huge budget.

The graph outlines that bigger budgets bring you a higher rank.  The trendline is negative because a higher rank is actually a lower number (e.g. the highest rank is 1, the lowest 15.)  Relative to RAND, places like Brookings, Carnegie, and CFR look pretty resource efficient in that they get significant acclaim at a much lower cost.  Unfortunately, the linear regression has a terrible fit (low R-squared) and annual budget is a statistically insignificant coefficient.  The bottom line is that RAND is such a quirky and important outlier, that you can’t really say the comparison yields any reliable insight.

If we exclude RAND given its unique military ties and projects, we get a much more insightful outcome to the linear regression.

While the equation does not explain a majority of the differences in how the think tanks were ranked, it has much higher r-squared and the annual budget coefficient is statistically significant (p-value of 0.01 through Excel.)  Without RAND, Brookings and most of the think tanks exhibit similar resource efficiency.  The bigger their budgets, the higher their rank.  Thus, they are all mostly equally efficient at translating monetary resources into a good ranking.  Carnegie stands out as having a very high ranking at a noticeably lower cost.  On the other extreme, the National Bureau of Economic Research and Human Rights Watch appear inefficient.  Ro0dman’s CGD is about where you’d expect given the data set.

There are obviously MANY MANY flaws to this way of analyzing efficiency. My intent is to demonstrate how to think about efficiency in the social sector and particularly in the policy advocacy space.  I agree with Roodman that we need to take productivity into account when ranking think tanks and also agree that we need to start prototyping approaches to figuring that out.

You’d Think Hugo Would Know How to Write a Ballot

Why did Chavez lose his pro-socialism referendum? It might not have to do with his political power, as some are probably going to spin. It might have to do with something a lot less glamorous. Namely, how the ballot question was written.

For those that aren’t paying attention or care Chavez was pushing a referendum seeking to change 69 of the 350 articles of the Venezuelan constitution. Many of the provisions were rightfully criticized in the American media. The proposal weakened checks and balances by doing things like allowing Chavez to keep running for re-election or giving him control of the central bank. Other changes were potentially progressive, like the creation of community councils for some budgeting purposes, something that resembles the Brazilian PTs successful experiments with participatory budgeting. Chavez was covering a lot of ground in this attempt to create a more socialist legal and political structure in Venezuela. And my hypothesis is that his electioneering skills were not up to the task.

Chavez narrowly lost with 50.7% voting against.

My hypothesis about why Chavez and his supporters failed is that the referendum was framed and balloted in a confusing way. People weren’t sure of what they were actually voting on, including his supporters. Not only was there probably confusion about the changes actually being sought, but bundling them into a single up or down vote further split Chavistas. Maybe if he had split up the articles into groups or even individual questions, he might have walked away with an overall victory.

Lower turnout could be taken as evidence of this hypothesis assuming confused voters generally don’t take the time to go cast a ballot . Turnout was reported to be 55% whereas turnout in last year’s presidential election was 75% and 70% in the 2004 referendum. Turnout might have been impacted by several factors. Maybe the Chavez’s base of poor citizens is not as excited by the macroeconomic mess they are starting to experience as they were with his early wave of redistributionist social spending. Or maybe the Chavez pullout machine is rusty – there’s evidence of this because several pre-election day polls had the referendum up by as much as eight points. But is seems that there is a good chance that plenty of Chavistas were confused, and some voted “no” because of the all-or-nothing nature of the question.

So “who cares?” you say. Does it matter if I am right?

I’m no fan of Chavez. He’s too statist in his economic policy and has been a terrible coalition builder in Latin America, even though the region is pretty much entirely run by centrists and committed progressives. But it would be a mistake – especially by those of us in the US that remember there’s a bunch of people South of the border that we share this continent with – to take the data from this election as some significant change in the political preferences of the country. Chavez won his last election with 63% of the vote. That’s just a 13 point spread between his election and this referendum. If his supporters stayed home because of a mix of (1) apathy, (2) confusion, and even (3) a change in policy preferences that’s NOT evidence for the US to think change is on the way and engaging Latin America should remain a low priority.

UPDATE:  McClatchy reports:

‘The question we’re wrestling now is whether a tipping point has been reached now with the kind of antics we’ve seen by Chavez of late,” said a U.S senior intelligence official who spoke on condition of anonymity because he wasn’t authorized to discuss the situation publicly.

This is exactly the kind of discussion that could lead to bad decisions if the data isn’t critically evaluated.  If we misunderstand what happened, it might lead to the US adopting a more hostile posture to expedite the “tipping.” Unfortunately, if we misread the loss of support and act on that belief, it could ricochet to America’s disadvantage and give Chavez a nationalism-inspired boost.