A few months before, on OpenGovernment Radio (2/23/10; see link below), we talked about "How to Measure Success in Open Government" with Dave Lewan of ForeSeeResults.com and their recent paper on their survey connection of "transparency" and "customer satisifaction".

http://ustransparency.blogspot.com/2010/02/opengovradio-22310-how-to-measure.html

vr, Stephen Buckley, http://www.UStransparency.com

There is a lot of wisdom in your sentence: "Sometimes, those measures are merely indicators (like our blood-tests) that must be interpreted as parts of a whole."

The BMI is a great example of a quantitative measure and of how such a measure helps assess what one usually really cares about - in this case, one's health, which is qualitative. For instance, to a first approximation, people would say that your BMI should be in a certain middling range for you to be considered healthy. But another part to that whole that is your overall health is your age. And Andrew Weil argues that, as you get older, a little more weight (and thus a higher BMI) may be correlated with longevity. (Of course, we can argue Weil's specific point, and whether longevity is all that, but let's accept it for the moment to think further about measures...)

In this case, we're working backwards from a large, roughly similar population, assessing their longevity (what we really care about), and then checking that against BMI measured earlier.

Relevance for measuring open government:

1. Understand that the measures we want to copy from medicine come from lots of studies over many years, and generally rely on a large number of individuals. (Weil cites a study of 9000 people. Compare that with the few dozen agencies we're looking at to understand Open Government, and you can see the first measurement challenge.)

2. Nail down what we really care about. Some readers will probably prefer to be slim and would be willing to give up a few months of life expectancy. It'd be pointless to argue about the study Weil cites until we've agreed how important slimness is (say for aesthetics or a sense of fitness) to someone's quality of life, vs. longevity at any cost.

For open government, that means (to me), that it would be useful to have a qualitative description of what an ideal open agency would look like, or how we might determine, if we had all the time in the world, whether one agency was more open than another. And then, when we have that nailed down, we can work backwards to measures that are useful shortcuts, since we don't have all the time in the world.

We should also think on what openness gets us. For some, it's a reduction in waste, fraud, and abuse - if we can see how the Federal Government spends money, we can detect and then reduce waste, fraud, and abuse. So we may want the advice of auditors and forensic accountants.

For others, the goal is to support increased citizen engagement and participation - citizens acting as owners or as a board of directors. In that case, we want the advice of management accountants, not financial accountants, cf. http://en.wikipedia.org/wiki/Management_accounting (Back to the medical analogy - a good medical lab can run blood tests and tell you a lot about the state of your health, but it may take a trainer and a nutritionist to get healthy.)

Based on this, I'd circle back to a "Big picture" versions of the opening questions: What does the public want from a more open government? For instance: reduced waste, fraud, and abuse, or a greater understanding of how government operates, or the sense that decisions are made for the good of the country as a whole, not special interests? Each answer leads to different measures.

Chris - Good point about needing to think about what openness gets us. One of our Fellows at NAPA talks about two distinct kinds of transparency: 1) "Fishbowl" transparency, which simply gives the public a glimpse into how things are currently being done, and 2) "Reasoned" transparency, which is really about government EXPLAINING why things are happening as they are. If our goal is simply to give people a look into the fishbowl, a lot of stuff will not make sense -- this has to be about a conversation, and that takes public responsiveness.

On the FOIA backlog as a measure of success, I remember discussing at the workshop the fact that you can't look at the backlog alone to determine if your opengov efforts are working. For example, some might assume that if your opengov efforts are succeeding, the FOIA backlog will decrease because people can find that information without FOIA request. However, you cannot use that as a standalone measure, because of dynamic effects -- your opengov efforts may INCREASE the public's desire for data, which may drive UP FOIA requests. An important point to consider, related to our BMI example.

Daniel - agreed on the vagaries of the FOIA measure and how it relates to BMI.

I like the contrast beween "Fishbowl" transparency and "reasoned" transparency. You could also distinguish between "audit" transparency - what a "Big Four" accountant (or an FBI investigator) needs to understand and certify a business's or agency's books, and "management" transparency - the necessarily more filtered and forward looking information management provides the board of directors.

I think that "management" transparency gets lost in the discussion of efficiency, and that could undermine public responsiveness. Imagine that you've just inherited a medium-sized business from an older relative. You specifically would not want a complete data dump - you'd drown. Assuming that you could trust the business's senior managers, you'd want filtered and structured information that lay out "owner level" choices you'd be required to make. (Of course, you might also want a good accountant to review the books, but let's stipulate for the moment that everything was on the up and up.) So you don't just want data, you want the interpretations of people skilled in that particular business.