• If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!


How do we measure performance

Page history last edited by Stephen Buckley 14 years ago


The question for this session was "How do we measure performance in a way that is meaningful to leadership and the public?  What would the public view as good indicators of success in open gov?"


Discussion's highlights as captured by Stephen Buckley:


According to the President's Memo on "Open Government", there are three explicit criteria (which are implicitly measurable):


  • Transparency
  • Participation
  • Collaboration


These three are QuaLitative criteria.  There is no to directly measure them.  That is, each one of them has its own "sub-criteria".  ALL QuaLitative measures (e.g., your general health) are based on QuaNtitative measures (e.g., your heart rate, your chloresterol count, your white-blood-cell count, etc.).  For example: your "body-mass index" (BMI) is a QuaLitative measure that is created from QuaNtitative measures, i.e., your weight divided by your height.


Here's the "junior-high science class" way to understand the difference: QUALitative measures are like "compounds" (H20) made up of "elements" of QUANtitative measures.  It's never the other way around.


And, just as your BMI can not be obtained without measuring the quantities of your height and weight, the level of "Transparency" by the government, for example, can only be obtained by measuring those of its components that are measurable.  Sometimes, those measures are merely indicators (like our blood-tests) that must be interpreted as parts of a whole.


For example:  Are citizens spending more time on your website because they find it more useful, or because they are lost and confused about how to use it?  Obviously, then, a fuller picture requires the consideration of additional metrics.  The trick is to find the best metrics that will provide the best picture of progress (or lack thereof) for "Open Government".


Here are some of the topics/suggestions made by the group's members:


1.  FOIA Backlog - Has it been reduced by the agency?


2.  Website Visits - one indicator of transparency


2.  Standardized Survey of citizens' interaction with government (e.g., websites, meetings, etc.) - currently hard to do under the PRA (Paperwork Reduction Act)


3.  "CPIC" Foundation has done some kind of work considering this previous point (PRA and citizen surveys)


4.  Data Usage (example: Intelligence)


5.  Internal Surveys - because the external level of Open Government is based on the level of "openness" inside a federal agency.  In other words, if the government workers are unclear about "what's going on" in their own agency, then they can NOT make it any more clear to people on the outside.  Likewise, if the agency's employees do not have clear guidance and direction for collaborating with each other, then it will be no more clear to them how to collaborate with the public.  Surveys can show the level of that clarity concerning Open Government guidance and direction.




Postscript:  On the day before this Workshop (4/28/10), OMB had just released initial self-evaluations of agencies' OpenGov Plans.  As such, there was not time for Workshop participants to review those plans with respect to "measurement" (Criteria #29 in the Open Government Directive).   However, it is readily apparent from those self-evaluations that most of the agencies (including OMB and OSTP) have admitted that they do not yet have any measures for determining OpenGov progress.


Comments (5)

Stephen Buckley said

at 5:21 pm on Jun 10, 2010

A few months before, on OpenGovernment Radio (2/23/10; see link below), we talked about "How to Measure Success in Open Government" with Dave Lewan of ForeSeeResults.com and their recent paper on their survey connection of "transparency" and "customer satisifaction".


vr, Stephen Buckley, http://www.UStransparency.com

Chris Berendes said

at 12:08 am on Jun 11, 2010

There is a lot of wisdom in your sentence: "Sometimes, those measures are merely indicators (like our blood-tests) that must be interpreted as parts of a whole."

The BMI is a great example of a quantitative measure and of how such a measure helps assess what one usually really cares about - in this case, one's health, which is qualitative. For instance, to a first approximation, people would say that your BMI should be in a certain middling range for you to be considered healthy. But another part to that whole that is your overall health is your age. And Andrew Weil argues that, as you get older, a little more weight (and thus a higher BMI) may be correlated with longevity. (Of course, we can argue Weil's specific point, and whether longevity is all that, but let's accept it for the moment to think further about measures...)

In this case, we're working backwards from a large, roughly similar population, assessing their longevity (what we really care about), and then checking that against BMI measured earlier.

Relevance for measuring open government:

1. Understand that the measures we want to copy from medicine come from lots of studies over many years, and generally rely on a large number of individuals. (Weil cites a study of 9000 people. Compare that with the few dozen agencies we're looking at to understand Open Government, and you can see the first measurement challenge.)

2. Nail down what we really care about. Some readers will probably prefer to be slim and would be willing to give up a few months of life expectancy. It'd be pointless to argue about the study Weil cites until we've agreed how important slimness is (say for aesthetics or a sense of fitness) to someone's quality of life, vs. longevity at any cost.

Chris Berendes said

at 12:08 am on Jun 11, 2010

For open government, that means (to me), that it would be useful to have a qualitative description of what an ideal open agency would look like, or how we might determine, if we had all the time in the world, whether one agency was more open than another. And then, when we have that nailed down, we can work backwards to measures that are useful shortcuts, since we don't have all the time in the world.

We should also think on what openness gets us. For some, it's a reduction in waste, fraud, and abuse - if we can see how the Federal Government spends money, we can detect and then reduce waste, fraud, and abuse. So we may want the advice of auditors and forensic accountants.

For others, the goal is to support increased citizen engagement and participation - citizens acting as owners or as a board of directors. In that case, we want the advice of management accountants, not financial accountants, cf. http://en.wikipedia.org/wiki/Management_accounting (Back to the medical analogy - a good medical lab can run blood tests and tell you a lot about the state of your health, but it may take a trainer and a nutritionist to get healthy.)

Based on this, I'd circle back to a "Big picture" versions of the opening questions: What does the public want from a more open government? For instance: reduced waste, fraud, and abuse, or a greater understanding of how government operates, or the sense that decisions are made for the good of the country as a whole, not special interests? Each answer leads to different measures.

Daniel Honker said

at 9:11 am on Jun 11, 2010

Chris - Good point about needing to think about what openness gets us. One of our Fellows at NAPA talks about two distinct kinds of transparency: 1) "Fishbowl" transparency, which simply gives the public a glimpse into how things are currently being done, and 2) "Reasoned" transparency, which is really about government EXPLAINING why things are happening as they are. If our goal is simply to give people a look into the fishbowl, a lot of stuff will not make sense -- this has to be about a conversation, and that takes public responsiveness.

On the FOIA backlog as a measure of success, I remember discussing at the workshop the fact that you can't look at the backlog alone to determine if your opengov efforts are working. For example, some might assume that if your opengov efforts are succeeding, the FOIA backlog will decrease because people can find that information without FOIA request. However, you cannot use that as a standalone measure, because of dynamic effects -- your opengov efforts may INCREASE the public's desire for data, which may drive UP FOIA requests. An important point to consider, related to our BMI example.

Chris Berendes said

at 3:10 pm on Jun 14, 2010

Daniel - agreed on the vagaries of the FOIA measure and how it relates to BMI.

I like the contrast beween "Fishbowl" transparency and "reasoned" transparency. You could also distinguish between "audit" transparency - what a "Big Four" accountant (or an FBI investigator) needs to understand and certify a business's or agency's books, and "management" transparency - the necessarily more filtered and forward looking information management provides the board of directors.

I think that "management" transparency gets lost in the discussion of efficiency, and that could undermine public responsiveness. Imagine that you've just inherited a medium-sized business from an older relative. You specifically would not want a complete data dump - you'd drown. Assuming that you could trust the business's senior managers, you'd want filtered and structured information that lay out "owner level" choices you'd be required to make. (Of course, you might also want a good accountant to review the books, but let's stipulate for the moment that everything was on the up and up.) So you don't just want data, you want the interpretations of people skilled in that particular business.

You don't have permission to comment on this page.