Twenty-five posts about evaluating the impact of arts projects, from blog.artsusa.com:
The idea of assessing impact through the eyes of strangers intrigues me.
Young people were delighted with public art that spoke to them. So, here’s another possibility for viewing impact. How does the community’s public art speak to young people? Does it say those who live here…
- …have a sense of humor?
- …welcome different points of view?
- …understand young people?
- …like to have a good time?
Introduces the mobile ideation kit, which facilitates:
- interesting conversations with passers-by in a variety of communities, with an easy, attractive way to ask about what would be important to them
- potlucks around food policy
As is typical with these kinds of gestures, there are interconnected, but distinct intentions for this Public Kitchen: one that is an aesthetic intention, and one that is social/political.
The evaluation: Without mucking up the magic, find out (a) Did they stop to eat? (b) To talk? (c) What did they think/feel/do?
The best case studies of measuring the impact of social change incorporated the arts as part of the evaluation process:
- The Boston Youth Arts Evaluation Project (BYAEP)
- Two Way Mirror
- The Center for Digital Storytelling (CDS)
- The End of the Line
Visual evaluations matter. Which one (left or right) grabs your attention and makes you pay attention?
(Marc’s comment: Obviously the left works. So when will we finally get evaluation tools (excel, SPSS, SAS, SenseMaker, NVIVO) that spit out visual results instead of text and spreadsheets? Art imitates life, but evaluations are slaves to time and software.)
Instead of asking, “are we making an impact?” we should have been asking, “are things changing for the better?” We can measure the second, and these measurements can come from traces we leave behind as we live life.
Four art visionaries you should know, because they also use data to improve their “product”:
Jim Henson, VI Hart, David McCandless, and Jonathon Harris.
The Awesome Foundation – people give $100 a month and grants go back to groups who have just the right size need – meaning that $1000 will be just enough to flip the program from failure to success.
I don’t think the Awesome Foundation folks believe that a $1000 contribution in and of itself will herald world peace but it can add enough gas to a creative engine to make something happen. And if we can hold our assessment of the arts to the same standard: that we don’t expect instant revolution but that we do expect something to happen I think we’ll be on our way to constructive conversations informed by new methods of assessment, about what the arts can really do and what support they need, in each instance, to make an impact.
[Learning civic engagement by doing it in the theater.]
We wanted to create a show in which the audience members were active, vocal participants who ultimately determined the outcome every night. What makes us care? “A crisis” was our initial answer.
In Beertown, the audience was asked to bring a dessert to share in a pot luck; everyone received name tags and commemorative t-shirts; we said the pledge of allegiance together and sang a town hymn; and, ultimately, we were given a task to complete collectively: determine what items should go into the town time capsule to represent us as a community on this exact day.
Through the simple tasks and rituals we built into the framework of the show, Beertown allowed audiences to participate first-hand in an artistic-based act of civic engagement. What was exciting for me about every conversation during Beertown were how real they were, despite being completely fabricated. Becoming Beertonians allowed rooms full of strangers night after night to expose their values and debate passionately about what mattered to them.
Evaluating social justice, inclusion, and well-being:
Ultimately, our index included measures of neighborliness (whether residents feel that they trust their neighbors or feel like they “belong”), civic participation, the presence of social organizations, and some demographic characteristics (percent of single person households, residential mobility).
These two dimensions of social connection do appear to line up with the classic sociological distinction between community (or as the Germans say, gemeinschaft) and society (or gesellschaft), or the distinction between a social order based on face-to-face relationship and one based on a set of institutional intermediaries.
What’s the middle ground between rigor and authenticity in the arts?
Short-term and narrow quantitative performance indicators have contributed to the collapse of our global economy, we have falsified test scores, and potentially unhealthy quotas for arrests and police stop and frisks.
Years ago, at the end of a talk with the evaluation director of a major national foundation, I asked him about “evaluating our impact.” He asked, “How many kids do you have in the program?”
“Do you know them well?”
“I think so.”
He told me to just call each of my students at the end of the cycle, to have an open-ended conversation about what they got out of the program and what could make it better.
Don’t overthink it. For honest evaluation, maybe we just ask our beneficiaries.
We want to own “rigor” and “accountability” on our terms and create ways to honor their voices…
Marc’s post and many of the other examples suggest that when it comes to evaluation and the arts, surveys and statistics are out; stories and experiences are in.
Penny Balkin Bachdescribes using storytelling to deepen each artwork’s engagement with a general public.
Rachel Engh describes a feature allowing users to record their own stories about experiencing art in public spaces.
Also, I have still only been able to find a relatively small number of such recorded anecdotes per public art organization—nowhere near the “40,000 stories and counting” described by Maxson in relation to Global Giving’s work.
How do we collect such stories from the people who are not already Facebook and Twitter followers of a public art agency?
And when we do get a lot of good feedback, how do we manage all this information to tell a larger story about public art?
Evaluation should engage and enrich art, not serve as an empty distillation of findings for funders who need to assess impact.
To invest in art in our systems, in schools, public spaces, or in community spaces requires a significant paradigm shift towards acknowledging the importance of our essential joy, our bond and connections with others and our communities. And evaluations should help us build art that paints this picture.
What substantive value do the arts bring?
Standard evaluations miss out on the uncertainties, nuances, and grand questions that are part of complexity.
There is something bigger out there: We can identify the situations where arts contribute to sweeping social change.
If any other sector could stake its claim to the power of the arts, they wouldn’t be trying to play with variables to make their impact stack up in regression equations.
A Brief Conversation on Evaluation, Privilege, & Making Trouble – CAROLINA ‘CJ’ JIMENEZ (MARC’S PICK)
It’s hard to find out who you are when no one knows your name. When I started high school, I was no longer Carolina Jimenez or CJ.I became my student number (8259745).
Locker number (367)
My GPA (2.3)
My test scores (97 percentile in English;
35 percentile in Math;
85 percentile in Writing/Reading;
I still have no clue what that means…)
I became obsessed with how I looked on paper, not what I was learning,
becoming remodeled into a receptacle for lectures and test scores.
Learning should result from curiosity, not obligation.
Excellent Conversation follows…
Jason Yoon (JY): I think these pieces [above] powerfully talk about “measuring impact.”
One of the most common barriers to evaluating public art is defining exactly what we are evaluating. [She provides a list of examples of indicators others have used to evaluate public art.]
How can small or midsized arts organizations measure their impact without the resources of large institutions like the NEA?
We decreased the number of in-school residencies we had in order to invest those resources in evaluation efforts that could help us measure our long-term impact.
After years of being asked by funders how my program evaluates its outcomes and answering with anecdotal stories and satisfaction survey results, I decided to try to find something more meaningful. These clusters of outcomes emerged:
- An emotional shift: Participants experience engagement, joy and a feeling of being uplifted. (76 percent of practitioners described this outcome)
- Increased sense of self: Participants gain the ability to be self-expressive which in turn affirms participant strengths and identity, builds confidence, self-esteem, and self-awareness. (84 percent of practitioners described this outcome)
- Increased sense of community: Participants experience increased connectedness even across differences. They understand new perspectives and experience an ability to contribute to culture and community. (100 percent of practitioners described this outcome)
- Empowerment: Participants are able to make good choices and determine their own future in a way that allows them to be self sufficient & self-actualized. (92 percent of practitioners described this outcome)
- Creative problem solving: Participants learn to think for themselves, connect divergent ideas, think imaginatively, refine ideas, and focus on multiple constructive ways to solve a problem. (53 percent of practitioners described this outcome)
- Social change: Participants become agents of change who address systemic issues. They become helpers, healers, and advocates who contribute to social cohesion and a shift in thinking about culture and stereotypes. (53 percent of practitioners described this outcome)
- Skills gained: Participants gain job, art, and life skills. (46 percent of practitioners described this outcome)
When I’m called upon to help design an evaluation for an arts initiative, I look for inspiration:
- In the new relationships that are formed and the existing ones that are strengthened, especially from those boundary-crossing opportunities that the arts nurture so well;
- In the overt recognition of stories untold and painful episodes previously unspoken;
- In public policy openings, even when not successful;
- In new roles assumed by key players, whether formal or informal;
- In subtle shifts in power; and
- In the involvement of the next circle of artists who can spark subsequent engagements over future issues.
“Is my new home another Congo, where I will live with many unanswered questions about my future?” asks Claude Rwagange, the founder of Community Financial Literature.
“I’ve seen this city grow older, while its people grow younger still. It struggles and soars, keeps open its doors,” wrote Lee Urban, Portland’s former director of planning and development in his poem entitled Old Munjoy.
These are a few examples that highlight the importance of mutual trust between community and city leaders, and a means of collecting them (shared poems).
Perhaps the recorded messages will spur dialogue between viewers, conversation about public space, art, and relevant social issues reflected in the artwork. Maybe viewers will lobby for additional public art work in their communities.
Soul of the Community – community attachment creates an emotional connection to place (which also correlates to local economic growth) and the key drivers were social offerings, openness, and the aesthetics of place–all potential attributes of public art.
Public art can create community attachment, if we overcome perceived barriers and open pathways for engagement. With this in mind, the Fairmount Park Art Association developed Museum Without Walls™: AUDIO —a multi-platform interactive audio experience, available for free on the street by cell phone, audio download, Android and iPhone mobile app, QR code, or online as streaming audio and audio slideshows.
Some valuable lessons:
- Be realistic (you can only accomplish so much in 45 minutes with 30 third graders);
- Plans can be adjusted (and improved) when you know the endgame;
- Assessment is linked to impact and change;
- If you can observe it, you can measure it.
- Impact work allows artists for be more intentional.
I propose that art makers ease into measuring impact by trying some of the approaches of educators:
- What is the essential question of the work–the open-ended question that frames audience inquiry? What ideas or understandings will audience “uncover” through the performance?
- What behaviors do you want to see the audience engage in during this performance? When you see them do those things, what message does that send you?
- What activity do you want audience involved in immediately following a performance of this show?
- What will the audience learn or be inclined to do?
1) Watch the audience during performances.
2) Ask the audience (comment card or emailed Google form)
- What did the show make them think about?
- Did they [insert list behaviors] during the show?
- What questions about the work or for the artists linger after the show?
3) Look over all the answers.
Rethinking Social Impact: “We Can’t Talk About Social Well-Being Without the Arts & Culture” – MARK STERN
We must move beyond purely economic yardsticks in judging well-being. Nussbaum suggests ten core capabilities:
- bodily health;
- bodily integrity;
- and thought;
- practical reason;
- other species;
- and control over one’s environment.
These are the basis for the Human Development Index, European Union, and Organization for Economic Cooperation and Development (OECD) studies of social inclusion and social justice.
We need to place the arts in a frame that includes all of the dimensions of social well-being.
- Every meaningful social change movement for the last 1,000 years, at least, has been driven, in large or small part, by the arts.
- Many arts-based civic works contribute little or nothing to individuals, communities, or societies.
In an evaluation culture focused on efficiencies, what works models, and causal attribution for low-risk outputs, how do you forecast impact when arts are deployed in service of social change? [When the connections are not linear?]
Are the strategies that you are employing truly effective in the long-term?
The beauty of seeking the answer is this: if you learn what works and what doesn’t, you have the opportunity to adapt and change your approaches to do what is most effective. And that makes every task you perform more meaningful and rewarding.
Some things we might believe (or hope) we do in the arts are difficult to prove.
What kind of things can be assessed?
Artist – Personal communication skills, being on time, and be able to work/lead as part of a team.
Artistic – Aesthetic judgment that consider use of color, drawing skills, negative space, conceptual appropriateness, safety, etc.
Fiscal – Was the budget appropriate and adequate? Were payments timely?
Organizational – Did the sponsoring agency provide respectful and timely communication and support? Were the goals for participation by community members met? Was the PR rollout a success?
Partner – Did the project meet the appropriate needs of partner(s)? The significant word here is “appropriate.”
Political – Did Fox or CNN do an exposé. (Yes, that is a joke.)
So, if those items aren’t enough to assess public art—what else is needed?
(part 1 of 2)