Course: Dissemination | Monitoring and Evaluation in Art for Social Change

The following Evaluative Research Mini Course is an expansion of the content presented in our step-by-step guide.

This mini course explores some more advanced topics that would more typically be addressed in a research project.

The course is covered in 5 major sections. You may proceed in the order presented below, or jump ahead to the section of your choice.

You may proceed in the order presented below, or jump ahead to the section of your choice.

You may proceed in the order presented below, or jump ahead to the section of your choice.

1. Basics

2. Ethics

> 3. Theory & Methods <

4. Techniques

5. Analysis

6. Dissemination

You may proceed in the order presented below, or jump ahead to the section of your choice.

You may proceed in the order presented below, or jump ahead to the section of your choice.

You may proceed in the order presented below, or jump ahead to the section of your choice.

What is “Art for Social Change”?

Regardless of the art form or tool used, art for social change engages the members of an identified community using creative imagination and expression to work out identity, shared values and aspirations. It is hoped that participation in this kind of creative engagement would lower the threshold for many forms of social participation and social action, and would help people to find new ways to see and be engaged in the world.

What makes this “Social Change”?

While there is great potential for individual learning and development in this work, art for social change is community-focused, aimed at groups rather than just individuals. Issues affecting individuals are always considered in relation to group awareness and group interests. Building skills for social change requires meaningful participation in shaping collaboratively created work.

(Reference: Goldbard, A. (2006). New creative community. The art of cultural development. Oakland, California: New Village Press)

Photo by Stephanie Sicore, distributed under a CC BY 2.0 license.

Types of Art For Social Change Projects

There are 3 major types of art for social change projects:

Artist creation: the artist(s) create their own artwork, and the social change content is in the work itself (e.g., Pablo Picasso’s Guernica)
Group problem solving: the artist(s) act as facilitators using art as part of a problem solving process, where group art creation is not the final goal of the project (e.g., image theatre to improve corporate teamwork)
Group artwork creation: the artist(s) act as facilitators for group art creation for social good (e.g., see the Examples page)

This guide assumes that your project being evaluated is going to fall into the category of “group artwork creation.” It's important to clarify that art for social change projects are not focused solely on the production of art, but on the process of making art together, for example to work out identity, shared values, and aspirations.

Photos by Isengart, Medialab Prado, Jumblies Theatre distributed under a CC BY 2.0 license.

Art Forms

In addition to considering the project outcomes, it is helpful to consider the art form(s) being used in the project as this may also influence the monitoring and evaluation method(s) that you select.

Below are some of the art forms that may be involved in an art for social change projects. This is not an exhaustive list; an art for social change project could utilize combinations of these art forms, or others not listed here.

Performing arts
- music
- dance
- theatre
- circus
- interactive performance
- spoken word
- oral storytelling
Multimedia
- digital storytelling
- art installation

Visual arts
- drawing
- painting
- sculpting
- murals
- graffiti
- urban arts
Media arts
- photography
- video
Literature
- poetry
- creative writing

Photos by Phil Roeder, Frank Kovalchek, Mike Goren, Fredrik Rubensson, distributed under a CC BY 2.0 license.

Evaluation of Art For Social Change Projects

There has been growing interest in better understanding and documenting whether and how community-based arts initiatives can bring about social change. However, for some artists and community groups involved in art for social change projects, the topic of evaluation can be fraught with frustration.

This resource seeks to better inform those involved in art for social change projects regarding the diverse theories, methods and techniques that may be useful for evaluating such projects.

You will need to address questions like:

What is the theory behind the program?
(What does the program hope to accomplish, and how?)
Who is the evaluation for?
(The internal staff? The community? The funder? All of the above?)
What is the purpose of the evaluation?
(Improve the program? Decide whether funding will be renewed? Other?)

Whose perspective?

There are different groups that may be involved in the project:

Artists
Program participants
Facilitators
Researchers
Funders
Community groups
Social activists

These groups may all have different objectives. As such, the perspective from which an evaluation is being conducted is important to consider and articulate.

Quantitative Evaluation

Quantitative data are data that can be measured and assigned a numerical value. For example: average age of participants of each gender, number of participants enrolled, and so on.

In art for social change projects, you can collect primary data, which is data you collect for the purpose of your evaluation. For example, quantitative evaluation can be based on comparing the number of people who answer one way versus another in a survey.

Or, the quantitative data can be secondary data, which is data taken from already existing sources. For example, secondary data could be crime rates, school drop out rates, or substance abuse rates. These numbers can be used in analyzing outcomes.

Qualitative Evaluation

Qualitative data are data that can be observed or experienced, but not directly measured. For example: colours, textures, smells, tastes, phrases, expressions, statements and stories.

In art for social change projects, qualitative evaluation is based on what people say or otherwise express. The data can be used to gain insight and understanding of certain phenomena.

Photo by Pink Sherbet Photography, distributed under a CC BY 2.0 license.

Quantitative and Qualitative

While studies can be quantitative or qualitative, keep in mind that projects may collect both types of data.

Also, a quantitative study could have an open-ended question on a survey that would collect some interesting qualitative data.

Likewise, information collected from a qualitative study may also include some information summarized into a quantitative format; for example, the proportion of the people interviewed that mentioned a certain topic.

Mixed Methods

A mixed methods approach involves integrating methodologies, traditionally quantitative and qualitative, but arts-based methods can also be integrated with quantitative and qualitative.

The benefit of mixing methods is that it can help you overcome the limitations and weaknesses that arise when each method is used alone and allow you to ask a broader set of questions. If different data sources reveal the same findings, or findings that are coherent with each other, this can lend credibility to your evaluation. A mixed method evaluation can also deepen understanding of your program, its effects, and context.

Examples

Visit the Examples page to see examples of projects using mixed methods approaches.

Some Issues to Consider in Mixed Methods Evaluations

What stage(s) of the evaluation to mix methods? (The design is considered much stronger if mixed methods are integrated into several or all stages of the evaluation.)
Will methods be used:
- sequentially (the data from one source inform the collection of data from another source), or
- concurrently (triangulation is used to compare information from different independent sources)?
Will qualitative and quantitative methods will be given relatively equal weighting?

Arts-Based Evaluation

Arts-based evaluation (ABE) utilizes art or artistic processes in the evaluation process itself.

It is important to distinguish between ABE methods and the evaluation of arts-based projects. ABE methods can be used for ASC projects, but are not necessarily always the best methods to use. ABE methods conversely, can be very effective in gathering data on projects that have little to do with art or social change. While both ASC projects and ABE methods utilize art, ABE methods are evaluation tools. As such, the arts-based evaluation methods should be planned for and prepared in advance of implementing the project, as with other evaluation methods.

ABE methods do not necessarily have a social change component. For example, adolescents attending a course on financial accounting may be asked to draw a picture or write a poem to indicate how they felt about the course; this would be an example of ABE, but the intervention had nothing to do with social change.

Generally, when ABE is used for ASC projects, the activity can both itself be “the ASC” activity, serving as an intervention to achieve a desired outcome. At the same time, this exercise can have great evaluative or research value that also sheds light on the effectiveness of the ASC intervention at achieving its desired goals, including the challenges faced.

One commonly used art-based technique that can be seen both as an “arts-based intervention to promote social change”, as well as an “arts-based evaluation” technique is photovoice. In photovoice, participants are asked to photograph images that represent the research subject and/or how they feel about an intervention underway. The photographs can be interpreted to derive themes and narratives for further research. Some may reflect directly on what a community-based project means to them. For example, persons living with a disability could be asked to photograph what is meaningful to them - some participants may photograph the barriers they face on a daily basis; others show various accommodations implemented. The photos are analyzed to identify common barriers as well as to reflect on how participants feel about measures offered.

Indeed when art forms are used in evaluation, by allowing participants to express themselves creatively, this exercise itself can be seen as “arts for social change”.

Examples

Visit the Examples page to see examples of projects using arts-based evaluation methods.

See an example of a photovoice exercise as well as an example of an embodied inquiry/performative inquiry workshop - as used in our arts-for-social change research.

Arts-Based Evaluation

Arts-based evaluation (ABE) utilizes art or artistic processes in the evaluation process itself.

It is important to distinguish between ABE methods and the evaluation of arts-based projects. ABE methods can be used for ASC projects, but are not necessarily always the best methods to use. ABE methods conversely, can be very effective in gathering data on projects that have little to do with art or social change. While both ASC projects and ABE methods utilize art, ABE methods are evaluation tools. As such, the arts-based evaluation methods should be planned for and prepared in advance of implementing the project, as with other evaluation methods.

ABE methods do not necessarily have a social change component. For example, adolescents attending a course on financial accounting may be asked to draw a picture or write a poem to indicate how they felt about the course; this would be an example of ABE, but the intervention had nothing to do with social change.

One commonly-used ABE method is photovoice. In photovoice, participants are asked to photograph images that represent the research subject, and then the photographs can be interpreted to derive themes and narratives. For examples, persons living with a disability are asked to photograph the barriers that they face on a daily basis. Then the photos are analyzed to identify common barriers.

Arts-based approaches may be used in combination and/or in conjunction with more traditional approaches.

Social Context

Social context is fundamental to understanding the adequacy of program conceptualization and design:

do interventions adequately address the social context within which target outcome are created, sustained and socially distributed in time and space?

Social context is also fundamental to program implementation:

are interventions optimized to take advantage of the unique confluence of opportunities available in each local context?

Finally, social context shapes the production and utilization of evaluation findings:

the influence of key assumptions and stakeholders on the design and implementation of the evaluation.

Advantages of Art-Based Evaluation/Research

Advantages of arts-based methods include:

Participants have the opportunity to express themselves through different art forms which may reveal insights that they may not have otherwise been articulated
May be more conducive to exploring and communicating subjective experiences that can be difficult to capture with traditional methods
The arts can accommodate people who learn in different ways, people who have different cultural backgrounds and/or who are less articulate
The arts can accommodate skills and abilities of vulnerable populations
Encourage participation of those who may otherwise be reluctant
Increase participant engagement and empowerment in producing arts-based work
Address, challenge and rebalance traditional power dynamics
Utilizes alternative ways to produce and communicate research findings, which are often more accessible to diverse audiences
The arts may be better suited to capture complexity and multidimensionality that is otherwise difficult to capture comprehensively with traditional methods

Challenges of Art-Based Evaluation/Research

Challenges of arts-based methods include:

Some participants may be reluctant to engage with arts-based techniques
Arts-based data may be more time consuming and/or resource intensive to collect (e.g., planning a performance, preparing an exhibit)
Resources and costs required to produce art (e.g., cameras, art supplies, theatre space)
Arts-based data may be more difficult to analyze and interpret
May provoke painful or unexpected feelings for the participants and/or evaluators/researchers
May be limited to small sample size and subsequent impact on generalizability
Ethical issues regarding privacy and intellectual property surrounding group artwork creation
Challenge to define, measure and collect longitudinal outcomes
Difficulty in establishing validity in the case of multiple interpretations

Examples

Visit the Examples page to see examples of projects using arts-based evaluation methods.

Reference

Boydell, K.M., Gladstone, B.M., Volpe, T., Allemang, B. and Stasiulis, E. (2012). The production and dissemination of knowledge: A scoping review of arts-based health research. Forum Qualitative Sozialforchung/Forum: Qualitative Social Research. 13(1), Art. 32.

Funder-driven and/or Self-driven

It is always useful for an organization to review what they are hoping to accomplish and how they are doing. Unfortunately, evaluation is sometimes associated with “funder’s imposition”, with an expectation of well-defined quantitative data showing the “value” of the program, rather than a more thoughtful critically reflective process.

We'd like to see the numbers. What percent improvement do you have for the indicators you are evaluating?

Who is initiating the evaluation?

Depending on who requests or initiates the evaluation – whether it is the funder, you as the project organizer, the participants and/or the community – there can be different intentions for undergoing an evaluation. For instance, funder-driven evaluations are often associated with an expectation of well-defined quantitative data showing the “value” of the program to demonstrate a return on investment. On the other hand, a self-driven evaluation may focus more on qualitative data, such as examining the experience of the participants.

If the evaluation is funder-driven, it can be helpful to discuss with the funder the evaluation process and how the results will be used. Keep in mind that research does not always have to be quantitative; it may be preferable to use qualitative or arts-based methods, or even a combination of quantitative and qualitative methods.

We'd like to see the numbers. What percent improvement do you have for the indicators you are evaluating?

While we can provide some numbers, we'd also like to examine the experience of the participants.

Collective Impact & Shared Measurement

The social challenges and problems we face today are often at the intersection of many systems, contextual factors and the relations between them- in other words, the social processes involved.

It is hoped that when those who have been marginalized from social, cultural and economic access artistically create together, social inclusion and their sense of worth and sense of community will improve, thereby increasing social equity and improving collective living conditions (Spiegel & Parent, 2017). However, actual collective impacts are linked to how individuals engage with larger social and collective activities and conditions (Spiegel & Parent, 2017).

Social determination of health:
The analysis we adopt in the tool, as explained by Spiegel et al, 2018, is the notion of “social determination of health” whereby health is conceptualized as:

“a complex multidimensional dialectic deeply rooted in social and political processes in which social groups have “ways of living” defined by their position in class/gender/ethno-cultural relations, in turn expressed in individual lifestyles and bio-psychological embodiments (Breilh, 2010, 2013; Krieger, 2011). The social determination of health approach, in contrast to the more traditional social determinants of health analytic framing, focuses attention not merely on the discrete factors or conditions that impact health and wellbeing (e.g. nutrition, housing, education, income, etc.), but rather on the structural processes at the societal level that lead to these social inequities, and the interrelationships among these (Breilh, 2008).”

The overall objective in trying to better understand “collective impact” of an arts for social change project thus should focus on:

“understanding the complex ways by which social policies, as well as their associated social interventions employing the arts (sociocultural interventions), intercede in the dominant modes of constructing ways of being and lifestyles at the individual and collective level.” (Spiegel et al., 2018)

References & Resources:

Breilh, J. (2008). Latin american critical ('social') epidemiology: New settings for an old dream. International Journal of Epidemiology, 37(4), 745-750.

Breilh, J. (2010). Las tres "s" de la determinación de la vida y el triángulo de la política. 10 tesis hacia una visión crítica de la determinación social de la vida y la salud. En: Passos R, organizador. Determinacao social da saúde e reforma sanitaria. Rio de Janeiro: CEBES, 88, 94.

Breilh, J. (2013). La determinación social de la salud como herramienta de transformación hacia una nueva salud pública (salud colectiva)/The social health determination as a tool of transformation towards a new public health (community health). Revista De La Facultad Nacional De Salud Pública, 31, 13-27.

Krieger, N. (2011). Epidemiology and the people's health: Theory and context. New York: Oxford University Press.

Spiegel, J., and Parent, S. (2017). Re-Approaching Community Development through the Arts: A ‘Critical Mixed Methods’ Study of Social Circus in Quebec. Community Development Journal.

Spiegel JB, Ortiz B, Campaña A, Boydell KM, Breilh J, Yassi A. (2018, under review). Social transformation, collective health and community-based arts: Buen Vivir and Ecuador’s social circus program. Global Public Health.

Measuring collective impact:

In their article on collective impact, Kania and Kramer (2011) focus on the collective benefits within the partnership of those involved (rather than society as a whole, which would be too big a task for an “evaluation” exercise.). Kania and Kramer focus on shared measurement, which goes hand-in-hand with authentic partnerships.

As with other types of evaluation, it is important to consider the theory of social change underlying the program as this will inform the focus of the evaluation. In shared measurement, this is especially important to establish among the collective impact partners, as the partners may otherwise approach social change at different levels – for example, at the micro (individual) level versus the meso (group) level – and this would influence the respective choice of measures or indicators to collect and analyze within the shared measurement system.

One example of a method relevant for collective impact and its shared measurement is outcome mapping. In outcome mapping, the focus shifts from measuring direct causal outcomes to focus on behavioural changes and the subsequent ripple effect as those changes move into the wider community and into other domains. This is particularly relevant for collective impact, where the overall goals of social change may be too complex and longstanding to attribute as direct causal outcomes to individual programs. Learn more about outcome mapping in the e-book, Outcome Mapping: Building Learning and Reflection into Development Programs, published by the International Development Research Centre.

References & Resources:

Cabaj, M. (2017). Shared Measurement: The Why is Clear, the How Continues to Develop. Tamarack Institute.

Cabaj, M. (2014). Evaluating Collective Impact: Five Simple Rules. The Philanthropist, 26 (1).

Cabaj, M. Collective Impact & Shared Measurement: Tough But Necessary. Tamarack Institute.

Earl, S., Carden, F., & Smutylo, T. (2001). Outcome Mapping: Building Learning and Reflection into Development Programs. International Development Research Centre.

FSG & Aspen Institute Forum for Community Solutions. (2014). Collective Impact Forum.

Kania, J., & Kramer, M. (2011). Collective impact. Stanford Social Innovation Review, 9(1), 36-41.

Kramer, M., Parkhurst, M., & Vaidyanathan, L. (2009). Breakthroughs in Shared Measurement and Social Impact. FSG Social Impact Advisors.

Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.). Thousand Oaks: Sage Publications.

Guiding Principles

Evaluating community-based social change initiatives often entails working within inequities, such as those caused by systemic discrimination and unequal distribution of power. In working within inequities, a set of principles may be helpful to guide ethical and equitable work in this area. The Canadian Coalition for Global Health Research (CCGHR) developed “Principles for Global Health Research,” which can be applied to other areas of research and evaluation beyond global health research. The principles are:

Authentic Partnering
Inclusion
Shared Benefits
Commitment to the Future
Responsiveness to Causes of Inequities
Humility

Read more about the principles on the CCGHR website.

Funder-driven and/or Self-driven

However, traditional evaluation methods are not always appropriate for art for social change projects. These reasons may relate to:

inappropriateness or difficulties in defining quantifiable outcome; and/or
small sample size; and/or
complexity of defining the target population; and/or
the nature of the program itself.

While we can provide some numbers, we'd also like to examine the experience of the participants.

From Evaluation to Research

There is a continuum from “operational process evaluations to show funders how funds have been spent” to “inquiry for more in-depth understanding of the social impact of an art for social change (ASC) project that contributes new knowledge to the field of ASC”. Some authorities consider only the former to be “evaluation” and the latter to be “research” best conducted by academic scholars. It may be wise to see this as a continuum rather than a dichotomy (either one or the other).

If the intention is to publish in a peer-reviewed journal, this would be to the right of the continuum and would require research ethics approval (a requirement for publishing in a peer-reviewed academic journal). Alternatively, if the evaluation is being conducted for the organization's own internal use and not intended for a broader audience, this would be to the left of the continuum.

Regardless of the position along the continuum, it is important to conduct the research/evaluation ethically and systematically so it can be used and interpreted by other people.

Terminology

The words “evaluator” and “researcher” are used interchangeably in this tool, for simplicity.

Similarly, the words “data” and “information” are used interchangeably; both mean the results of quantitative, qualitative and/or arts-based evaluation or research.

Hiring an evaluator

We designed this tool so any organization can conduct evaluation in-house. However, if your funding allows it, you may want to hire an evaluator. It is important to consider this early in the planning stages, as an evaluator will likely have valuable input.

Keep in mind that the evaluator your group selects will have an impact on the type of evaluation they conduct. It's important to select someone who shares your vision.

You'll want to consider previous experience in:

The art
The community
Conducting evaluations or research

It may also be the case that you want to conduct the evaluation without an external evaluator. This could be beneficial as nobody will know your program better than your own team.

What does the project hope to accomplish?

The answer to this question will determine what are suitable method(s) to use for your evaluation. For instance, a particular outcome may be best evaluated with quantitative methods whereas a different outcome may be better suited to a qualitative evaluation, or even an arts-based approach.

Below are a few of very many examples of desired outcomes from art for social change projects.

Improve school readiness
Enhance the quality of life of people living with a disease
Build relationships across intercultural, intergenerational, and other differences in life experiences
Include people with developmental disabilities in a university environment
Empower social change within educational institutions
Reduce reoffending rates in incarcerated youths

Examples

The list above provides only a few examples. Visit the Examples page to see more examples of ASC project outcomes.

Logic Model

Note that “evaluation” should never be an afterthought. Projects and programs should try to plan their activities with as clear as possible idea of the outputs, outcomes and impact they wish to have — and how each of these will be assessed. This is referred to as a “Logic Model”.

Examples of Evaluations or Evaluative Research*

We have compiled a searchable list of examples. In addition to filtering by outcome, you can filter the list by:

Type of evaluation (arts-based, quantitative, qualitative)
Art forms used in the evaluation
Population that the program targets
Issue that the program studied
Country that the program took place in

* As these examples were mainly drawn from peer-reviewed academic journals, most of these examples fall more on the side of evaluative research than program evaluations. In other words, the authors were likely funded as researchers not only to evaluate the specific program, but to contribute new knowledge to the body of scholarship on the impact of ASC project.

Please do not feel that these kind of in-depth undertakings are necessarily needed for routine ASC evaluations.

Click here to go to examples.

Who should you engage with during the evaluation?

It's important to involve your partners throughout the evaluation process.

Across the field, ASC practitioners emphasize the value of long-term partnerships and projects both as crucial to deepening impact and as an ethical imperative in community-engaged work. As an integral part of this community-engaged work, partners should be actively involved in the evaluation process, including involvement in:

Defining the project outcome(s) of interest
Planning the evaluation
Selecting the method(s)
Interpreting the observations
Sharing the results

Throughout the project and evaluation process, key success factors in developing partnerships include:

Relationship development
Trust
In-depth planning time

The way in which partners are involved in the evaluation process may depend on the governance and/or operational models of the project. There are various ways to organize and structure partnerships, both at the governance and day-to-day operational levels.

Governance models of partnership include:

Joint: partners collaborate with equal ownership and accountability; often involves shared funding and/or resources.
Stratified: one partner assumes primary ownership while supporting the other partner(s).

Operational models of partnerships include:

Co-location: use of shared studio and office spaces to avoid rising property costs, as well as opportunities for networking and the creation of new relationships and partnerships
Integration of activities: institutions such as libraries and parks boards are integrating ASC activities into their own programs and facilities.

Evaluating Partnerships

Navigating partnerships effectively can be complex and challenging, so ongoing monitoring and evaluation of the partnership itself can be beneficial to help ensure alignment and mutual benefit among partners.

The Canadian Coalition for Global Health Research (CCGHR) offers a Partnership Assessment Tool to guide and support the joint evaluation of health research partnerships, although it can be applied to other sectors including art for social change evaluations. The tool consists of a series of questions and exercises, leading through four stages:

Inception
Implementation
Dissemination
“Good endings and new beginnings”

Download the Partnership Assessment Tool from the CCGHR website.