A schema of measurable outcomes for cultural engagement

Measurable outcomes are an integral part of successful planning and evaluation. To know if we have been successful in achieving our objectives, we must be able to understand and assess the change we seek. However, the articulation of the value of cultural development activities, including the arts, is considered particularly challenging.

Historically, efforts to measure the contribution of cultural activities (including the arts) has focussed on inputs (such as expenditure and investment in facilities, staffing and programs); or outputs, (the amount of activity undertaken, as measured by number of arts products created, event participants or tickets sold). While recording the amount of activity that occurs is important, it does not tell us anything about whether that is beneficial, and does not consider the possibility of neutral or negative outcomes, or relative benefit to costs of options for cultural participation. As yet, there are no nationally agreed standards in local government in Australia for collecting cultural data outputs.

Many emerging evaluation methods focus on measuring audiences’ perceptions of artwork made or experienced. While these tools assess the quality of the product, they do not measure outcomes – what difference the work ultimately makes – to those who receive it. A causal relationship might be expected, with the quality of arts products related to their impact, but this cannot be assumed.

When outcomes of engagement in cultural activity have been considered, they have often been categorised as either intrinsic or instrumental, where intrinsic is seen to be directly related to the cultural experience itself, and instrumental occurring outside, or in addition to, the cultural experience. Evaluation processes have often use social and economic outcomes (those often identified as instrumental) as proxies of cultural value, partly because outcome measures are better established in those domains, and also because economic outcomes are often prioritised above others. Assessment of intrinsic (i.e. cultural) value has been perceived as problematic, because culture is considered to be intangible and therefore difficult to measure. Thus, cultural outcomes of cultural activities have largely not been measured. More on this topic.

CDN has been working with the National Local Government Cultural Forum since 2014 to establish an agreed set of measures for inputs and outputs of local government’s cultural activity. The data schema of inputs and outputs is now available here.  While this schema has been developed specifically for local government, it may also be applicable to other levels of government and other cultural organisations.

The impetus for establishing this outcome schema came from CDN’s work with the National Local Government Cultural Forum to develop two on-line tools, a Framework for Cultural Development Planning and a program logic tool and recording system for cultural development projects. The Framework for Planning offers a step by step guide for cultural development staff to create strategic plans, while the program logic tool assists with outcome-focussed planning of cultural activities. However, for these tools to be useful, a means of measuring outcomes of cultural activity was required.

The schema was developed through reference to theoretical material and input from practice experts. The process began with a wide and deep survey of the literature, encompassing theoretical pieces on the nature and function of cultural practices, policy documents from international bodies advising cultural policy-makers and outcome studies. Ideas about, and evidence of, outcomes of cultural activity were extracted from this literature. Findings from this material were sorted first into policy domain areas of cultural, social, economic, governance and environmental. For example, findings from studies indicating that cultural activity can contribute to employment and business success was grouped in the economic domain. The domain definitions offered by Community Indicators Victoria were the original impetus for this step, as they offer clear distinctions between the domains, as well as clear outcomes (desired futures) in each domain that are categorically different. Thus, the cultural domain is distinct from the social domain, because the desired outcome in the cultural domain is cultural richness and vibrancy, whereas outcomes in the social domain are health, safety and inclusion. CDN continues to re-think the definitions and understandings of these domains, now that CIV is no longer operating.

Then material relevant to each policy domain was further distilled into a set of outcomes that are considered to be categorically different from each other, while still being related and contributing to the desired future in each domain. Each domain has several outcomes that the literature indicates can be directly achieved or impacted by cultural activity. The outcomes in the cultural domain are currently the only ones fully developed, with outcomes in the other domains the focus of CDN’s work from 2017.

These cultural outcomes and the associated measures and evaluation resources including suggested questions, are continually being improved through discussions with experts and peers. This has included presentations to more than 1000 researchers, bureaucrats and practitioners from fields such as cultural development, local government, arts management, evaluation and regional development, in Australia, Asia and Europe between 2012 and 2017. Further refinement has been enabled by focus group discussions and informal Delphi-style consultations (Rowe & Wright, 2001) with arts managers and decision-makers at federal, state and local government levels across Australasia. Consistently positive responses were received, indicating the potential of the framework and related measures to address evaluation needs of these professionals. The cultural outcomes schema was endorsed by the Australian National Local Government Cultural Forum in April 2016, and is now being trialled by diverse institutions, governments and organisations within the sector to validate their use and efficacy.

CDN consulted with a wide range of stakeholders across different sectors of the cultural arts. This includes discussion with representatives from every state in Australia, an array of countries in the Asian region and many local artists. Over 1000 representatives were involved in these consultations, of which the details can be found here: consultations re the measures.

This schema of measurable outcomes of cultural engagement is based on the premise that cultural products and activities do not hold intrinsic value in and of themselves. Value is generated or experienced as humans engage with the artwork or experience, with different individuals perceiving or receiving this value in different ways. Therefore, the outcomes are not assessing ‘quality’ or ‘excellence’ of the cultural experience, but the impact on the person who engages with it.

This schema posits five domains (cultural, social, economic, governance and environmental), of public policy and activity. All activity can be considered as initiating in one or other of those domains, and all outcomes can also be categorised within them. Activities are assumed to have outcome/s in the domain in which they are sited, but they may also have outcomes in other domains, depending on their focus. For example, cultural activities obviously have cultural outcomes, but may also have economic and social outcomes.

In this approach, it is assumed that outcomes occur at an individual level, which, when aggregated, can become community, society or population level change. This is distinct from many other evaluation approaches that posit that there is different types of change at different levels (e.g. Brown 2006; O’Hagan 2016). Thus, data can be elicited by understanding the experience of individual participants, either by direct questioning, participant observation or other methods.

Each of the Outcomes in the schema is presented with a handle, a one word heading that encapsulates the concept, supported by short and detailed Descriptions. These headings are supported by measures expressed such that change can be assessed, both in amount and direction (positive or negative). This addresses the challenge that positive change cannot be assumed to result from any activity. Evaluation questions are provided that could be used to elicit a response from participants or other stakeholders about each outcome. These questions can be used for direct probing or as a stimulus for other types of data collection.

Evaluation methods that could be used are offered in this detailed resource. Examples of activities that could be supported by policy makers and funders to contribute towards each outcome are listed. These outcomes and measures are mapped to other measurement schema, and supported by Theory and Evidence from outcome studies, along with Processes documented as contributing to this outcome.

The outcomes

This section is a growing resource to support local government and other organisations evaluate outcomes of their cultural development activity. Information provided here is regularly updated as the outcomes and methods to measure them are trialled in different communities and for different events.

This site is a developing resource. As aspects of this framework are revised and refined, changes are made directly to this site.

The cultural outcomes represented on this site are stable and in trials with councils, cultural organisations and producers. We are working with experts in other policy domains to assist with alignment of existing measurable outcomes in domains of public policy such as economy, society, environment and governance. Please check back to the site regularly or contact CDN directly if you have queries or can contribute to the refinement of these other outcome measures.