Data collection methodologies for creative and cultural audiences
Following on from our previous article exploring different types of audience research projects, here we’re going to take a deeper dive into the different data and feedback collection methods that you can use in your audience research and evaluation.
How do I decide which audience research methods are the right ones for me?
To choose research methods, you should think about the practical aspects of your research and how you’ll conduct it. That means thinking about your available time and resources, and the best fit for your audience.
The most common method for audience research and evaluation is to collect surveys. However, to achieve reliable and accurate data, you need to collect hundreds of surveys – perhaps 200 to 250 as a starting point. In How to decide your survey targets and improve your sampling you can find out more information about survey numbers and data quality.
But you may struggle to collect enough surveys: it could be that you have a smaller audience, or a lack of staff time or opportunities to approach audiences to encourage survey completion. If it’s not possible to collect a good number of surveys, you might find it more realistic to collect audience feedback using more conversational and qualitative methods with a smaller number of people. We’ll explore some of these methods later in the article.
However, your research goals and objectives can also be important to the decision. As you’ll see below, quantitative and qualitative methods can be more suited to different questions – think of all these methods as a set of tools that can be used in different situations and for different purposes.
What does quantitative and qualitative data mean in audience research for arts and culture?
Quantitative data (such as statistical information from a survey) is great for answering ‘what’ questions: what prompted audiences to visit, what happened on their visit, and what things did they enjoy or take away from the experience. It allows you to take a detailed look at trends in audience behaviour and opinions, with a high degree of statistical accuracy and confidence.
Qualitative data (such as written and spoken feedback, or interviews and conversations) can be more helpful to unpick ‘why’ questions: why are audiences behaving in a particular way, why are they more or less interested or engaged, or why do they have a particular response to a creative work or production. Qualitative data allows you explore ideas and stories, and dive more deeply into them.
What data collection methods are used for audience research?
Here are some data collection methods that are most often used by cultural and creative organisations. We’ll explore each one in more detail below.
Quantitative methods:
- Face-to-face interviews
- Self-completion surveys
- Online panel surveys
- Observations
- Eye tracking
Qualitative methods:
- Depth interviews
- Vox Pops
- Focus groups
- Creative evaluation
- Online communities
- Mystery visits
- Video ethnography
Your audience research might include more than one data collection method. For example, It’s common to conduct a survey ahead of focus groups, or to include observations and vox pops alongside a survey to get some added context.
Data Collection Methods – Quantitative
Firstly, let’s look at some Quantitative research methods. Like all quantitative methods, you’ll need to collect larger amounts of responses or data for them to be most effective.
Surveys
Most arts and culture organisations use surveys in their audience research. When they’re done well, they provide accurate and representative data. For some organisations, surveys can be a relatively easy way to get feedback from a large number of people. But, for other organisations, it can be a real challenge to collect enough data for a survey to be statistically reliable and provide useful insight.
It’s also much easier to get responses to your survey when it’s shorter in length – asking someone for 5 minutes of their time is much easier than asking for 10 or more minutes. As a rough guide, each question might take 20 seconds to complete, though it does vary. This means that it can be challenging to narrow down your list of questions to achieve a shorter survey length – when every question seems important and useful!
You can find more information about audience surveys in this article: Top tips for collecting audience surveys in museums, theatre, arts, and the creative and cultural sector – Digital Culture Network
Does asking people in person improve survey responses?
It’s most effective to have a human being make an appeal for a survey. Audiences are more likely to listen to and consider a real person asking them for help, rather than a poster, a QR code on a leaflet, an email, or a survey kiosk in your venue. In-person approaches can provide a useful boost if other methods aren’t providing the numbers of completed surveys that you’re aiming for. However, it involves dedicating more of your staff time.
In-person appeals can also improve your survey data, as they can convince a wider range of people to take part. When people self-select into a survey (for example: responding to an email), you are more likely to find your survey responses skewed towards certain types of people. Those more likely to self-select include older age groups, women rather than men, people who are more comfortable and culturally active, or who have strong opinions and are looking to share them. This can lead to situations where your self-selecting survey respondents don’t fully represent your audience.
With members of your staff or volunteers using a random sampling approach, you can help to more accurately represent the makeup of the audience. You can find more information about in-person approaches, how to randomly sample your audience, and improving the quality of your survey data in this webinar recording, and this article:
Webinar recording – Increase your audience surveys and improve survey data – Digital Culture Network
How to decide your survey targets and improve your sampling
When you’re ready, you can get free one-to-one support including free survey training sessions for your team from the Digital Culture Network: Ask a Tech Champion – Digital Culture Network
Face-to-face interviewing
The traditional method for collecting research data involves having an interviewer help your respondents to complete the entire survey. Many researchers still swear by face-to-face data collection as the most reliable and accurate methodology.
Carrying out the full survey with respondents allows you to build rapport with the respondent. An interviewer can make the survey more engaging and enjoyable. This can help to prevent frustration or audiences abandoning the survey. An interviewer can also make sure respondents understand the questions, and probe them for more specific feedback and picking up on interesting points.
If your organisation is conducting face-to-face research, it’s important that staff are trained correctly on how to approach and engage audiences, and the key concepts including potential biases and errors that can occur – another reason to consider a free survey training session from Digital Culture Network.
Self-completion
Self-completion surveys are when the respondents fill in surveys themselves, without your help or supervision. The survey could be presented or shared in a few different ways. You may want to offer multiple ‘channels’ to guide people to take your survey, to get a higher uptake and to reach different groups. One example is to have staff or volunteers approach audiences in-person, but then hand them a paper survey, a QR code link, or a tablet to complete the survey.
Many organisations still find that paper surveys are more accessible and preferred by audiences for self-completion. However, keeping the number of pages low, and making effective use of space on the page is key to success – audiences may think twice if you hand them a booklet with several pages of questions. If you can fit your whole survey on two sides of A4, you’ll be in a very strong position.
If audiences are pre-booking to visit or interact with you, sending an emailed link to your survey after their visit is a great way to reach out to them. The audience then completes the survey on their own device or computer at a convenient time. However, it’s important to communicate how important and useful the survey is to you as completion rates can be low without encouragement.. A good response rate to a post-visit survey email might be around 10%. This means that organisations with large numbers of ticket bookers will benefit most from emailed surveys.
One important consideration with any self-completion methodology is making sure there’s an accessible route to take the survey. Some people have access needs or aren’t comfortable or familiar using a digital survey. Signposting that staff or volunteers can assist them if needed, and giving colleagues the training to do so makes sure that people with access needs can take part, so that you don’t miss out on responses from anyone.
If somebody indicates that they are too busy to help immediately, handing them a QR code link to the survey to complete later could be helpful. However, typically only about 25% of those people will actually complete the survey, but it’s better than nothing.
Online panel surveys
Have you ever wanted to survey people who are NOT in your audience?
An online panel methodology distributes your online survey link to large databases of people (‘panellists’) who complete it for you, at a relatively low cost. This has generally become the standard approach for commercial market research. This service is provided by large panel companies like PureSpectrum or Dynata.
If you’re working with the panel company directly, it could cost as little as £2-3 for one person to complete your survey. If you’re employing a research agency or consultancy to build and deliver your research on your behalf, they will probably mark up the panel fees slightly.
The advantage of using an online panel survey is that you can gather large numbers of surveys from a wide range of people without having to find them yourself. Online panels are great for market studies and getting a representative sample of the population – including those potential audiences who aren’t currently visiting you and aren’t on your mailing list. You can use panel surveys to try and understand what could motivate new visitors, and see how interested the general public might be, and how likely they are to visit upcoming programming or new offers, products or concepts.
One disadvantage is that not all the responses you get back will be high quality. Sometimes, bots or fraudsters will complete the surveys as they are paid for each survey they complete. In most cases of poor data quality, the people taking the survey are genuine, but they are overwhelmed or bored by long, unfriendly or tedious surveys, affecting the quality of their responses.
If you take steps to identify and remove these poor-quality survey responses, the panel companies will replace them for no additional cost as they are aware that this is a common problem.
There are different ways that you can access panel surveying. One of the easiest ways is to create your survey directly on a panel company’s platform, using their survey builder tools. It’s a lot more complicated to use an existing survey (e.g. one you’ve built on a platform like SurveyMonkey or Qualtrics), requiring more technical knowledge and experience.
It’s worth getting in touch with the Digital Culture Network to ask about panel surveying, to make sure you have a good understanding of how it works and whether it’s right for you.
Observations
Observations are an interesting form of data collection which doesn’t have to involve gathering any surveys at all. It’s an exercise that works best when evaluating how people interact with a space, for example, a room in an exhibition, or how visitors navigate a lobby or larger space.
An observer is placed in a room (or camera footage is reviewed) and visitors selected at random will be observed and details recorded about what they did.
You might record in what order they interact with different displays and objects, how engaged they appear to be with them, if they speak to a gallery attendant, and how long they spend in the space. You might also record some very basic observable details about the respondent, e.g. are they visiting alone, with another adult, or with children.
Eye Tracking
Another newer methodology involves the use of eye tracking glasses or cameras to monitor where people are looking as they interact with something. It’s quite a common method in website user-experience testing – to track and generate statistical data and ‘heat maps’ to show where people’s attention is drawn to.
Some researchers are bringing the technology into physical spaces. That could be useful to evaluate retail and shop displays, wayfinding information or exhibitions and galleries. Eye-tracking data might show patterns in the way that people view products, visual content or information. Currently, eye tracking glasses and software are expensive to buy, but they can usually be rented from the manufacturers for a more reasonable price.
One advantage is that the technology can record accurately and quantify where people’s attention was directed and in what order, which respondents may not be able to accurately remember if asked. However, eye-tracking is only one data point. It would be useful to combine eye tracking with other methodologies to find out why people focused on certain things and what they thought about them.
You could recruit people in your venue to take part in eye-tracking, or they could be pre-recruited and incentivised in cash. However, at the moment, eye-tracking tends to be used in more specialised research, and could be more expensive than other methods.
Incentivisation – Quantitative
Most Quantitative methods usually don’t need to be incentivised, unless the interview length is longer than 15-20 minutes. In some cases, a prize draw might be used to tempt more people to take part, such as for methods like survey kiosks or QR codes, but in general a good appeal from a member of staff or good copy with an explicit call to action can be more effective.
Methodologies – Qualitative
Let’s look a few methods for gathering qualitative data. This is often more free form and conducted with much smaller numbers of people. By gathering stories, asking questions in greater depth or having conversations with different audience groups, you can fill in the gaps or make sense of trends in the statistical quantitative data.
Depth interviews
A depth interview is an in-depth conversation with someone. It might be arranged in advance and could take place in a video or telephone call. Rather than asking them to answer multiple choice questions, you’re aiming to have a longer and more freeform conversation to explore topics. An interviewer would typically have a discussion guide covering several areas, along with prompts, with time to talk about examples or tangential topics.
Depending on the number of topics for discussion and how much a respondent can speak to them, a depth interview could be anywhere from 15 to 60 minutes. You should try to manage expectations with an agreed length in advance.
If you’re recruiting and interviewing members of the public, it’s typical to offer a pre-agreed cash incentive as a thank you for their time, particularly for longer conversations. However, colleagues, people in your professional network, or other stakeholders might be willing to talk to you for free.
Vox pops
A vox pop is a much shorter interview (5-10 minutes) that is qualitative in nature, so more freeform than a quantitative survey. The idea is to generate quick and useful insight or quotations that can help illustrate your research findings.
Although a couple of straightforward yes or no questions might be useful to establish facts about the respondent, typically you’ll want to ask more open questions and probe for useful insights.
Rather than pre-recruited respondents, vox pops are usually gathered by ‘intercepts’ – people or visitors you’ve approached at random. Vox pops are typically not incentivised.
Focus groups
A focus group is a small group discussion (usually around 4-6 participants). They work like a depth interview but each participant gets a turn at every question and group discussion is encouraged if time allows.
It’s unlikely that everyone in a group will agree on everything, but ideally a group should be composed of people who have a similar behaviour or profile. For example one group of your current audience, and one group of potential audiences who haven’t yet visited you. This way the questions to each group can be made more relevant.
An experienced moderator is a major asset to keep things moving, prevent the discussion going off the rails, and to stop people dominating the discussion. Focus groups might explore concepts, or perceptions and responses to images or content. Some focus groups include activities such as drawing mind-maps about concepts, or post-it note exercises for people to jot down and stick their thoughts on images or stimulus materials.
Usually between 1 to 2 hours, focus groups are generally pre-recruited and offer cash incentives to those taking part. Having refreshments available throughout also helps to put respondents at ease. They can be held in-person or online, though a higher incentive should be considered for in-person groups to account for travel costs and time.
Creative evaluation methods
We work in the creative and cultural sector, so why not bring some of that creativity into our data collection? Creative evaluation and feedback methods loosely encompass any methods that utilise creative, imaginative or artistic elements. Ideally, creative evaluation is a more collaborative, inclusive and engaging process that breaks down barriers between the participant and the researcher.
That could include anything from respondents drawing a picture or a cartoon, telling a story, or using dance or music. It might involve imagining the future or a perfect experience or expressing how they feel now or might feel in the future. These kinds of methods may work particularly well with respondents who are participating in a series of workshops or a programme. When you’ve built trust, and might be developing participants’ confidence or creative skills, building in creative reflection and feedback feels like a natural extension of the project.
A few useful resources for creative evaluation are listed below:
The Little Book of Creative Evaluation includes some great examples but also defines and explores the purpose and value of creative evaluation methods.
A webinar from the Centre for Cultural Value that explores creative evaluation with some case studies: Evaluation Principles In Practice | Workshop 3: Taking a Creative Approach
A lovely example of a ‘journal’ or diary approach to an evaluation can be found here: Blank Beauty Book – The Beauty Project
Online communities
A popular, recent alternative to focus groups is an online community. Participants will be invited to join a group chat – usually on a dedicated online space or platform designed for this purpose. Professional platforms include Recollective or Incling, but you could use something like Slack or Google Groups as a DIY solution.
Typically, two or three topics per day are revealed and participants are asked to comment and give feedback on them. While they would ideally engage each day, they can give feedback when it’s convenient to them. This methodology is useful for busy people or groups who have different schedules that would be hard to bring together into one session.
Participants can take time to reflect and respond to each other’s feedback. They allow more time and space for discussing tangents and related topics than focus groups, where the emphasis is always on keeping it snappy and moving to the next question. Some audience groups, such as young people, may also prefer an online community format. They can take part when and where they choose, with fewer barriers to them speaking up and sharing their opinion than an in-person or online focus group.
Participants would be pre-recruited and usually offered a cash incentive. An online community discussion will usually run for around 3-5 days total, and potentially over the weekend when most people might be more available to contribute. When setting the incentive, you should consider the total time commitment you expect to be required over the lifespan of the community.
It’s more common to see an online community approach being managed by a research agency, such as this lovely example from MEL Research and the Royal Shakespeare Company: Royal Shakespeare Company – Longitudinal Online Community in the Culture Sector – Social Research Agency ¦ Market Research Company
Mystery visits
A mystery visit is often more focused on a visitor’s experience or journey. It involves recruiting somebody to visit your venue, digital offer, or event. They will have some instructions about things that they should try to do, as well as questions to feed back on. A mystery visit might cover things like whether staff approached them, what information they were given, facilities, and general impressions.
Some companies offer panels of mystery visitors and you can arrange to have it done on a regular basis. Or you may decide to recruit members of the public directly or via a fieldwork agency to carry out mystery visits, typically incentivised in cash.
Video or digital ethnography
In the last decade or so, the widespread adoption of smartphones has led to the emergence of a new methodology. Using dedicated platforms and apps like Indeemo or Ethos, participants are asked to make a visit to a venue or experience. They’ll usually record video responses (though other response types are available) to questions or prompts which are revealed to them at various points.
This can be fully tailored to the research goals you have. For example, you could let them loose in a large museum or gallery for a ‘normal’ visit and see what takes their interest and how easy it is to find their way around. Or you could point them to specific areas and ask them for responses on defined topics.
Video ethnography is quite novel and has lots of applications – it’s a bit like user experience testing in the physical world. Combining elements of vox pops, focus group testing, and mystery visits, it’s at its best when it focuses on the in-the-moment emotional responses and reactions that people have during their experience. Because of this, it generates engaging video clips that can be presented to colleagues to illustrate key findings and topics.
Similar to a mystery visit, respondents are usually pre-recruited, briefed on using the video ethnography app, and incentivised in cash.
Incentives – Qualitative methods
For many forms of Qualitative research, incentives are recommended as the duration of interviews or activities tend to be longer. The UK’s Market Research Society (MRS) code of conduct also suggests that incentives should be cash, or vouchers unrelated to the organisation being researched.
For example, it would not be acceptable to give a gift voucher for your café, shop or free admission to your venue or events, because this could be seen as promoting your brand rather than purely conducting research. It also might influence the responses and participants, both by self-selection of people who want to visit again or use your other services, and by positively influencing people’s perception of your organisation.
Recording – Qualitative methods
In most of the Qualitative methods described above, you will need to make a recording of the conversation, research session or interaction with your respondent. It’s very difficult to take notes during a focus group or interview and keep the conversation flowing at the same time.
For face-to-face methods like vox pops or in-person focus groups, you could use a Dictaphone or a sound recording app on your phone. Make sure you’ve got enough battery charge and free storage space to avoid any mishaps.
In a focus group, you might use more than one microphone, either connected to a splitter or wirelessly, feeding into the voice recorder. This allows audio coming from different people around the room to be captured more clearly and distinctly than a single built-in microphone on the smartphone or recording device. It’s also good to consider background noise – conducting vox pops in an echoey, noisy lobby or on a windswept hillside may lead to disappointing results.
Platforms like Zoom and Teams allow you to make recordings of virtual sessions but make sure you check the features available to you – there may be limitations depending on whether you have a free or paid account. Again, make sure you have enough free storage space where the recording will be saved (this could be on your own computer, or online in the ‘cloud’ in your account) as video recording files can be very large. Long sessions could use several Gigabytes (GB) of storage space – it’s worth doing some tests in advance.
Phone conversations can be more difficult to record. Some smartphones allow you to record phone calls natively. If yours does not, you may be able to find an app that will allow you to make a recording of a phone call on a smartphone, though these generally require you to pay a fee. If you use speakerphone and a dictaphone, the sound quality will likely be quite poor and it could be hard to make out parts of the recording.
Transcription – Qualitative methods
Once audio and video recordings have been captured, they are usually transcribed. This allows the contents to be more easily analysed, as the text is searchable, and can be tagged or rearranged to be understood as part of different themes or topic areas.
Traditionally, recordings would be sent off to professional transcription agencies to be transcribed by humans. However, with improving speech recognition technology, there are now services like Otter.ai which use machine-learning and language models to produce automated transcription. The results aren’t always perfect, but you can edit and correct any errors on the platform, and over time the services even grow to learn who is speaking and any specific words or jargon used regularly. There are also hybrid services, like Rev or TranscribeMe, which offer both AI and human transcription, allowing you to pick an approach depending on your needs.
Recordings with multiple speakers and people talking over one another still pose a challenge for even the best automated transcription services, so noisy and lively focus groups might need more correction than a one-on-one interview. That might involve paying a transcription service for human transcription, or spending the time to edit and correct the AI transcription yourself.
Privacy and Data Protection considerations
For most research methods, making your organisation’s privacy policy available to respondents is sufficient to comply with UK privacy and data protection law (such as the General Data Protection Regulation or GDPR).
But your privacy policy must cover the scope and purpose of any processing of personally identifiable data. This includes any third parties involved, with whom you should have data sharing agreements, and the duration that records will be kept before deletion. This is a legal requirement if your survey is collecting any personally identifiable information such as the respondent’s name and contact information.
Even if you don’t gather directly identifiable information, if a combination of indirect data could be used to deduce a person’s identity then this can still count as personally identifiable information. For example, full postcode, age, gender, ethnicity and job title used in combination could be used to identify a specific person.
If you are making an audio or video recording of the respondent, as well as having a robust GDPR-compliant privacy policy, standard practice is to also gather written consent that it is OK to make a recording. Explain the purpose and scope of how it will be used, lay out who will see or hear the recording or transcript (ideally you would specify that it would be used for internal research purposes only), and set a timeframe for deletion of the recording. This is because audio and video recordings of a living person count as personally identifiable information under GDPR and data protection legislation.
Methodology Matrix
Now, we can look at which data collection methodologies covered above are suitable for each of the common research project types in the creative & cultural sector. The table below shows which methodologies you may want to consider, depending on the aim of the research you’re carrying out.
Audience research doesn’t have to involve multiple strands, methodologies or spending a lot of time and budget. If you have a clear strategy in place, it can be as simple as collecting a few pieces of key data to see how effective it is. Or if you’re not sure what to do next to develop your audience, think about what you’d like to ask the public and find out about them, and design your research methods in response to those needs.
What next?
If you’re interested in taking your first steps, scaling up your existing data collection or thinking about new approaches and projects, why not book a call with Jack? Creative and cultural organisations and individuals in England can access unlimited free one-to-one support from the Digital Culture Network.
Already running an audience survey? Check out our top tips to get the most out of it.
Further support
The Digital Culture Network is here to support you and your organisation. Our Tech Champions can provide free one-to-one support to all creative and cultural organisations who are in receipt of, or eligible for, Arts Council England funding. If you need help or would like to chat with us about any of the advice we have covered above, please get in touch. Sign up for our newsletter below and follow us on LinkedIn and X (Twitter) @ace_dcn for the latest updates.
