In this article
Following on from our previous article exploring different types of audience research projects, here we’re going to take a deeper dive into the different methods for fieldwork and data collection to support these studies, and which ones are useful for different research objectives.
We’ll be looking at the following methodologies:
Quantitative methods:
Qualitative methods:
Research projects will often include more than one methodology. It’s common to conduct a survey as well as focus groups, or to include observations and vox pops alongside a survey to get some added context.
Firstly, let’s look at some Quantitative research methods – that produce larger amounts of statistical data that can be analysed.
The traditional method for collecting research data involves having a research interviewer approach people and ask them to take part in a survey. Tablets, smartphones and cloud-based survey platforms have largely replaced pen-and-paper surveys, allowing you to do more interesting and dynamic things in your survey these days using modern software.
Many researchers still swear by face-to-face data collection as the most reliable and accurate methodology. By approaching people directly and making an appeal, an interviewer can convince a wider range of people to take part, whereas other methods are more self-selecting. Employing a random sampling approach, they can help to more accurately represent the makeup of the audience.
By building rapport with the respondent, an interviewer can make the survey more engaging and enjoyable, as well as making sure they understand the questions, probing them for more specific feedback and picking up on interesting points.
It can also provide a useful boost if other methodologies aren’t providing the numbers of complete surveys that you’re aiming for.
It’s worth saying that the added accuracy of this method is dependent on interviewers and project managers having a good understanding of the rigour, neutrality and scientific approach that they should be taking to data collection. If your organisation is conducting face-to-face research, it’s important that staff are trained correctly on the key concepts and potential biases and errors that can occur. If you need support with this kind of training, please do reach out to the Digital Culture Network.
There are a few different approaches that we can group under the umbrella of ‘self-completion’. These are methods where the respondents fill in surveys themselves, without your help or supervision. You may want to employ more than one of these ‘channels’ to guide people to take your survey, to get a higher uptake and to reach different groups.
If audiences are pre-booking to visit or interact with you, sending an emailed link to your survey after their visit is a great way to reach out to them. The audience then completes the survey on their own device or computer at a convenient time. However, communicating how important and useful the survey is to you and making an appeal is important, as completion rates can be quite low without encouragement.
Some venues have a tablet or kiosk available for people to complete an on-site, self-completed survey. However, you need to be sure that you can control this to make sure visitors don’t do anything they shouldn’t do on the tablet. Using device management software, or Guided Access mode, you can restrict the device to a single app. If your survey is only available as a web link, not on a dedicated app, you may need to restrict areas of the screen too, to avoid people navigating away from your survey.
Another distribution method for self-completed surveys is to use QR codes and/or a shortened link to allow people to take the survey on their own device. Or, so that people can complete the survey later, the QR code could instead take them to a form where they enter an email address, and the survey is sent to them to complete at a more convenient time. Or, you could hand out a piece of paper with the QR code on for them to scan and use later.
One important consideration with any self-completion methodology is making sure there’s an accessible route for people who have access needs or aren’t comfortable or familiar using technology to self-complete the survey. Signposting that staff or volunteers can assist them if needed, and giving colleagues the training to do so makes sure that people with access needs can take part, so that you don’t miss out on responses from anyone.
An online panel methodology distributes your online survey link to large databases of people (‘panellists’) who complete it at a relatively low cost. This has generally become the default approach for commercial market research. This service is often provided by large panel companies like Panelbase, PureSpectrum or Dynata but other are available and if you intend to employ this method, you should satisfy yourself that you have sourced the correct provider. If you’re working with the panel company directly, it could cost as little as £2-3 per complete survey if you’re aiming at a broad sample of the public. If you’re employing a research agency to build and deliver your research on your behalf, they will probably mark up the panel fees slightly.
The advantage of using an online panel methodology is that you can gather large numbers of surveys from a wide range of people without having to find them yourself. Online panels are great for market studies and getting a representative sample of the population – including those potential audiences who aren’t currently visiting you and aren’t on your mailing list.
One disadvantage is that not all the responses you get back will be high quality. Sometimes, bots or fraudsters will complete the surveys as they are paid per-complete. In most cases of poor data quality, the people taking the survey may be genuine, but they are unengaged and speed through it as quickly as possible. Research has shown that this is not generally duplicitous or intentional behaviour, but rather a result of poor survey design, user experience and over-long or repetitive questions. If you take steps to identify and remove these poor-quality survey responses, the panel companies will replace them for no additional cost as they are aware of the issue.
The big panel companies generally offer support services to help you configure your survey to work with their software and you can use either their self-service dashboard or ask for a dedicated project manager to control the fieldwork, remove poor-quality completes, and manage quotas. It can still be a bit technical – so if you’re less confident using advanced functions on your survey platform, you might want to engage a freelance survey analyst, or a research or fieldwork agency to run a survey on panels for you.
However, some panel providers now allow you to build a survey directly on their platform, or will do it for you for a fee, allowing you to circumvent some of the technical barriers to entry.
Observations are an interesting form of data collection which doesn’t have to involve gathering any surveys at all. It’s an exercise that works best when evaluating how people interact with a space, for example, a room in an exhibition, or wayfinding in a lobby. An observer is placed in a room (or camera footage is reviewed) and visitors selected at random to be observed and details are recorded about what they did.
You might record in what order they interact with different displays and objects, how engaged they appear to be with them, if they speak to a gallery attendant, and how long they spend in the space. You might also record some very basic observable details about the respondent, e.g. are they visiting alone, with another adult, or with children. In some cases, the person observed can then be approached to complete a survey, and the observations recorded would be matched up to them. However, there’s no guarantee that someone you observe will also agree to be surveyed.
Another newer methodology involves the use of eye tracking glasses or cameras to monitor where people are looking as they interact with something. It’s quite a common method in website user-experience testing – to track and generate statistical data and ‘heat maps’ to show to where a person’s attention is drawn.
Some researchers are bringing the technology into physical spaces, especially retail and shop displays, wayfinding information or exhibitions and galleries, to find patterns in the way that people view products, visual content or information. Currently, mobile eye tracking glasses and software are expensive to buy, but they can usually be rented from the manufacturers for a more reasonable price.
One advantage is that the technology can record accurately and quantify where a person’s attention was directed and in what order, which respondents may not be able to accurately recall if asked – but it’s only one data point. It would likely be useful to combine eye tracking with other methodologies to find out why people focused on certain things and what they thought about them.
You could recruit people in-venue to take part in eye-tracking, or they could be pre-recruited and incentivised.
Most Quantitative methods usually don’t need to be incentivised, unless the interview length is longer than 15-20 minutes. In some cases, an incentive could be used to tempt more people to take part, such as for methods like survey kiosks or QR codes, but in general a good appeal from a member of staff or good copy with an explicit call to action can be more effective. Incentives often offered include money or a prize draw entry, you should decide which incentive fits best with the project budget if you are considering this option.
Let’s look a few methods for gathering qualitative data – this is often more free–form, conducted with much smaller numbers of people (depending on the method, scale of the project and number of subgroups to research, you might speak to anywhere between 5 and 30 people). By gathering stories, asking questions in greater depth, or uncovering unexpected topics, you can fill in the gaps or make sense of trends in the statistical Quantitative data.
Simply put, a depth interview is an in-depth conversation with someone. Rather than asking them to answer multiple choice questions, you’re aiming to have a longer and more freeform conversation with them to explore topics. An interviewer would typically have a discussion guide covering several areas, along with prompts, as well as allowing time to talk about examples or tangential topics.
Depending on the number of topics for discussion, and how much a respondent can speak to them, a depth interview can be anywhere from 15 to 60 minutes, though you should try to manage expectations with an agreed length in advance.
If you’re recruiting and interviewing members of the public, it’s standard practice to offer a pre-agreed incentive as a thank you for their time, particularly for longer conversations. However, colleagues, people in your professional network, or other stakeholders might be willing to talk to you for free.
A vox pop is a much shorter interview (5-10 minutes) that is qualitative in nature, so more freeform than a quantitative survey. Typically, the idea is to generate quick and useful insight or quotations that can help illustrate your research findings.
Although a couple of straightforward yes or no questions might be useful to establish facts about the respondent, typically you’ll want to ask more open questions and probe for useful insights.
Rather than pre-recruited respondents, vox pops are usually gathered by ‘intercepts’ – people or visitors you’ve approached at random. Usually around 5-10 minutes long, vox pops are typically not incentivised.
A favourite of the strategist and the marketeer, a focus group is a small group discussion (usually around 5-6 participants) like a depth interview; but going around the group to make sure everyone gets a turn, and encouraging discussion within the group if time allows.
It’s unlikely that everyone in a group will agree on everything, but ideally a group should be composed of people who have a similar behaviour or profile, for example one group of your current audience, and one group of potential audiences who haven’t yet visited you. This way the questions to each group can be made more relevant.
An experienced moderator is a major asset to keep things moving and prevent the discussion going off the rails, and to stop people dominating the discussion where possible. Focus groups might explore concepts, or perceptions and responses to images or content. Some focus groups include activities such as drawing mind-maps about concepts, or post-it note exercises for people to jot down and stick their thoughts on images or stimulus materials.
Usually between 1 to 2 hours, focus groups are generally pre-recruited and offer monetary incentives to those taking part. Having refreshments available throughout also helps to put respondents at ease. They can be held in-person or online, though a higher incentive should be considered for in-person groups to account for travel costs and time.
A popular, recent alternative to focus groups is an online community. Participants will be invited to join a group chat – usually on a dedicated online space or platform designed for this purpose. Professional platforms include Recollective or Incling, but you could use something like Slack or Google Groups as a DIY solution.
Typically, two or three topics per day are revealed and participants are asked to comment and give feedback on them. While they would ideally engage each day, they can give feedback when it’s convenient to them, so this methodology is useful for busy people or groups who have different schedules that would be hard to bring together into one session. It’s also useful that participants can take time to reflect, and respond to each other’s feedback, so there is more time and space for discussing tangents and related topics than focus groups, when the emphasis is always on keeping it snappy and moving to the next question.
Again, participants would be pre-recruited and usually offered a cash incentive. Usually an online community discussion will run for around 3-5 days total, and potentially over the weekend when most people might be more available to contribute. When setting the incentive, you should consider the total time commitment you expect to be required over the lifespan of the community.
A mystery visit is often more focused on a visitor’s experience or journey. It involves recruiting somebody to visit your venue, digital offer, or event. They will have some instructions about things that they should try to do, as well as questions to feed back on. A mystery visit might cover things like whether staff approached them, what information they were given, facilities, and general impressions.
Some companies offer panels of mystery visitors and you can arrange to have it done on a regular basis. Or you may decide to recruit members of the public directly or via a fieldwork agency to carry out mystery visits, typically incentivised in cash.
In the last decade or so, the widespread adoption of smartphones has led to the emergence of a new methodology. Using dedicated platforms and apps like Indeemo or Ethos, participants are asked to make a visit to a venue or experience, and usually record video responses (though other response types are available) to questions or prompts which are revealed to them at various points.
This can be fully tailored to the research goals you have – it might be that you let them loose in a large museum or gallery for a ‘normal’ visit and see what takes their interest and how easy it is to find their way around. Or you could point them to specific areas and ask them for responses on defined topics.
Video ethnography is quite novel and has lots of applications – it’s a bit like user experience testing in the physical world. Combining elements of vox pops, focus group testing, and mystery visits, it’s at its best when it focuses on the in-the-moment emotional responses and reactions that people have during their experience. Because of this, it generates engaging video clips that can be presented to colleagues to illustrate key findings and topics.
Similar to a mystery visit, respondents are usually pre-recruited, briefed on using the video ethnography app, and incentivised.
As you’ve seen, for most forms of Qualitative research, incentives are recommended as the duration of interviews or activities tend to be longer. The UK’s Market Research Society (MRS) code of conduct also suggests that incentives should be cash, or vouchers unrelated to the organisation being researched.
For example, it would not be acceptable to give a gift voucher for your café, shop or free admission to your venue or events, because this could be seen as promoting your brand rather than purely conducting research. It also might influence the responses and participants, both by self-selection of people who want to visit again or use your other services, and by positively influencing people’s perception of your organisation.
In most of the Qualitative methods described above, you will need to make a recording of the conversation, research session or interaction with your respondent. It’s very difficult to take notes during a focus group or interview and keep the conversation flowing at the same time.
For face-to-face methods like vox pops or in-person focus groups, most researchers use a dictaphone, or you could record using a sound recording app on your phone, but make sure you’ve got enough battery charge and free storage space to avoid any mishaps. In a focus group situation, it’s common to use more than one microphone, either connected to a splitter or wirelessly, feeding into the voice recorder. This allows audio coming from different people around the room to be captured more clearly and distinctly than a single built-in microphone on the smartphone or recording device. It’s also good to consider background noise – conducting vox pops in an echoey, noisy lobby or on a windswept hillside may lead to disappointing results.
Online platforms like Zoom and Teams allow you to make recordings of virtual sessions, but make sure you check the features available to understand if there are any limitations depending on whether you have a free or paid account. Again, make sure you have enough free storage space where the recording will be saved (this could be on your own computer, or online in the ‘cloud’ in your account) as video recording files can be very large. Long sessions could use several Gigabytes (GB) of storage space – it’s worth doing some tests in advance.
Phone conversations can be more difficult to record. Probably the easiest method would be to find an app that will allow you to make a recording of a phone call on a smartphone, though these generally require you to pay a fee. There are devices that can be used to ‘tap’ a landline phone between the phone unit and the wall socket, and can be connected to a dictaphone, but this can be unwieldy. If you use speakerphone and a dictaphone, the sound quality will likely be quite poor and it could be hard to make out parts of the recording.
It is vital that you follow all legal processes when collecting data and recording your respondents, this is covered more later in the article.
Once audio and video recordings have been captured, they are usually transcribed. This allows the contents to be more easily analysed, as the text is searchable, and can be tagged or rearranged to be understood as part of different themes or topic areas.
Traditionally, recordings would be sent off to professional transcription agencies to be transcribed by humans. However, with improving speech recognition technology, there are now services like Otter which use machine-learning and language models to produce automated transcription. The results aren’t always perfect, but you can edit and correct any errors on the platform, and over time the services even grow to learn who is speaking and any specific words or jargon used regularly. There are also hybrid services, like Rev or TranscribeMe, which offer both AI and human transcription, allowing you to pick an approach depending on your needs.
Recordings with multiple speakers and crosstalk still pose a challenge for even the best automated transcription services, so raucous focus groups might need more correction than a one-on-one interview.
For most research methods, simply making your organisation’s privacy policy available to respondents is sufficient for GDPR purposes; but it must cover the scope and purpose of any processing of personally identifiable data. This includes any third parties involved, with whom you should have data sharing agreements, and the duration records will be kept before deletion. This is a legal requirement if your survey is collecting any personally identifiable information such as the respondent’s name and contact information.
Even if you don’t gather directly identifiable information, if a combination of indirect data could be used to infer a person’s identity then this can still count as personally identifiable information. For example, full postcode, age, gender, ethnicity and job title used in combination could be used to identify a specific person.
If you are making an audio or video recording of the respondent, as well as a robust GDPR-compliant privacy policy, standard practice is to also gather written consent that it is OK to make a recording. Explain the purpose and scope of how it will be used, lay out who will see or hear the recording or transcript (ideally you would specify that it would be used for internal research purposes only), and set a timeframe for deletion of the recording. This is because audio and video recordings of a living person count as personally identifiable information under GDPR and data protection legislation.
Now, we can look at which data collection methodologies covered above are suitable for each of the common research project types in the creative and cultural sector which we covered in our previous article. The table below shows which methodologies you may want to consider, depending on the aim of the research you’re carrying out.
Methodology Matrix
A good example of one project bringing together different research goals and data collection methods is a major audience research project conducted by Morris Hargreaves McIntyre for New York Philharmonic (often known simply as NY Phil), a philharmonic orchestra based in New York.
By conducting online self-completion surveys and focus groups with two groups, their existing regular audience (via NY Phil’s mailing list) as well as using online panel survey methods to interview the wider local population, with lots of questions asked to both groups and specific questions for current and potential audiences, they were able to produce one large dataset to answer a range of research objectives.
The research objectives covered a range of topics:
By researching a range of topics with different groups, Morris Hargreaves McIntyre were able to show opportunities to tap into a larger-than-expected classical music audience. The research identified the experiences, messaging and programming that would convince lapsed and potential visitors to attend.
By applying Culture Segments to the data, ways to counteract negative perceptions about classical music as an artform were identified that made an appeal to the main motivations of different audience groups.
Of course, audience research doesn’t have to involve multiple strands, methodologies or spending a lot of time and budget. If you have a clear strategy in place, it can be as simple as collecting a few pieces of key data to see how effective it is. If you’re not sure what to do next to develop your audience, think about what you’d like to ask the public and find out about them, and design your research methods in response to those needs.
If you’re interested in taking your first steps, scaling up your existing data collection or thinking about new approaches and projects, why not book a call with Jack? Creative and cultural organisations and individuals in England can access unlimited free 1:1 support from the Digital Culture Network.
Already running an audience survey? Check out our top tips to get the most out of it here.
Beginner Read Audience Data Collection and Evaluation
Audience research is the process of gathering information about the people who visit you or see your work, use your services, or engage with your organisation.
Beginner Read Audience Data Collection and Evaluation
Audience surveys are a valuable tool for organisations that want to learn more about their target audience. These top tips should help you focus on what matters and get results.
Beginner Read Audience Data Collection and Evaluation
There are many different types of audience research projects. In this article we’re going to take a look at a few types of research projects that creative and cultural organisations and individuals might carry out, and the research objectives that underpin them.
Beginner Read Audience Data Collection and Evaluation
Audience research is the process of gathering information about the people who visit you or see your work, use your services, or engage with your organisation.
Beginner Read Audience Data Collection and Evaluation
There are many different types of audience research projects. In this article we’re going to take a look at a few types of research projects that creative and cultural organisations and individuals might carry out, and the research objectives that underpin them.
Beginner Read Audience Data Collection and Evaluation
Audience surveys are a valuable tool for organisations that want to learn more about their target audience. These top tips should help you focus on what matters and get results.
