How to analyze open-ended survey responses (without reading all 500)
Three ways to turn hundreds of written survey answers into themes and reasons you can act on, what each costs in time, and when each is worth it.
Ashik Salman P
· 7 min read
To analyze open-ended survey responses, sort every written answer into themes, note how each person felt, count how many raised each theme, and keep a few quotes in their own words. You can do that by hand, with keyword counts, or with an engine that reads every answer, and the right choice depends on how many answers you have and how many languages they arrive in.
The scores in a survey tell you that something moved. The written answers tell you why. A drop from 8.1 to 6.4 starts a meeting; "registration timed out twice and I nearly gave up" ends it.
What "analyzed" should give you
Before choosing a method, decide what you need at the end. A useful analysis of open-ended responses gives you five things:
- Themes: the subjects people wrote about, named so a colleague recognizes them.
- Counts: how many people raised each theme, counted, not estimated.
- Feeling: whether each theme was praise, complaint or mixed.
- Quotes: two or three answers per theme, word for word.
- Requests: what people asked you to change, kept apart from what they disliked.
If a method can't give you all five, you'll end up reading the raw answers anyway.
Method 1: reading and coding by hand
The classic approach is a codebook. Read the first 50 answers, write down the subjects that keep appearing ("registration", "food", "wifi"), then go through every answer and tag it with one or more codes. Count the tags, pick quotes, done.
It works, and for 50 answers it's the best option: you'll understand your respondents better than any tool can tell you. It stops working at scale. Coding 500 answers takes most of a day, two people coding the same set rarely agree on more than three quarters of the tags, and by answer 400 you're skimming.
Method 2: keyword counts and word clouds
The fast option: count the words that appear most and draw them large. It takes seconds, and it misleads in predictable ways:
- Negation: "not bad at all" counts as "bad".
- Sarcasm: "Brilliant. Forty minutes in a queue" counts as praise.
- Synonyms: "lecturer", "teacher" and "sir" become three small words instead of one big subject.
- Mixed languages: "slow aanu, randu thavana try cheythu" (Manglish for "it's slow, I tried twice") contains no English keyword at all, so it vanishes.
Word clouds are fine for a first glance. They're not an analysis.
Method 3: an engine that reads every answer
The newest option reads each answer the way a person would: what it's about, how the person felt, and what they asked for. Then it groups answers into themes. Done well, it gives you the codebook result in minutes. Done badly, it gives you a confident paragraph with invented numbers.
One answer, read: three subjects, and how the person felt about each.
Before trusting one, check three things:
- Are the counts counted? Every number should come from your responses, not from the model's estimate. Ask where "47 people mentioned registration" comes from.
- Are the quotes real? Quotes should be copied from answers, never paraphrased.
- Does it read every language your people write in? Including Romanized text like Manglish and Hinglish, which translate-first tools miss.
The three methods compared
| By hand | Keyword counts | Engine reading | |
|---|---|---|---|
| Time for 500 answers | most of a day | seconds | minutes |
| Consistent between runs | partly | yes | yes, when counts come from data |
| Catches sarcasm and negation | yes | no | yes |
| Mixed languages | if you read them all | no | yes, if it reads them natively |
| Best for | under 100 answers | a first glance | 100 answers and up |
A worked example: 30 answers from a partner summit
Sample answers written for this article; the analysis is the engine's unedited output.
We wrote 30 answers to "What's the main reason for your score?" after a fictional partner summit, in English, Malayalam, Manglish, Hindi and Arabic, then let the AqiloPulse engine read them. Its summary opened:
Respondents praised the practical sessions, great speakers, and lovely venue, particularly highlighting the networking dinner and workshops. However, widespread frustration was directed at technical and logistical issues, most notably the failing registration process, long food queues, and unreliable wifi.
The themes, with counts taken from the 30 responses:
| Theme | People | Feeling | One answer, as written |
|---|---|---|---|
| Registration | 8 | negative | "The registration form timed out twice before it took. I nearly gave up." |
| Sessions | 7 | positive | "Sessions were good but getting registered was a pain." |
| Food | 4 | positive | "Lovely venue, but lunch took forty minutes to reach." |
| Workshops | 3 | positive | "Loved the workshop on research methods. Want more of those." |
| Wifi | 2 | negative | "Wifi kittiyilla, slides download cheyyan pattiyilla." |
Registration's eight include answers in four languages, among them Malayalam typed in English letters and a line in Arabic. A keyword count would have found five. The smaller themes, such as parking and networking, were still counted and listed, just below the main ones.
Ask open-ended questions that are easy to analyze
Analysis starts when you write the questionnaire. Three choices make the answers far easier to read later, whichever method you use.
Ask after a score, not instead of one. "How likely are you to recommend us?" followed by "What's the main reason for your score?" gives every written answer a number to sit beside. A complaint from someone who scored you a 9 is a detail to fix; the same complaint from a 3 is why they're leaving. Tools that read the answer alongside the score can tell those apart.
Ask one thing per question. "What did you like and what should we change?" invites two subjects in one answer, and half the time people only answer the first half. Two short questions, "What worked best?" and "What one thing would you change?", give you praise and requests already separated.
Keep it optional, and keep it to one or two. Required text boxes fill up with "ok" and "na". One optional question that people actually want to answer beats four they skip. In a short pulse of two to five questions, a single open question at the end is usually enough.
A good open question also names the moment: "What's the one thing that would have made today's workshop better?" gets sharper answers than "Any other comments?", because people answer about the thing you named instead of everything at once.
| Instead of | Ask |
|---|---|
| Any other comments? | What's the one thing we should change before next time? |
| What did you like and dislike? | What worked best? · What would you change? (two questions) |
| Please explain your rating. | What's the main reason for your score? |
| Feedback on the course? | Which part of the course helped you most, and why? |
Mistakes that skew the result
- Reading only the first 50. Early answers come from your most engaged people.
- Translating first. Machine translation flattens Manglish and Hinglish into nonsense, and the complaint never reaches a theme.
- Dropping short answers. "Parking was a nightmare" is short and very specific.
- Merging everything into a few themes. "Logistics" with 40 mentions tells you nothing to fix.
From themes to decisions
An analysis is only finished when someone acts on it. Turn the themes into a short list before the debrief:
- Rank by people, not by noise. The theme eight people raised outranks the one a single person wrote three paragraphs about, however vivid.
- Split by feeling. A theme that is mostly praise is something to protect; mostly complaint is something to fix. Mixed themes usually hide two subjects, so open them before deciding.
- Look for the score behind each theme. If the people who complained about registration also gave you low scores, it's costing you; if they scored you 9 anyway, it's an irritation.
- Name one owner and one change per theme. "Registration: move to a two-step form before the next event" is a decision; "improve registration" isn't.
- Ask again. The second pulse tells you whether the change worked, and the theme count falling from eight to two is the clearest proof you'll get.
Keep the quotes for the people who'll make the change. "I nearly gave up" persuades a team in a way that "registration: 8, negative" never will.
Frequently asked questions
How many responses do I need before analysis is worth it?
Once you have more than about 50 written answers, reading them all carefully becomes the bottleneck. Below that, read them yourself.
Can I trust AI sentiment analysis of survey responses?
Trust it when the counts come from your data and every quote is copied verbatim, so you can check any claim against the answers behind it. Be wary of any summary you can't trace back.
Do I need to translate answers before analyzing them?
Not if the tool reads each answer in the language it was written. Translating first is where mixed-language answers get lost.
Related reading: inside an AqiloPulse readout and Manglish, Hinglish and the comments translate-first tools miss.