The best questions about stories

This is another "came up in conversation" post, based on several recent conversations about asking people to reflect on stories they have just told. At this point I've seen people ask questions about stories (and then watched the responses come in) more than a few dozen times, and some fairly predictable patterns are starting to form. I've observed four dominant factors that make the difference between strong, clear, useful answers to questions about stories and weak, messy, can't-do-much-with-them answers. The best story collections get these four things right.

1. Ask questions that matter and resonate.

Find questions that want to be asked. Don't just pull out a standard list: think more deeply, more emotionally, more aspirationally. Think about what you want to achieve with your project -- its fondest hopes and dreams -- and find questions those dreams want to ask. Pretend you are sitting with the stories and answers. In a perfect world, what do they tell you? Then come at it from the other side: what do your storytellers have to say, want to say, need to say? If you can't guess, collect some unadorned stories before you design any questions about them. Find the place where what you need to hear and what needs to be said come together.

2. Transmit your excitement and energy.

You are doing your narrative project because you hope to achieve some goal that is important to you. What is it about that goal that moves you the most? Does that excitement come through in your questions? Do your questions, as a whole, feel like a contribution to something positive and helpful, or do they feel like a tax form? If you can't tell, try your questions out on anyone you can find who is in your group of interest. Watch their face as they look over the questions. Are your questions communicating your energy to them?

3. Avoid conflation.

One of the worst destroyers of answers to questions about stories is conflation: answers that mean "I feel this way" mixed indistinguishably together with answers that mean "I couldn't find the answer I wanted so I picked this instead." Badly conflated answers usually have to be thrown out because they cannot reveal useful patterns.

The best antidote to conflation is pilot testing of questions. When that is not possible, I use and suggest a simple exercise to detect possible conflation. It goes like this. First, make a list of people with widely varying backgrounds and personalities. It's best if they are in the group of people you will be asking to tell stories, but if that's not possible, just consider people you know or remember. They can even be fictional or historical. The important thing is that they be varied in their perceptions.

Now that you have some people in mind, take your question list and mentally picture each person answering the questions. On each question that has predefined answers, picture the person with their pencil hovering over the possible answers to the question. Is there an answer for that person? Do they find something that works for them? If they can't find an answer, what can you do to help them answer the question? You can also multiply the people by situations and picture them telling stories that come out well and poorly, and so on. When you get to the point where everyone can find an answer in every situation, you have a good set of answers.

 4. Use your precious cognitive budget wisely.

Your cognitive budget is the limited amount of time, attention, mental energy, and patience your storytellers are willing to give you. It's almost always far, far smaller than you would like it to be. Your goal is to have your storytellers use as much of the cognitive budget as possible on telling the story and reflecting on it. Think of it like one of those charity reports that tell you how much of your money is going directly to the people being helped. Reduce your administrative overhead to the minimum. Every time someone spends more than a few seconds on a question trying to puzzle out what you want, you have lost some of their priceless narrative reflection.

Here are some of the worst cognitive budget wasters.

Too many questions. This one is of course the grand-daddy of all cognitive budget wasters. Face it: people aren't going to be as prolific with their answers as you would like them to be. But if you don't meet people where they are, you will waste what they do have to give you. It's kind of like that joke about fighting with your spouse: do you want to be right or do you want to be married? Do you want perfection or do you want stories?

The general rule here is: ask three questions, plus two multiplied by your interest level. Rate the interest level like you would a hot sauce: from mildly interested (one) to wildly interested (five). If you are standing in a shopping center accosting people with a clipboard, ask three questions per story, because you can expect a zero rating on the mild-to-wild scale, and beyond three questions you will get little of any utility. If you are running a kiosk, they stopped didn't they, so that's a minimum one rating, so ask five questions per story. As the interest level you expect goes up, keep adding questions.

You can guage interest by knowing your storytellers well and by testing your questions. If this is impossible, you can hedge your bets by making some of the questions optional. This means separating the questions into groups and offering them in stages. Ask the most important ones first, then say something like "if you are done answering questions for this story, you need not fill these in" or "these questions are optional" or "would you please answer these additional questions?" or some such thing. The optional-stages route puts more of a burden on cognitive budget, so you should only do it if your storytellers are extremely varied or there is no way you can guess at what they will answer (and you need as many answers as you can get). A combination of knowing and testing is the better way.

Too many answers. For multiple choice answers, the number of possible answers you should provide follows the same rule as the number of questions: three plus two times your interest level. When a question has too many answers, people either ignore the last ones or ignore the whole question. I've seen questions with too many answers where the first three answers are disproportionately chosen. This usually does not mean those answers are best; it means people looked at the long list, said "ugh" to themselves, and truncated it for you (but each in their own way!). The result is muddied patterns and thrown-out questions.

Note that above I said having too few answers to a question can be a problem, because people can't find the answers they need in the list. This seems like contradictory advice, and it is! You need both accuracy and brevity, and you need to find the best balance between them.

Answers that are too similar. Question designers sometimes try too hard and design overly fine-grained answers. For example, if you are asking "when did this story happen," how much resolution do you need? Do you need to distinguish between events that happened six months and a year ago? When I'm looking over answers I often find random assignments when the meanings of answers are too close together. People rarely have the time or energy to make precise distinctions between feelings. It may sound useful to differentiate between whether people told a story "to explain" or "to inform" but storytellers often will not want to slice things so finely.

An exercise that works here is to explain the differences between answers to yourself (or in a group). If the explanation is long or confusing, maybe you don't need two separate answers. Justify each small difference. For example, say you have "frustrated" and "irritated" in your answers to your "how do you feel" question. Why do you need both of those? What situations are you trying to separate? Do they need to be separated? Do you want to use some of your cognitive budget on having people assign their feelings to those small categories? (Sometimes such a fine division is necessary; if so, be sure to free up some budget from somewhere else to make room for it.)

Nowhere to put non-response responses. This can be a point of pride: sometimes the people designing questions do not want to admit (even to themselves) that those responding may not want to choose an answer. Admit to yourself that sometimes, for some people, they will not have an answer, or will not want to think hard about it, or will not want to tell you what they really think.

My favorite example of this was on a project where we were doing a pilot story collection to test some questions. On the "how do you feel about this story" question, most of the respondents chose the answer "good." They did this for all sorts of stories, from the mundane to the catastrophic. What happened was, not seeing a non-committal option, they decided that "good" was the non-committal option. The clear message was that for this particular set of people telling stories about this particular topic, some of them needed a way to say "I'd rather not tell you that." When we added that answer to the questions, the utility of the story collection improved. Some of the worst misfires I've seen were ones where respondents could not find a place to put non-responses and muddied up the real responses with them.

Inconsistency or poor grammar. I know this sounds silly, but trivial things like subjects and verbs that don't agree, such as "How long ago does the events in this story take place" can stop people up. (One of the worst offenders is the non-question with a question mark at the end, such as "Please rate this story?") Or, say you start out your questions with first-person questions like "I live in..." and then three questions later switch to second-person questions like "How old are you?" You'd be amazed by how much this sort of thing can derail people. Be ridiculously consistent in how you ask your questions. Snags in the flow of the question-answering conversation waste cognitive budget.

Be careful here: once you've been iterating on a question set for a while it all begins to seem perfectly logical even if it isn't. Try your questions out on somebody you can watch, and see where they pause and ponder. Where does their pencil hover? Is it because they are thinking about the story, or because they are trying to figure out what you mean? Increase the former and reduce the latter.

Not being skim-ready. Yes, you know every nook and cranny of your question set; but your storytellers will be skimming it. (Another blow to our precious pride.) A common problem is to write a question like this: "How much do you..." When people are skimming and encounter a question like that, they have to look at the answers to find out what the question is about. Memorize this fact: people don't look at the answers. They skim the questions, and then they come back to the answers, when they think the questions are worth answering.

For this reason, each question has to stand on its own. It has to communicate the intent of the question with or without the answers. The same thing holds for the answers -- they must stand alone as well. Yes, I am actually saying that people forget what question they are answering while looking at the answers to that question. (All who have done this and marked the wrong answer in consequence raise their hands.... Now I have to go back to typing....) So the above question should say "How much do you like ice cream?" And then the answers should say "I like ice cream a lot" and so on. The rule is: don't make people grope for context. That wastes cognitive budget.

Confusing question order. Order your questions like a conversation. Make them flow naturally. If you were asking someone questions in person, which would you start with? Having to switch gears or answer questions in an order that doesn't seem natural wastes cognitive budget.

One thing I've found is that the "how do you feel" question likes to be asked right after the story is told. Doing this also sets the stage for the questions that come after it. It communicates to the storytellers that you will be asking them to think about their personal feelings about the story, not asking them for facts and figures. In general, the further away you get from the story the less vivid their emotions about it will be, so it is often best to ask the more factual questions later. Asking factual questions first can put people into classification mode and reduce their emotional response where you need it most.

Cultural confusion. The issue of whether your storytellers are "educated" is important in asking them questions about stories. By "educated" I don't mean smart or well-informed. I mean something closer to cultural homogeneity, more like "educated about the way people usually ask questions in surveys." Social scientists are used to throwing around precise terms, graphs, scales, ratings, and other social constructs, and sometimes they forget that not everybody is equally used to these things. For example, if your storytellers are not academic scientists, the question "How do you feel about this story?" is far superior to "Please rate your emotional intensity with respect to this story." And if they are academic scientists, the latter version will probably work better than the more conversational, candid, personal form. Think about the mindset of your storytellers and meet them where they are.

And be consistent: don't mix casual statements like "I like my work" with more formal statements like "I am satisfied with my current situation." The answers you write are the storyteller's part of the conversation. If you give them a voice that consistently matches what they would really say if they were speaking to you, they will be more likely to find answers they can choose, and they will waste less time finding them.

Happy question asking, everyone!