Mail bag: How many stories?

Last week somebody asked me a question via email that I've already answered lots of times: How many stories should a PNI project collect? 

I was about to say "it's on page whatever in my book," like I usually do, but then I thought -- why don't I write something new this time, just to see what happens? I'm glad I did, because I think my answer is getting better as I keep doing more projects. Anyway, here's what I wrote. Maybe it will be helpful to you as well.

The "how many stories" question comes up often when people are planning story projects. The answer is a bit complicated, but it depends on six things: issues, ambitions, abstractions, experiences, engagement, and people.

Issues: One or many?

If you want to talk about one big, simple issue, you need one set of stories. However, if you want to talk about multiple issues, or one very complex issue with a lot of other issues embedded within it, you need more stories. 

One way I like to use to figure out if an issue is complex is to keep asking "And what issues lie within that?" and then stop when the answer is "there aren't any issues within it."

For whatever number of stories you plan to collect, you must multiply it by the number of discrete issues you want to talk about. For example, if I wanted to help people talk about jobs and homelessness, I would gather two sets of stories (with some common questions to tie them together), so people can explore each issue with the depth it requires.

Ambitions: Exploratory or in-depth?

If you want to:

• prove without a doubt that something is happening (in a way that cannot be dismissed),
• represent the voices of people who have not been heard (in a way that cannot be ignored),
• help people think through an issue deeply enough to arrive at useful conclusions and plans (in a way that will not fall apart later on), then
  
• you need more stories than if you just want to explore a topic and see what happens.
  

Ambitious projects need 2-4 times as many stories as exploratory projects. In an ambitious project, the patterns in the stories must be obvious, plentiful, and complex enough to be explored in depth. In an exploratory project, it's okay if the patterns are just interesting hints at things people might want to explore more fully in the future. 

Abstractions: Concrete or vague?

If you want to explore abstract issues that are difficult to explain in ordinary words, you will need more stories than if you want to explore simple, concrete issues. 

For example, say you want to know how people feel about the new traffic lights in your neighborhood. You can just ask people how they feel about the new traffic lights in your neighborhood. But if you want to explore how your community is building resilience for a 21st century future, or some other string of jargon that means a lot to some people and nothing to others, you might have trouble gathering relevant stories. Most likely, you'll get a lot of "scattershot" stories based on people's guesses as to what you might be asking them to talk about.

A good test is to write down a question you would like to ask people, then translate it into simple, everyday language. Search for the "1000 most common words" in whatever language the question will be in, then remove all the words in the question that are not in that list. Then ask yourself: if you frame your question in common words, will the stories told in response adequately address the issue you want to address? If yes, just ask the question that way, and you're fine. You won't need extra stories.

But if rephrasing your question with common words will push it far away from the issue you want to address, then you will need to collect more stories, so that some of the scattershot stories you collect will fall onto your target.

Experiences: With stories, or with stories and patterns?

If you want people to meet in rooms, share stories, and do some sensemaking exercises together, you can gather as few as twenty stories per session. You might do that a few times within a project, but as long as it's people talking, you can see and work with patterns in a few dozen to several dozen stories.

On the other hand, if you want to do what I call catalysis (which is just analysis without the definitive conclusions), you need at least 100 stories to start finding statistical patterns in your data (answers to questions about stories). At 100 stories most such patterns tend to be weak. At 150 or 200 stories patterns are stronger (and less likely to be considered fake or irrelevant). I get pretty nervous when I have do catalysis with only 100 stories to work with. At around 200 stories I start to feel more comfortable, because the patterns I find are easy to see and talk about (without worrying that people will say "there's nothing there"). 

This more-is-better trend continues until about 600 stories, when you start running into diminishing returns. At that point you are better off using your time to collect stories on a different issue (unless, of course, some other aspect of this list means you need to push the number up for other reasons).

Catalysis is not important to, or even advisable for, every PNI project. Sometimes you do need to generate a lot of graphs and statistics. But sometimes you can get the same result with fewer stories by having people work with the stories directly, in sensemaking exercises. It all depends on what sorts of experiences you want people to have.

I always advise people to imagine the people they want to help or reach (whoever they are) responding to patterns in the stories and other data they plan to collect. If you can picture those people looking at graphs and statistical patterns and saying, "Oh, wow, now I get it," then you want those things to show those people, so you need catalysis.

But if you can picture the same people saying the same things because they are working with the stories directly (i.e., without any graphs and statistics), you don't need catalysis. In fact, it might be a bad idea. It might waste time you can use for other, more important things, like talking to more people, holding more sessions, covering more issues, getting more stories to more people, helping more people learn how to gather and work with stories, or iterating over the project more times.

On my web site I have an excerpt from a catalysis report which a client allowed me to share. If you look at it, you can see what the patterns that come out of catalysis tend to look like. If that seems like it would not be useful to your project, you don't need catalysis, and you don't need hundreds of stories. On the other hand, if that sort of report seems like just the kind of thing you need, then you can look at the numbers listed on the second page of the report. Those are typical numbers for projects that support catalysis well.

Engagement: Deep conversations, or messages in bottles?

A lot of "what works" in story work has to do with facilitation and engagement. I once saw a project with 80 stories work far better (in the sense of generating more useful insights) than a project with 1600 stories. 

• The stories in the first project came from a group session with 20 people that was carried out by an expert facilitator who helped the people in the room feel welcome, safe, and heard. As a result, the people really spoke to the issues, and their stories and answers to questions contained many striking insights. 
• The second project used a web form that had embedded in it some constraining expectations about what respondents ought to say. Those 1600 people said more surface-level things, so even with 20 times more stories, less useful insight came out of the project. It was still a good project, but it did not explore its issues as deeply as the project with 80 stories.

So there is a quality-quantity balance. The more quality you can get in your stories (in terms of how deeply and authentically people can explore the issues at hand), the fewer stories will provide the same result. Conversely, if for some reason you cannot gather quality stories (maybe people are reluctant, or you can't talk to them in person), a greater quantity of stories can make up for it, to some extent. 

On some projects, quality is the primary constraint (so you need more stories), and on other projects, quantity is the primary constraint (so you need deeper engagement in the stories you can collect).

People: Small or large community? Small or large need?

The more people you want to listen to, and the more the people in that group need to feel heard, the more stories you need to collect. Participatory story work never results in statistical sampling (because it's self-selecting), but you do need more stories to talk about issues in a community of 10,000 than in a community of 100. And you need more stories in a community with a strong need to be heard than in a community where people have already had plenty of chances to speak up.

My general rule is that if at least 20% of the people in any community have shared stories in a project, people tend to feel that the collected stories are representative of the community. In cases where people in a community feel especially unheard, that percent has to go up, maybe to 30% or 40%. The story collection also has to be balanced to represent all relevant viewpoints, but that is the shape of the collection, not its size. 

Having said that, a rule of thumb based on percentage doesn't work as well if the population is huge. If, say, there are 50,000 people in a community, hearing from 20,000 of them might pose logistical problems. I have seen story projects collect 10,000 stories, but it's not the norm. Most projects have fewer than 1000 stories, just because the people doing the projects have limited time to gather and work with the stories.

In the case of a larger community, it's reasonable to say that 20-40% of the community should be invited to share stories. After all, it's more about who is allowed to speak than who actually speaks.

Web-based surveys tend to get a 5-10% response rate, so if 20,000 people are invited to speak, you would get something like 1000-2000 stories, which is doable logistically.

If stories are collected in person, in interviews or groups, it's hard to get 1000 stories, even if you invite 20,000 people. It takes more time and energy to come to a session or interview than to fill out a web form, so instead of a 5-10% response rate you will tend to get more like 1%. On the other hand, stories gathered in interviews and sessions are so much deeper and richer than web-collected stories that smaller numbers of stories may not be a problem (see above).

Another thing is that, if a project contains multiple sub-projects that explore different issues (also see above), they can together add up to hearing from 20-40% of the population, even when the population is large. You can link sub-projects together by using some common questions. If you do that, you can get to huge numbers of stories, spread across sub-projects within a larger, overarching project.