Questionnaire Design Worklow

Data collection is the first step in data analysis and ultimately gaining knowledge and insights. All too often, a survey is thrown together last minute. It has a poor design and very little input from those who the questionnaire will target. In all honesty, it is pretty embarrassing looking back at ADF’s first survey. The following advice comes from first-hand experiences and lessons learned. This article will help you create a questionnaire that will set the stage for later data collection success.

1. Define the Data Collection Goal

The first step in your data collection workflow should be to define the main goal of the data collection project. A well-defined goal will help to create a roadmap for the project and guide the development of the questionnaire. Here are some questions you can ask yourself at this point to help you establish the purpose of the survey:

1. What questions do we want to answer

   1. When the survey is finished, what are the key things we want to know

2. What information do we need

   1. Why do we need it

3. What are we going to do with the results

Step 2. Develop the Initial Set of Questions

Open Questions

Once the goal of the survey is clear, you can move on to elaborate on the questionnaire. A good technique for creating your questionnaire is to go from the general to the specific. Start with open-ended questions, that is, questions where the person can answer in their own words without having a defined list of answers. Don’t limit the number of questions that get developed at this stage. List any questions that come to mind that align with the goal of the project. These initial questions will serve as the basis for discussion groups that will begin to refine your final set of questions.

Discussion Groups

After establishing the open-ended questions, organize a meeting where you invite stakeholders to discuss the topic of the survey and answer the open-ended questions you just developed. You need to accurately measure the participants’ opinions, feedback, how to interpret and understand the survey, and whether they think it will be useful or necessary.

These meetings will give you a wealth of insights regarding what direction to take your survey. Use these insights in the next step.

Closed Questions

After group discussions, we should develop closed-ended questions that list more people's responses to open-ended questions. Closed-ended questions are questions that provide a predefined range of possible answers to the question.

Questions should:

• Be clear and precise

• Ask only one thing

The responses to close-ended questions should:

• Reflect public opinion

• Be independent (Mutually Exclusive)

  • One cannot replace or stand in for the other

• Be easy for everyone to understand

  • The choice of words is essential to express the meaning and intention of the question

  • Questions should be written in plain language and tailored to the target audience vernacular

Step 3. Review

In this step, you will address errors and shortcomings in the questionnaire.

Internal Review
If you have staff members, have them go over survey questions together and openly discuss their thoughts. Everyone will see each question and its possible answers differently, so opinions and suggestions will differ. This is a good thing. The responders of your survey will have differing opinions and perspectives too. One person might want the overall length of the survey reduced. At the same time, another might think that their favorite section can be expanded. Others may suggest improvements to specific questions based on the wording, vagueness, ambiguity, and a number of other issues.

External Review
Find five to ten people outside of the organization to review the questionnaire. These individuals can be informed stakeholders or specialists in the field of data collection. Give them a deadline of three to five days to carefully go through and analyze the questionnaire and provide their insights and comments on what could be changed or improved. They will see things from different perspectives and give you an idea on whether any questions need additional clarification or if there are any opportunities for asking other relevant questions.

Step 4. Pilot Test

In this step, you will test the questionnaire on five to ten percent of the target population. Give the questionnaire to this subset of the population as though it were the final version. This will bring to light any remaining confusion and ambiguity surrounding specific questions, and provide further recommendations and comments from the responders. Additionally, the pilot test will enable you to determine the time required to complete the questionnaire with a respondent. Overall, a pilot test is a great way to obtain critical input that can be used to improve the survey before it goes live.

Step 5. Final Considerations and Corrections

Final Considerations

Length

Take into account how long it takes to finish the survey. This is crucial in determining whether or not the responders would get discouraged, which can impact the data collected. The amount of time should be sufficient for individuals to supply the information but not so long that they get bothered or feel like they are wasting their time.

Questions Order

The questionnaire should be grouped by topic and go in a logical order, like a conversation. Simple questions should come first.

Corrections

At this point, you have done:

• A correction after the first internal review

• A correction after the external review

• A correction after the pilot test

Ideally, you will repeat these steps until you have a hard time finding any problems.

When doing final corrections, you can ask yourself these questions:

• Does this question help us achieve the goal of the project

  • You can end up removing questions when you realize they don’t align with the data collection goal

• Does this question make sense for data analysis

• Is the list of answers sufficiently complete

• Is the questionnaire too long

• Are there any leading questions

Conclusion

There you have it. By following the steps above, you come out with a decent questionnaire to start your data collection project. It is important to remember that while this is an essential first step of any research project, it is only the first step. You still need a method for collecting the data with the questionnaire you created. Beyond that, you will need data analysis techniques to get insights from the data. In a future post, we will go over useful tools to create the questionnaire and collect data. Spoiler, we use https://web.fulcrumapp.com/ for our surveys.