Questionnaire design

Start from your objectives before you write any question

Take your synopsis and list the objectives. Against each one, note what you must measure to answer it. These are your constructs. For example job satisfaction, perceived ease of use, brand loyalty, anxiety or financial literacy. Next to every construct, write which scale you will use and which test you will run on it.

We call this small table the questionnaire blueprint. Whatever you do later in the study comes from this table only. The illustration below is one for a small study on bank staff. Many scholars think it is only paperwork. We still ask every scholar to show it to the guide first, before printing anything. From the questionnaire blueprint you can see, well before data collection, whether your data can answer each objective.

Then check the questionnaire blueprint both ways. Every objective needs some items, and every item must link to some objective. Many scholar add a few “interesting” questions, for example on social media use or monthly savings, which no objective needs. They make the form longer, tire the respondents and are never used in Chapter 4. Remove them.

Borrow tested scales, and say so

In management, education, psychology and nursing, most constructs already have a scale that someone has developed and tested. Use that scale. A published scale comes with reliability figures and a validity record that the examiner can check. Ten items written by you have no such record.

When you adapt a published scale, change as little as you can. Suppose the scale says “organisation” and you put “bank” in its place, that much is no problem. If you remove three items, reword five and add two new ones, then you are no longer adapting the scale. It is a new scale and you must test it like one. In the method chapter write down each change you made, and cite the published scale as the source.

Two more checks. Some scales are under copyright and need permission or a licence, so look at the original paper or the author’s page before printing. Also use only one scale for one construct. Three items from one source and four from another, added into one “customer satisfaction” score, usually give a weak alpha and a difficult viva question.

Are you planning AMOS or SmartPLS for analysis later? Then keep at least three items for each construct. The reason is in the measurement model, and we have written about it in our note on AMOS vs SmartPLS.

Writing items people answer the same way

Sometimes you have to write your own items, for background questions or for a construct where no published scale is available. Then the goal is that all respondents understand the item in the same way. These are the mistakes we are seeing most in thesis questionnaires, with corrections.

Two questions in one. Take “The training was useful and well organised.” A respondent who found the training useful but badly organised do not know what to tick. Make it two separate items.

Leading wording. If the item reads “Don’t you agree that online banking saves time?” then the respondent feels he should say yes. Write “Online banking saves me time” and leave the answer to the scale.

Negatives that tangle. Someone reading “I do not feel unsupported by my manager” has to read it twice to get the meaning. Never use double negatives in your items.

Overlapping options. With age groups of 20–30, 30–40 and 40–50, a person aged 30 fits in two boxes. Write 20–29, 30–39, 40–49 instead. Each person filling the form should get some box that fits him, and for that “Other” or “Prefer not to say” is handy.

No time frame. For “How often do you use UPI?” the respondent does not know what period you mean. Add “in the last month” or “in a typical week” to the question.

Likert scale choices: points, labels and direction

Scholars keep asking whether to use five points or seven. Both are fine for a thesis. Respondents filling a paper form in their second language find five points easier. Seven points spread the answers slightly more. When you borrow a published scale, keep the format it already uses, and keep one format for your own items.

Give a label to every point of the scale and not just to the first and last. For example “1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = Agree, 5 = Strongly agree”. When only numbers and two end words are printed, people start guessing what 2 or 4 means.

We usually tell scholars to keep the neutral option in the middle. We remove it only for a clear reason, because some people really don’t know, and if you force them to agree or disagree you only get noise in the data.

Some items are worded in reverse. They are useful to catch a respondent who ticks the same column in every row till the last item. In a commitment scale, one example is “I often think of leaving this job”. These items have to be recoded before you total the scale. On a five-point scale, 1 is changed to 5, 2 is changed to 4, and the rest follow the same way. Forgetting this mostly gives a low or even negative alpha. Our guide on Cronbach’s alpha goes through this problem in detail.

Translating into Tamil, Telugu, Kannada, Malayalam or Hindi

In South India many theses collect data in the respondent’s own language. Farmers, factory workers, school students and many patients give more honest answers in their mother tongue than in English. Still, a translated questionnaire is a new instrument till you show it measures the same thing.

The common method is back-translation, which Brislin described in 1970. Say your study is in Tamil. One bilingual person translates the English questionnaire into Tamil. Then a second person, who has not seen the English version, translates that Tamil version back into English. Compare the two English versions item by item. In some items the meaning will have changed, and for those both translators should sit together with you and fix a better wording.

Keep forward and back translations as appendix in the thesis, and in the method chapter give the qualifications of both translators. The pilot also must be done with the translated questionnaire. Doing the pilot in English is of no use here. A word may look correct to the translator and still mean something different to a respondent in a village.

Order, length and the first page

The first page should have a short consent note. Say who you are, what the study is about and how many minutes it takes. Also tell them the answers are anonymous and they can stop in the middle if they want. Where an ethics committee has approved the study, put the approval number on this page.

Keep the easy, neutral questions in the beginning. Sensitive questions on income, health or conflict at work can come later. Background questions many researchers put at the end, so that respondents give their attention to the main scales first. Some others put them at the start. Both ways are fine, only keep to one order in the whole form.

Fifteen minutes or so to fill the form is a good limit. After 60 to 80 Likert items, people start ticking the same column just to finish. If your questionnaire blueprint has more items than that, go back to it and ask which objective actually needs each scale.

In Google Forms, make the main scale items required. When only some people can take part, put a screening question in the start itself. Don’t switch on email collection if you don’t really need it, because then the survey is no more anonymous.

The pilot, and showing the questionnaire is valid

Before the pilot, give your draft to a small panel of subject experts. Usually they are faculty from your department or another college. Ask each of them to rate every item on relevance and clarity. Those ratings become your evidence for content validity. Lawshe’s content validity ratio is one formal way to score this. Its table of critical values begins at five panellists, so get at least five experts if you use it. Many Indian theses report a simple content validity index in place of it.

Next, run the pilot with 15 to 30 people who are similar to your actual respondents. Write down the time each person took to finish. With two or three of them, sit nearby and ask them to speak out what they are thinking as they answer. Mark where they pause or ask you the meaning of an item. In the pilot data, look for items everyone skipped and items where everyone gave the same answer.

Alpha can be calculated on the pilot data also, though with small numbers we treat that figure only as a rough idea. With only 20 respondents, alpha moves up and down a lot. Use the pilot to correct the wording. The final reliability you report should come from the main sample and not from the pilot. If you have not yet fixed the size of the main study, our guide on sample size for a thesis will help.

One respondent answering everything

In most thesis surveys one person rates the cause and also the effect. For example, an employee rates the manager’s support and also rates their own satisfaction. Because the same person gives both ratings, the relation between them may come out stronger than it really is. Podsakoff and colleagues called this common method bias in their 2003 review. Many times examiners in management now ask about it in the viva.

Some simple steps at the design stage will help. Keep the predictor scales in one section and the outcome scales in a separate section. Use different anchors for some of the scales. Promise anonymity clearly and keep the promise. Then write these steps in the method chapter. Harman’s single-factor test is commonly reported after data collection, but by itself it is a weak check, so don’t depend on it alone.

When the data comes in, the questionnaire blueprint already tells you which test to run for each objective. To cross-check, see our guide on choosing a statistical test, which has a table of objectives and the tests that suit them.