How could we make sense of surveys?
With the proliferation of surveys because, the question now would be, what should we basically know about it to allow us to astutely read survey results?
All of us have a certain degree of familiarity with surveys. In at least one point in our lives, we have possibly been asked to participate in a survey and we have probably made a decision based on survey results (opting for the highest-rated online seller sounds familiar, right?). Thus, it is common for us to encounter survey results, from commonplace findings about consumers’ product and service preferences to findings about the public’s government satisfaction ratings.
On generalizability of results.
While there is a general expectation that survey results should reflect the characteristics, sentiments, and behaviors of the population, we must take note that for results to be generalizable, the survey’s design needs to follow the protocols for generating a representative sample and that results should have passed the standards of inferential testing.
The rationale behind representative sampling is rooted on the science of probability. If the goal is to generalize survey results, then the sample should be representative of the population that it is supposed to describe.
The random manner of selecting the members of the population is a key component.
The survey’s design should thus follow the fundamental ingredients of generating a randomly drawn and adequately-sized set of respondents. A randomly-drawn sample uses probability sampling techniques so that all members of the population would have a chance of being selected as respondents.
Meanwhile, the size of the sample should sufficiently account for the extent of heterogeneity of the population and there are statistical tools employed in computing the most appropriate sample size. Then again, while an adequate sample size is a necessary factor in generating a representative sample, it is not sufficient to ensure population representativeness. The random manner of selecting the members of the population, who will collectively represent the population, is a key component. Without the fundamental elements of a randomly drawn and adequately sized set of respondents, then we cannot ascertain that the findings are derived from a sample that is representative of the population. Thus, if a survey claims to describe the population, then we should review its sampling declaration to verify whether findings were indeed derived from a representative sample.
Furthermore, data derived from a representative sample needs to be tested through inferential statistics to determine whether results can be generalized to describe the population. Findings from a representative sample that pass inferential testing estimate the characteristics, sentiments, and behaviors of the population. However, such findings will not claim 100% accuracy, rather, findings that pass inferential testing estimate population values within an acceptable confidence level and tolerable margin of error.
Conventionally, it is commonplace to read results that are within 95% confidence level and 5% margin of error, which means that the findings estimate the true population value 95% of the time or that the estimate has a 5% probability of error. Hence, apart from inspecting a survey’s sampling declaration, if a survey claims to describe the population, then we should also check whether its findings report values that are within the acceptable confidence level and tolerable margin of error.
Kang Garcia
On phrasing of questionnaire items and response options
Surveys utilize questionnaires that contain items and response options that would generate the data necessary to address the survey’s research objectives.
We should make sure to examine how the questionnaire items and response options were phrased so that we could accurately interpret what the survey result means.
Survey questionnaires are carefully crafted so that the items and response options are properly articulated to serve the purposes of the study and to be able to clearly and efficiently communicate with its set of respondents.
Since most survey reports release the exact phrasing of questionnaire items when disseminating results, we should then make sure to examine how the questionnaire items and response options were phrased so that we could accurately interpret what the survey result means.
On the social, economic, and political landscape at the time of survey roll-out
Surveys are implemented within a social, economic, and political climate and there will be survey variables, such as sentiments and behaviors, that are unavoidably and unevenly vulnerable to the current social, economic, and political landscape. As such, we should always be cognizant of the timeframe of survey implementation and we should comprehend survey results within the temporality of the broader social, economic, and political context.
With an increased fluency on representative sampling and generalizability, attentiveness to questionnaire item and response option phrasing, and consciousness of the social, economic, and political landscape at the time of survey implementation, we can be better equipped to astutely read survey reports.