Comparison between Research Methodologies: Experiments, Surveys, and Case Studies

The Experiment

The idea of an experiment is not often used in the area of public administration and management although several types of experimental elements are sometimes used (such as simulations, gaming or evaluating policy).  Van Thiel (2014) highlights several types of experiments including the classic experiment, simulation/gaming, and field experiment.  The experiment’s reliability is high due to the level of control over standard research however, that also plays into the experiments lack of realism as a controlled experiment is hardly similar to ‘real life’.  For the classic experiment the subjects are usually people and are separated into either the experimental group or the control group.  The experimental group are subjected to a specific type of stimulus whereas the control group is not.  The classic experiment is time-based and after the experiment ends, the effect of the stimulus can be analyzed.   Normally the classic experiment is a controlled space or area and in order to reduce influences to the experiment, the researcher will limit the number of variables that may influence the experiment.  The minimum set of variables in an experiment is comprised of an independent variable (stimulus) and dependent variable (the outcome to be measured).  The field experiment is yet another type of experiment.  This type of experiment takes place outside of a controlled space or laboratory.  In the realm of Public Administration, the field experiment is normally carried out by a government or public entity rather than a researcher (Van Thiel, 2014).  Experiments can also in the form of simulations and gaming.  In this case, reality is imitated in a setting that is controlled or manipulated by the researcher (Van Thiel, 2014).  Dissimilar to the classic experiment, gaming and simulation do not split the group into a control and experiment and normally more than one variable can be introduced into the experiment.  

The Survey

The survey is arguably one of the best-known and widest used forms of research.  It can easily reach thousands of people, collect considerable amount of information and can be tailored to almost any form of research questions, polls or opinions.   A standard survey is normally large-scale, multiple variables and many units of study, known as respondents (Van Thiel, 2014).  To allow large scale data collection the researcher uses standardized forms of measurement, such as answer scales or numerical answer categories (DeVellis, 2012).  The most common type of survey is the written questionnaire complete with closed-ended questions that include a set of fixed answers (Van Thiel, 2014).  For example, a respondent may be asked how safe do they feel if a hurricane impacted their city and the answers would include, ‘very safe’, ‘somewhat safe’, ‘nether safe nor unsafe’, ‘somewhat unsafe’ or ‘very unsafe’.  The written questionnaire normally consists of five key steps: The design of the questionnaire, testing in a pilot (or beta test), respondent sample size or type, filling out questionnaire, and entering the respondent data into database to extract and analyze data.  Specific attention must be made to ensure the questionnaire is clear, has no leading statements, uses similar answer categories and the categories cover a complete range as possible.  Use of the Likert scale is often employed in surveys as it produces accurate and consistent statistical data.  The layout of a survey or questionnaire is also important.  The questionnaire always begins with an introduction or instruction that gives the point or aim of the study and who is conducting the research.  The order of the questions also have to be in a logical and easy to follow process and layout has to be inviting and clear for the respondent (Van Thiel, 2014). 

Case Study

The case study is simply a strategy in which one or more cases of the subject of study are observed in a real-life setting.  A case can literally be anything from a law or process to group, city, country or relationship.  Normally the case study takes on a holistic approach meaning qualitative data is gathered on everything that is done or has to do with the case.   Case studies can also be inductive or deductive and will normally produce results that have high validity and reliability.  Case studies are conducted in real life (field experiments) scenarios and generally are designed to solve or formulate an answer or solution to some type of issue or problem.  Contrary to the survey, case studies generally focus on a limited number of situations but the situations are more detailed, producing the goal of more depth than breadth (Timney Bailey, 1992).  Several key elements should be considered when conducting case studies.  The number of cases (single, multiple, contrasting, homogeneous), the number of measurements (time frame, period, spacing) and the research methods such as how many and which ones are all key elements in designing a successful case study.  To ensure a successful case study triangulation, or mixed methods, will increase the validity and reliability of the case study.  Combining methods such as observation,  content analysis, and interview are all combined.   Following the proper case study protocol helps to keep the researcher on track and plan day to day tasking and activities.  Case study protocol involves procedures, methods and analysis. 

Quantitative vs Qualitative

Quantitative data collection is normally used to measure a problem or issue through the generation of numerical data or data that can be collected and transformed into statistical information to illustrate an observation or issue.   Quantitative data collection methods include surveys, interviews, polls or observations.  In a quantitative study, researchers assign numerical figures to various variables for example, a question might state, “How satisfied are you with your cities emergency response following power outage?” and the answer would be assigned a certain score for every answer (e.g. 1 = Very satisfied, 2 = neither satisfied or unsatisfied, 3 = very unsatisfied).  The answers can then be used in a numerical or statistical representation that can be used in a myriad of ways.   According to Van Thiel (2014), “quantitative data consists of three phases: data collection, data ordering and data analysis”.   The analysis of data can either be theory (deductive) or data (inductive) driven (Robson, 2002).  Analyzing quantitative data can include statistical features such as mean and standard deviation, which is useful to determine how large certain scores are from the statistical mean or average.  Because of the nature of the numerical data in a quantitative study the results may be recorded in tables, figures and charts to be better visually understood. 

Qualitative studies or research are different from quantitative research in that they mainly exploratory research.  The qualitative study is used to learn and understand certain reasons, motivations or opinions relating to a problem, issue or topic.  It is used to gain an understanding of underlying reasons, opinions, and motivations. The qualitative study helps to provide understanding into the problem/issue or helps to spur ideas or hypothesizes into possible future quantitative research.  Qualitative data is non-numerical for example; research may include observations, interviews, photos, etc. According to Van Thiel (2014), qualitative research might take on different forms such as grounded theory, ethnography and thick description.  Each form of research varies to the number of cases studies, how in-depth the study becomes, and the group or issue being studied.  Examples of qualitative studies may include observation and interviews of a specific group of people, observing others while recording your own behavior (ethnographic), or studying environmental policy such as the study done by Knill and Lenschow (1998) in Germany and the UK.  As with quantitative data analysis, qualitative analysis research is recommended to follow certain guidelines to create the most accurate set of data.  Miles & Huberman (1994) define six such guidelines: make a good representation of cases, units and respondents, use of a computer program for data collection, be mindful of researcher interference, pay attention to unexpected or deviant outliers in data, don’t fixate on confirmation of hypothesis, and try to replicate results. 

Reporting Research Results

Every experiment, research or analysis ends with some type of report that is prepared for a specific audience.  Experiments and research are normally done with some type of end goal in mind whether to provide supporting data, contradictory data, unique data or merely providing some type of information via observation or interviews.  Reporting research has to be ethical and generally falls within some type of ‘rules’.  Burnham, Gilland, et al. (2008) defined five rules that apply to all phases of research to include reporting research results.  First, beneficence.  A study should contribute some kind of knowledge or solution and if there are negative outcomes, the researcher should consider all consequences that may negatively impact the respondents or units of study before publishing results.  Second, veracity. Refrain from producing misleading or biased reporting.  Stick to the facts and research and if it does take you in a different direction then follow through with the data.  Third, privacy.  Every respondent or unit of study has the right to not participate, withhold information they deem private.  Fourth, confidentiality.  The researcher must make it absolutely clear to the respondents or units of study exactly how the information and research will be used, why the study is being done, and to what end the information will provide to the sponsor or researcher.  Lastly, informed consent.  There must be full disclosure and permission from the respondents and/or units of study to carry out and use/publish the results of the research.   Regarding the final report, the format of such report can be nearly any format or delivery.  An article in a science or professional journal, part of a published book, policy advice, political or respondent report, a case study report or a press release.  Recording or writing down the report is also critical as it allows the information to be used, verified, enhanced or in some cases even disproven.  Writing the report may take several shapes depending on the targeted audience, whether scientific, academic, political or whatever. Research can take on as many shapes as the reports themselves and in the end may provide significant or critical data regarding a problem, issue, solution or insight.

References

• Burnham, P., Gilland, K., Grant, W. & Layton-Henry, Z. (2008). Research Methods in Politics. Basingstoke, UK: Palgrave Macmillan.
• DeVellis, R. (2012). Scale Development: Theory and Applications. London, UK: Sage Publishing Inc.
• Knill, C and Lenschow, A. (1998).  Coping with Europe: The Impact of British and German Administrations on the Implemnetation of EU Environmental Policy.  Journal of European Public Policy, 5(4).
• Kuhn, T. (1996). The Structure of Scientific Revolutions. Chicago. Il: University of Chicago Press.
• Miles, M. and Hubuerman, A. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage Publishing Inc.
• Robson, C. (2002). Real World Research A Resource for Social Scientists and Practitioner-Researchers. Malden, MA: Blackwell Publishing. 
• Timney Bailey, M. (1992). Do Physicists Use Case Studies? Thoughts on Public Administration Research. Public Administration Review (52)1, 47-54.
• Van Thiel, S. (2014). Research Methods in Public Administration and Public Management. New York, NY: Routledge.