Essentials of research methods in neurosurgery and allied sciences for research, appraisal and application of scientific information to patient care (Part I)

Hypothetical illustrative case scenarios

What study designs best describe the following case scenarios?

Scenario 1: Dr. JJ has just designed a new technique for operating suprasellar meningiomas. He wants to test the effectiveness of this new technique in terms of tumor resection, complications and recurrence rate. He applies this technique on 100 consecutive patients operated in his hospital from 1995-2005. He followed-up these patients for 10 years. A1- Study Design: Prospective Descriptive Cohort (If the investigator is different from Dr. JJ). A2- Study Design: Uncontrolled Intervention (experimental) Study (If Dr. JJ is the investigator).

Scenario 2: During the follow-up of the 100 cases (in scenario 1) Dr. JJ notices early tumor recurrence in one patient at one month, in a group of 5 patients at 5 months, and 7 patients at one year follow-up. A review of their pathologic report showed these patients had sub-type “Y” of aggressive meningiomas scantily reported in literature. B1- Study Design: Case report (if reporting at 1-month follow-up. Sample size = 1 case); B2- Study design: Case reports (if reporting at 5-month follow-up. Sample size = 5 cases); B3- Study design: Case series (if reporting at 1-year follow-up. Sample size=7 cases).

Scenario 3: In 2010, Dr. JJ’s resident in writing his thesis does a retrospective review of the institution’s database and assesses the recurrence rate amongst the patients operated for suprasellar meningiomas irrespective of the operative technique. Two possible study designs: C1-Retrospective descriptive cohort-If the resident retrospectively samples all patients initially operated by Dr. JJ’s technique (exposure) and evaluate them to see if they had a recurrence (outcome); C2-Case series: If the resident assembles all cases with “recurrence of meningioma” (outcome) and then “looks back” to what technique they were operated with (exposure).

Scenario 4: Suppose a concomitant comparison of the outcome of Dr. JJ’s technique is made with respect to 100 patients operated during same time period using the traditional sub-frontal approach. Possible study designs; D1-Prospective (analytic) cohort (if at the time of study, the outcome has not occurred); D2-Retrospective (analytic) Cohort (if at the time of study, exposure and outcome have occurred); D3-Case control study (for example (e.g.) if the resident assembles patients with (“cases”) and without (“controls”) tumor recurrence and then retrospectively assesses the technique (“exposure=intervention”) they were initially operated with); D4-Study design: Randomized controlled intervention (experimental) study (If Dr. JJ is the investigator and there is random allocation of intervention in 2 treatment groups).

Scenario 5: Suppose Dr. JJ does a literature review to understand the trends and summarizes the current knowledge on techniques of operating suprasellar mengiomas and their outcome(s). Possible Scenarios include: E1-Study design: traditional review (if Dr. JJ does a selective review of the literature); E2-Study design: Systematic Review (If Dr. JJ does a methodical literature review); and E3-Study design: meta-analysis (if Dr. JJ does a systematic review and additionally includes a quantitative assessment outcome). The relevance to make a distinction between the study designs stems from the fact that different study types answer different research questions; with different statistical effect sizes measured for each and provide different class of evidence and recommendation.

Methods

We reviewed more than a dozen reference books of clinical (neuro-) epidemiology15-30 and other previously related articles addressing research methods in clinical epidemiology,1,3,31-33 surgery,4 and neurosurgery.28,34 Based on existing theories and models, our expertise and experience, we synthesized and presented a holistic interpretation of research methods frequently encountered in routine neurosurgical practice.

An overview of research methods in neurosurgery.24,27,29-30,25

Perusing through literature, one will find an avalanche of inconsistent and confusing algorithms for clinical research methods. Herein, we provide a simple and logical algorithm for the different types of quantitative clinical research methods encountered in routine neurosurgical practice (Figure 1). The criteria used are aim of study, presence or absence of a comparison group, unit of analysis, exposure allocation, and the directionality of the study.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Types of common research designs in neurosurgery.

Depending on the “aim”, clinical research falls into 2 general categories viz: primary study designs involve conducting “de novo research” (case scenario 1-4), while secondary research usually makes use of existing data, summarizing them to answer specific clinical questions (case scenario 5) (Figure 1). Primary research designs can be descriptive and/or analytic, based on whether a “comparison group” exists or not. Descriptive studies do not have a comparison group and depending on the “unit of analysis”, they can be population-based, such as ecological studies or focused on individuals, such as case reports, case series, and descriptive cohorts. Analytical studies feature a comparison (control) group and include observational designs (investigators does not assign the exposures/intervention) and experimental studies (investigator assigns the exposures/intervention). According to the “temporal direction”, observational studies can be: cross sectional (if the study determines both exposures and outcomes at one time point); case-control study (if it begins from the outcome and looks back in time for an exposure) and cohort study (if it begins from an exposure; and follows the subjects to measure an outcome). Experimental studies include clinical, laboratory, or field trials. Secondary or integrative studies pool data from primary studies to draw conclusions and include quantitative decision making, traditional reviews and systematic reviews with/without meta-analyses (Figure 1). Every study begins with a research topic, which might be a question, or a controversy that the investigator seeks to answer or to clarify respectively. The sorts of questions that research addresses may be etiologic, diagnostic, therapeutic, prognostic, or economic/decision analysis.35-37 They aim to answer the “5Ws” of ‘Who’ ‘What’, ‘When’, ‘Where’ and Why?32

Different questions require different study designs.38,39 These research questions might be addressed using approaches, which might be descriptive and/or analytic, and/or integrative. Descriptive studies are useful primarily for describing the pattern of disease occurrences, and for allowing the formulation of an etiologic hypothesis. Analytic studies although can be used to generate additional research questions are mainly used to test research hypotheses. Integrative studies pool and summarize results from primary studies.

The choice of study design best suited for the investigation at a given time is influenced by particular features of: exposure (intervention) and disease (outcome), logistic consideration of time and resources, and previous studies and gaps in knowledge that remain to be filled. Other study designs common in neurosurgery include anatomical, physiological, genetic studies, case illustrations, technical notes on instruments or equipment that are innovative or useful to clinicians and researchers in the field of neuroscience, and history of persons or events related to neurosurgery. The EQUATOR Network41 described in details in part II of the review works to improve the reliability and value of medical research literature by promoting transparent and accurate reporting of research studies. An understanding of some basic notions is important before plunging into the details of the different research designs (Table 1).

Concept of exposure (intervention) and disease (outcome)

An exposure may represent any intervention to which individuals are subjected to (such as surgery), a behavior (such as smoking) or an individual attribute (such as age, gender, and race). Common exposures in neurosurgery are the different surgical techniques. Exposure may represent a new surgical technique (Dr. JJ’s techniques as in case scenario above) while non-exposure may be the gold standard for that procedure (such as subfrontal approach). Exposure can be dichotomized as present or absent, or may be presented as graded levels of exposure, such as blood pressures (the higher blood pressure the higher the risk for stroke).31 The outcome refers to the disease state, event (such as complication), behavior or condition associated with health that is under investigation. Common outcomes in neurosurgery are pain level assessment, degree of tumor resection, complications or recurrences.

From the classic contingency table (Table 1), exposure might be present or absent and outcome status is typically dichotomous (such as pain or no pain) but might be an ordinal variable (such as mild, moderate or severe pain). Worth noting is the limitation with current statistical methods used in medicine over the last century based on traditional binary “Boolean logic” (such as yes/no, present/absent, a disease exists or not, and so forth). In reality, diseases occur in a continuum or spectrum in what has been recently described in fuzzy statistics.41-43 Fuzzy logic or mathematics accounts for the fact that for example someone can have mild, moderate, or severe intracranial hypertension. Nonetheless, this is subject for another review. In the pages that follow, we will sequentially discuss: descriptive, analytic and integrative studies and their sub-types with a focus on their anatomy.

Descriptive studies

Descriptive studies describe the general characteristics of a disease in relation to person, place, and time. Indices of “person” include but are not limited to: socio-demographic factors (age, gender, race, marital status, occupation), as well as life-style variables (diet, smoking status, alcohol consumption, medications, exposure to toxins). Characteristic of “place” refers to geographic distribution of disease, and “time” variable relates to the time of year, day, progression over time, the speed of development of disease and temporal changes in disease with time30 for example seasonal variation of the occurrence of subarachnoid hemorrhage. Descriptive studies provide valuable information to enable medics to allocate resources efficiently, and to plan effective preventive or education program (health-care planning). Also they provide important clues regarding possible determinants of a disease and trend analysis.32 Due to limitations inherent in their design, they are primarily useful for the formulation of hypotheses that can be tested subsequently by analytic designs. Descriptive studies include: case reports, case series, and ecological studies,30 Additionally are descriptive cohorts and uncontrolled experimental (intervention) studies6 rarely mentioned in neurosurgical literature.

Case report and case series.30,25

Most basic descriptive study of individuals and consists of detailed report by one or more medics of the profile of a single to 5 patients or events are shown in Figure 2.7 It is the least publishable unit or basic structural and functional unit of research.32 Often, an observant clinician reports an unusual disease or association, which prompts further investigations with more rigorous study designs by the same or other investigators.32 Clinical case reports, which are often overlooked by mainstream scientific journals, represent an important, maybe even an exclusive, way of communicating new and unusual clinical findings in medicine to the scientific community. In addition, they are often the main source of new knowledge pertaining to rare clinical features in medicine, especially in surgery.2 As illustrated in case scenario 2 (B1 and B2) and in Figure 2 the early occurrence of an aggressive of sub-type Y meningioma is an example.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Case report(s) and case series.6

Case report

Basic structural and functional unit of research consisting of detailed report of the profile of a single to a maximum of 5 subjects (patients or events).

Case series

A collection of greater than 5 individual case reports.7 It samples participants with both a specific outcome (meningioma recurrence) and a specific intervention or exposure (Dr. JJ’s technique) and the cases are selected on the basis of a striking association between outcome and exposure epidemiologic case series or selection is based only on a specific outcome and data are collected on previous exposures surgical case series.

Case series are commonly considered as several case reports of similar observations or procedures that can be aggregated, or grouped together in one report or publication.32 In its simplest form, a case series may be considered as a collection of individual case reports or study of multiple occurrences of unusual cases that have similar characteristics, such as appearance of several similar cases in a short period that may herald an epidemic.32 It does not follow-up the patients for a period (unlike cohort studies) and uses no control or comparison group (unlike the case control design), thus cannot establish cause-effect relationship.44

Sensu stricto, a case series may be a study that samples patients with both a specific outcome and a specific intervention or exposure (epidemiologic case series), or one that samples patients with a specific outcome and includes patients regardless of whether they have specific exposures (surgical case series)31 (Figures 2 and 3). The essence of this study design is that they look backwards, from the outcome (disease) to putative exposure (intervention) (Figure 3). Case series have outcome-based sampling in common with case controls however, case control studies differ from case series in their inclusion of a comparison group without the disease.6

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Anatomy of case series in neurosurgery.6

Case series have outcome-based sampling. They sample patients with a specific outcome and look backwards, from the outcome (disease) to putative exposure (intervention). “Selection is based only on a specific outcome and data are collected on previous exposures [surgical case series].

Case reports/series are simple to perform (report can be rapidly written and published) and are often the first form of reporting for new diseases or rare complications. Although case reports are sometimes often confused with case series, Esene et al6,7 proved that 5 should be the maximum number of subjects in case reports and cases presented individually (without statistical pooling of data), whereas studies with more than 5 subjects should be reported as case series provided sampling is from outcome to exposure. This distinction is important when conducting meta-analysis as a meta-analysis of case reports ought to produce much more detailed information than that of case series.6 Case series are however, limited in discerning cause-effect relationship, or comparison of intervention.2 A convenient feature of case series is that they can constitute the case group for a case-control study, which can then explore hunches about causes of disease.32

Descriptive cohort studies.6,28

A control group is not an essential component of a cohort study and some studies do not include one. Such are typical in neurosurgical practice and are termed “descriptive cohort studies”.6 The investigator “observes” patients allocated to an exposure (Dr. JJ’s technique) then follows-up to assess the outcome (tumor recurrence). There is no experimental protocol for allocation of patients to treatment (exposure). Just like the under-mentioned “classical cohort study” (the analytic cohort with a comparative group), they can be retrospective (case scenario 3-C1) or prospective (case scenario 1- A1). They describe the outcome over time for a specific group of individuals without a comparison group (Figure 4). Natural history studies fit into this group. They are limited by the fact that causation cannot be determined since they lack a comparison group, and are prone to sample-related biases. Their importance stems in the fact that they are often confused with the case series design with up to 50% of the studies published as case series in neurosurgical literature being actually retrospective descriptive cohorts.6 Statistically, it is possible to calculate the risk (incidence of tumor recurrence), a measure not possible with case series where only “odds” can be calculated (Table 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Anatomy of the descriptive cohort design

The investigator observes patients allocated to an exposure (Dr. JJ’s technique) then follows-up to assess the outcome (tumor recurrence). At the time of study, the outcome has not occurred hence “prospective”. The absence of a control group distinguishes them from the classical analytic cohort studies.

Population (ecological) studies

Include correlation/aggregate studies and time series. Ecological studies are studies of risk-modifying factors on health or other outcomes based on populations defined either geographically or temporally. Both risk-modifying factors and outcomes are averaged for the populations in each geographical or temporal unit, and then compared using standard statistical methods. Population studies can use data from the entire population to compare disease frequency between different groups during the same period of time (correlation or aggregate studies), or in the same population at different points in time (time series). They are also called ecological studies because people are classified by the general level of exposure in their environment, which may or may not correspond to exposure of specific individuals.30 They are flawed by a potential bias called ecological fallacy, in which affected individuals in a generally exposed group may not themselves have been the ones exposed.25 Since ecological studies refer to the “whole population” rather than “individuals”, it is not possible to link an exposure to occurrence of the disease in the same person.30 For example, links between diet and Alzheimer’s disease have been studied using both geographical and temporal ecological studies.45,46 Also death rates from cerebrovascular disease can be studied (correlated) with per capita sales of cigarettes using this methodology.

Analytic studies.30,25

Analytic studies are a more commonly encountered category of studies, involving comparisons between 2 or more groups (case scenario 4). They are based on a research question that is etiologic, diagnostic, prognostic, or therapeutic. Based on research architecture, the studies can be observational (investigator observes natural course of events), or experimental (investigator allocates exposure/intervention). The choice of study design in an analytic study depends on: nature of disease (cohort studies for common diseases and case control studies for rare diseases), type of exposure (cohort studies for rare exposures) and availability of resources.30 They include cross sectional, case control, and cohort studies (Table 2).

Cross-sectional (transverse or prevalence) study.30,35

Data on exposure and outcome assessed at same time (a snapshot in time). There is no longitudinal component and they are fast and inexpensive (since both exposure and outcome are ascertained at the same time) with no loss to follow-up. Examples are periodic surveys of the health status of the population (health interview survey) and decennial census, which provide a snapshot of the population at a particular time.32

There is limited inference between exposure and outcome and the temporal relationship between exposure and disease cannot be clearly determined. They should, however be cautiously conducted and interpreted for risk of potential biases (response/ participation bias [sicker patients are more or less likely to participate]). Some authors even consider them as descriptive!30,32

Case control studies.30,25

They compare the risk factors in a population with the disease (cases) to the risk factors in a sample without the disease (controls), and calculating an odds ratio (an approximation of the relative risk) (Figure 5). Odds ratio is the odds that an outcome will occur given a particular exposure/intervention, compared to the odds of the outcome occurring in the absence of that exposure (Table 1). In case scenario 4: D3-the resident assembles patients with tumor recurrence (“cases”) and without tumor recurrence (“controls”) and then retrospectively assesses the technique (“exposure=intervention”) they were initially operated with.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

Anatomy of a case-control study.

Advantages

This is the best study for relatively rare diseases because it guarantees a sufficient number of cases with the disease, permits evaluation of multiple possible risk factors for a disease, relatively small sample size, relatively inexpensive, and of short duration.

Disadvantages

It can only measure one disease variable, does not provide information regarding prevalence or incidence, increased bias due to the retrospective design, and the need to obtain information on exposures by recreating events that happened in the past, selecting proper controls is challenging, and can lead to selection bias.

Cohort studies (CS).30,25

Cohort studies are observational studies, in which subjects are classified on the basis of presence or absence of exposure to a particular factor (a surgical intervention) and then followed up for an adequate specified period of time to determine the development of the outcome or disease (tumor recurrence) in each exposure group (Figure 6). Depending whether the outcome of interest has occurred at the time, the investigator initiates the study; cohort studies can be classified as prospective or retrospective. The CS is also known as longitudinal studies (emphasizing patients are followed over time), or incidence studies, which call attention to the basic measure of new disease events over time.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6

Anatomy of a cohort study (prospective)

Based on this temporal relationship between the investigator and the outcome of interest, studies in which the exposure and outcome have already occurred are termed retrospective (non-concurrent CS or retrospective or historic or database study) cohort studies, while studies in which the outcome has yet to occur are called prospective (concurrent) cohort studies.

Pros and cons

They indicate the temporal sequence between exposure and outcome, allow for the calculation of the incidence of the disease in each groups (absolute risk (incidence), relative risk (risk ratio or rate ratio), risk difference, attributable proportion (attributable risk %), which are particularly useful for evaluating the effects of rare or unusual exposures. The examination of multiple outcomes and exposure can be elicited without the bias that might occur if the outcome were already known (for prospective CS). However, they generally require large sample, are not useful for rare outcomes, and can never be assumed to be free of confounding bias.

Experimental or interventional studies.30,25

An experimental study is an investigation where the researcher assigns or allocates the exposure (treatment) to both study (exposed) and control (non-exposed) groups. (case scenario 4-D4). They can be randomized or not and can have a control group (controlled trial), or none (uncontrolled intervention). The structure of the experimental study is similar to that of the prospective cohort design (Figure 6) except for fact that the investigator does not allocate exposure in the latter. The exposed and unexposed are then followed forward in time to ascertain the frequency of outcomes.3 A good clinical trial is one in which the random allocation (by chance) of intervention is blinded and controlled. Randomization checks for both known and unknown confounders that may affect the outcome of trial, thus reducing the likelihood of bias in determination of outcomes. Well conducted RCT should be adequately large, with blinding of subjects and/or investigator and/therapist (single versus double versus triple blinding), with limited or no loss to follow-up, and with carefully standardized methods of measurement and analyses.25 The hallmark of RCTs is assignment of participants to exposures purely by the play of chance.3 The RCT is evidence-based medicine’s most powerful strategy to establish evidence regarding the treatment of patients. Clinical trials are prospective; first stating a hypothesis and then following patients forward in time. Although RCT produce strong evidence, they are rarely feasible when studying risk factors (cause) of diseases because it is not ethical to randomize intervention thought to be harmful (instead observational studies can be used!). Major drawbacks with RCT are external validity of results (generalizability) and cost.3 The flaws and limitations of RCTs are discussed in details in “part II” of our review.

Miscellaneous study types

Anatomical studies involve studying different parts of the human body through dissection. Cadaver studies give trainees hands-on experience with cutting into and handling the structures in a human body - something a computer program simply cannot replicate. They enable surgeons develop new surgical techniques and approaches. Cadaver studies are an important part of a medical education and research. They can be descriptive and/or analytic.

Physiologic studies

Physiologic studies have become an integral part of neurosurgery. They are often useful in identifying nerve or brain targets during functional neurosurgical procedures. They can also be descriptive and/or analytic.

Genes and genetic studies.15

Genetics is the fundamental basis of any organism so its understanding provides a powerful means to discover hereditary elements in disease etiology. In recent years, genetic studies have shifted from disorders caused by a single gene (Huntington’s disease) to common multi-factorial disorders (Sub-arachnoid hemorrhage) that results from the interactions between inherited gene variants and environmental factors. The integration of epidemiologic methods and approaches with those of basic genetics help to identify the role of genetic factors in disease occurrence in families and populations.

The first step in clinical or epidemiologic genetic studies is to determine whether a phenotype of interest is controlled by a genetic component. There are 5 key scientific questions that are addressed in sequence in genetic epidemiologic studies: 1. Is there familial clustering (familial aggregation)?; 2. Is there evidence of genetic effect?; 3. Is there evidence for a particular genetic model?; 4. Where is the disease gene (disease gene location)? 5. How does this gene contribute (gene contribution) to disease in the general population?; The first 3 questions do not require DNA data and are referred as photometric studies, but the latter 2 depend on DNA and referred as genometric studies.

Research methodology for studies of diagnostic tests.15

Although much of clinical research is aimed at assessing causality, it can also address the value of new medical tests, which will ultimately be used for screening for risk factors, to diagnose a disease, or to assess prognosis. Although traditional clinical research designs can be used to assess some of these questions, most of the studies assessing the value of diagnostic testing are more akin to observational designs, but with the twist that these designs are not aimed to assess causality, but are rather aimed at determining whether a diagnostic test will be useful in clinical practice.

Integrative studies.25

These are study types that do not fit specifically in either category of observation studies or experiments and are thus called integrative studies. They include quantitative decision-making and social sciences, systematic reviews and meta-analyses (Figure 1).

Quantitative decision-making includes cost-effectiveness analyses, which describe the financial costs required to achieve good outcomes, such as prevention of morbidity and mortality and decision analyses, which set the rational basis for clinical decisions and the consequences or choices. Social sciences studies describe how the social environment affects health-related behaviors and the use of health services. Examples are health service research studies. They study how non-biological factors (such as clinical workforce and facilities, how health is organized and paid for, and how clinicians’ beliefs and patients’ cooperation) affect health patients’ health.

Traditional reviews, systematic reviews, and meta-analysis.30,33,25,47

Traditional or narrative reviews: selective review of the literature that broadly covers and summarizes primary studies on a specific topic, from which conclusions may be drawn into a holistic interpretation contributed by the reviewers’ own experience, existing theories and models. They do not follow strict systematic methods to locate and synthesize articles, such as systematic review designs or meta-analyses. They are prone to selection or reference bias, and are graded lower in the hierarchy of evidence. They provide a qualitative but not a quantitative assessment of published results (case scenario 5-E1).

Systematic reviews represent the most formal, rigorous and extensive review of the evidence on a specific research question and therefore reside at the top of the evidence hierarchy (case scenario 5-E2). Meta-analysis is often performed as part of a systematic review, but the effort to dig-up the evidence goes beyond mathematical pooling or summation of multiple articles. Meta-analysis uses published information from other studies and statistically combines the results so as to permit an overall conclusion (case scenario 5-E3). A meta-analysis is similar to review articles, but additionally includes a quantitative assessment and summary of the findings. It is possible to do a meta-analysis of observational studies or experiments; however, a meta-analysis should report the findings for these 2 types of study designs separately. This method is especially appropriate when the studies that have been reported have small numbers of subjects, or come to different conclusions. The meta-analysis typically combines studies into a Forest plot (or blobbogram). Well-performed systematic reviews provide an excellent and reliable review of the evidence and are constructed in a way that the busy practitioner can efficiently evaluate the quality of the evidence with numbers that are most applicable to treating patients.

In conclusion, the research question might be addressed using approaches, which might be descriptive and/or analytic and/or integrative. The sort of questions that research addresses may be etiologic, diagnostic, prognostic, about harm, effectiveness, or qualitative. Different questions require different study designs. The choice of study design best suited for the investigation at a given time is influenced by particular features of: exposure (intervention) and disease (outcome), logistic consideration of time and resources, previous studies and gaps in knowledge that remain to be filled. Ecological studies assess exposure at the community level while case reports and case series do at the level of the individual. Case-control studies “look” from disease (outcome) to exposure (intervention) and cohort studies and vice versa. Randomized controlled trials although not suitable for most situations in neurosurgery are the gold standard to study the effectiveness of intervention under investigation. Integrative studies usually the combination of many primary studies includes quantitative decision making and social sciences studies, narrative, systematic reviews and meta-analyses. Understanding research methods and the critical analysis of published reports of clinical investigation is a fundamental skill of the physician, to enable the incorporation of new clinical knowledge to practice.