A Statistical Method for Synthesizing Meta-Analyses

Abstract

Multiple meta-analyses may use similar search criteria and focus on the same topic of interest, but they may yield different or sometimes discordant results. The lack of statistical methods for synthesizing these findings makes it challenging to properly interpret the results from multiple meta-analyses, especially when their results are conflicting. In this paper, we first introduce a method to synthesize the meta-analytic results when multiple meta-analyses use the same type of summary effect estimates. When meta-analyses use different types of effect sizes, the meta-analysis results cannot be directly combined. We propose a two-step frequentist procedure to first convert the effect size estimates to the same metric and then summarize them with a weighted mean estimate. Our proposed method offers several advantages over existing methods by Hemming et al. (2012). First, different types of summary effect sizes are considered. Second, our method provides the same overall effect size as conducting a meta-analysis on all individual studies from multiple meta-analyses. We illustrate the application of the proposed methods in two examples and discuss their implications for the field of meta-analysis.

1. Introduction

The last two decades have seen an exponential growth in the popularity of meta-analyses across scientific disciplines including medical research [1] and diagnostic medicine [2]. The purpose of meta-analysis is to assess the consistency and robustness of findings across populations, settings, and contextual factors in order to help ensure that a practice is likely to produce similar results when it is implemented. A single study cannot determine, with certainty, that an intervention works or does not work. Instead, studies that are combined together, across different settings, and conducted over time can establish a pattern of consistent findings that may be useful to justify new or refined practice. Several studies combined can establish both significance and repeatability of results [3]. Common statistical methods to combine studies in meta-analyses are based on a fixed-effects model which assumes a homogeneous treatment effect among studies or a more general random-effects model which allows heterogeneity among studies [4].

Multiple meta-analyses are sometimes conducted to investigate the effect of the same topic or intervention. The synthesis of these meta-analyses can be used to highlight what is known on a particular topic or intervention or what can contribute to more complete understanding of the extant empirical evidence [5]. However, the summarizing of meta-analyses can be very challenging because existing meta-analyses may be conflicting, may be reported differently or incompletely across studies, or may be more or less valid based on the quality of the research synthesis methods that were employed in conducting the reviews. This raises challenges for interpreting and drawing conclusions about what the results of these studies mean and how they should be used to inform future research, theory development, and practice. Systematic reviews are further complicated when a diverse array of health-relevant outcomes are assessed. This is essentially the same issue researchers and practitioners face when attempting to summarize primary studies, the same issue that makes meta-analysis appealing.

Several nonstatistical approaches to summarizing meta-analyses of the same topic or intervention are currently available. The Cochrane Collaboration has developed a set of recommended procedures for conducting overviews of reviews when multiple meta-analyses exist regarding different treatments for the same clinical condition [6–8]. Nonstatistical approaches to summarizing multiple reviews, which may be used for meta-analyses, include vote counting and using decision algorithms to identify the review(s) that are most salient [7]. However, narrative reviews can be far too subjective to reflect the knowledge that has been gained through research [9]. An ideal statistical approach to summarizing meta-analyses is to conduct a new meta-analysis of all primary studies included in multiple meta-analyses. That is, individual studies in all the related meta-analyses are identified, and their effect sizes are combined to calculate a summary statistic using meta-analytic techniques such as the random-effects model in DerSimonian and Laird [4]. However, while this approach is ideal and appealing when there are only two or three meta-analyses to combine, doing so is not as efficient as directly summarizing effect sizes reported in existing meta-analyses on similar topics. Recently, Hemming et al. [10] provided a Bayesian method to summarize multiple reviews. Their method assumes that estimated summary effect sizes follow a random-effects model based on exchangeability assumption. Differing types of effect sizes are not considered in their method. In addition, the authors did not study the fixed-effects models which may lead to nice properties for the combined overall effect size.

In this paper we start with describing methods to meta-analyze effect size estimates from individual studies extracted from several existing meta-analyses. However, the methods require a substantial amount of time and resources. Thus, we also introduce a more efficient statistical method to directly summarize information from existing meta-analyses without going back to individual studies. The paper is organized as follows. In Section 2 we introduce the notations and our method for synthesizing meta-analyses with the same type of summary statistics. We also describe two-step methods for combining different types of summary statistics commonly used in meta-analyses. We start with the fixed-effects model and show that summarizing the reviews yields the same overall effect size as conducting a meta-analysis on all possible individual studies. We then describe the method for the random-effects model. Section 3 applies the proposed method to two examples. Section 4 discusses some potential applications of this method.

2. Methods

The use of quantitative research synthesis techniques such as meta-analysis overcomes many of the limitations of traditional narrative literature reviews. Meta-analysis provides an objective and quantitative approach to research synthesis by taking into account factors such as sample size, magnitude, and direction of relationships and the methodological quality of the various studies analyzed [11–13]. Meta-analyses are not limited by a reliance on traditional indicators of statistical significance but instead rely upon effect sizes to give a picture of the size and scope of the impact of an intervention [14]. However, the more meta-analyses that are conducted on a given topic, the more difficult it can be to analyze and determine the cumulative findings from a body of literature.

Calculating the summary effect sizes across existing meta-analyses is a potentially valuable way to synthesize the knowledge base on a given topic. This process should be guided by the same rule that guides traditional meta-analysis which requires that studies being combined should be assessing a common clinical research question. Suppose several meta-analyses consider a common clinical question, and use similar search criteria to include and exclude individual studies. Summarizing these meta-analyses can provide better understanding of the clinical question than any single meta-analysis.

When multiple meta-analyses address the same topic, we propose that researchers should explore methods to synthesize the summary effect sizes from the existing meta-analyses. This proposed approach only requires summary effect sizes and their variances which are reported in the existing meta-analyses. The calculated overall effect size for the combined meta-analyses is essentially a weighted average of the summary effect sizes with the weights being the inverse of the variances; this approach is analogous to the techniques used to conduct a meta-analysis of primary studies.

Popular effect size estimates used in meta-analyses include standardized mean differences, odds ratios, and correlation coefficients [14]. In this section, we first consider the simplest case of combining multiple meta-analyses when all analyses use the same type of effect size estimates in a fixed-effects model. We then extend the method to the more commonly used random-effects model. Finally, we present a method for combining meta-analyses with different types of summary statistics in the fixed-effects or random-effects model.

2.1. Combining Effect Sizes from the Same Measure

2.1.1. Synthesizing All Individual Studies

Suppose 𝐽 meta-analyses using similar search criteria on a common topic are identified and need to be combined to obtain an overall summary statistic. We use 𝜃_𝑖𝑗 and 𝜃_𝑖𝑗 to denote a true effect parameter and its estimate, respectively, for study 𝑖, 𝑖=1,…,𝑘_𝑗, in meta-analysis 𝑗. The estimate 𝜃_𝑖𝑗 is commonly assumed to follow a normal distribution as 𝜃_𝑖𝑗∼𝑁(𝜃_𝑖𝑗,𝜎²_𝑖𝑗) where 𝜎²_𝑖𝑗 is the variance of 𝜃_𝑖𝑗. The variance estimator, 𝜎²_𝑖𝑗, obtained from the study tends to be smaller for a larger sample size. If all 𝜃_𝑖𝑗’s and 𝜎²_𝑖𝑗’s are available, researchers can conduct a meta-analysis using existing statistical methods based on a random-effects model (REM) or a fixed-effects model (FEM). As a popular model in meta-analysis, the REM assumes heterogeneity among 𝜃_𝑖𝑗 and is given as follows: 𝜃_𝑖𝑗=𝜃_𝑗+𝜖_𝑖𝑗,𝜃_𝑗=𝜃+𝑢_𝑗, (1) where 𝜖_𝑖𝑗∼𝑁(0,𝜎²_𝑖𝑗) and 𝑢_𝑗∼𝑁(0,𝜏²). Here 𝜎²_𝑖𝑗 is estimated using the sample variance 𝜎²_𝑖𝑗 from study 𝑖 in meta-analysis 𝑗. Conventionally, the true variance 𝜎²_𝑖𝑗 is assumed to be known and equal to 𝜎²_𝑖𝑗, and to simplify the notation we will use 𝜎²_𝑖𝑗 instead of 𝜎²_𝑖𝑗 throughout the paper. The random effect, 𝑢_𝑗, represents heterogeneous 𝜃_𝑗 among all studies, and its variance can be estimated using a moment estimator 𝜏² by DerSimonian and Laird [4]: 𝜏²=𝑄−∑^𝐽_𝑗=1𝑘_𝑗−1∑_𝑖,𝑗𝑤_𝑖𝑗−∑_𝑖,𝑗𝑤²_𝑖𝑗/∑_𝑖,𝑗𝑤_𝑖𝑗, (2) where 𝑄=_𝑖,𝑗𝑤_𝑖𝑗𝜃²_𝑖𝑗−∑_𝑖,𝑗𝑤_𝑖𝑗𝜃_𝑖𝑗²∑_𝑖,𝑗𝑤_𝑖𝑗, (3) and the weight 𝑤_𝑖𝑗 is given by 1/𝜎²_𝑖𝑗 indicating a larger weight for a larger sample size. The moment estimate, 𝜏_𝑗², should always be nonnegative. If a negative value of 𝜏_𝑗 is obtained, 𝜏 is set to 0 according to DerSimonian and Laird [4]. Using 𝑤^∗_𝑖𝑗=1/(𝜎²_𝑖𝑗+𝜏²) as a weight for study 𝑖 in meta-analysis 𝑗, an estimate for the summary effect 𝜃 is given by a weighted average of effect estimates: ∑𝜃=^𝐽𝑗=1∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗𝜃_𝑖𝑗∑^𝐽_𝑗=1∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗. (4) In the FEM, the random-effect component in (1) is 0 by assuming true effect parameters are homogenous across studies, and the REM in (1) becomes 𝜃_𝑖𝑗=𝜃+𝜖_𝑖𝑗. (5) The estimate for 𝜃 can be obtained using (4) by setting 𝜏=0.

2.1.2. Synthesizing Meta-Analyses

Obtaining 𝜃 with results from all individual studies as in (4) is an ideal way to synthesize multiple meta-analyses. In the traditional meta-analysis, hundreds of studies are often available on a similar topic, and when authors of some individual studies are not contactable, criteria can be set accordingly to exclude these studies. However, extracting individual study data requires tremendous time and resources. Alternatively, we propose to directly summarize meta-analyses. We start with showing the equivalence of synthesizing meta-analyses and synthesizing all individual studies under the simple FEM which assumes the same effect parameter, 𝜃, across studies. The effect parameter 𝜃_𝑗 in meta-analysis 𝑗 is estimated using 𝜃_𝑗=∑^𝑘𝑗_𝑖=1𝑤_𝑖𝑗𝜃_𝑖𝑗∑^𝑘_𝑖=1𝑤_𝑖𝑗, (6) where 𝑤_𝑖𝑗=1/𝜎²_𝑖𝑗. The variance estimate of 𝜃_𝑗 is given by var(𝜃_𝑗)=1/∑^𝑘_𝑖=1𝑤_𝑖𝑗. Using 𝑤_𝑗=1/var(𝜃_𝑗) as the weight for meta-analysis 𝑗, we can obtain a weighted average of 𝐽 meta-analysis results as follows: 𝜃_𝑓=∑^𝐽_𝑗=1𝑤_𝑗𝜃_𝑗∑^𝐽_𝑗=1𝑤_𝑗. (7) After some math, we see that the expression above is the same as (4) with 𝜏=0. The result implies that conducting a meta-analysis by pooling summary effect sizes of the meta-analyses of interest is equivalent to a meta-analysis of combining all individual studies from these meta-analyses. The variance of the weighted average is given by var(𝜃_𝑓)=1/(∑^𝐽_𝑗=1𝑤_𝑗).

The assumption of the same effect parameter across the studies may not be realistic considering studies’ heterogeneous characteristics. This assumption is relaxed in the REM as shown in (1). Using the data from meta-analysis 𝑗, the parameter 𝜏 accounting for the random component is estimated by the following moment estimator: 𝜏²_𝑗𝑄=max0,^𝑗−𝑘_𝑗−1∑_𝑖𝑤_𝑖𝑗−∑_𝑖𝑤²_𝑖𝑗/∑_𝑖𝑤_𝑖𝑗, (8) where 𝑄_𝑗=_𝑖𝑤_𝑖𝑗𝜃²_𝑖𝑗−∑_𝑖𝑤_𝑖𝑗𝜃²_𝑖𝑗∑_𝑖𝑤_𝑖𝑗. (9) The moment estimator 𝜏²_𝑗 has the same expression as the one given in (8) except that it is now obtained using only data from meta-analysis 𝑗. Meta-analysis 𝑗 then uses 𝑤^∗_𝑖𝑗=1/(𝜎²_𝑖𝑗+𝜏²_𝑗) as a weight for study 𝑖 to obtain an estimate for the summary effect 𝜃_𝑗 by 𝜃^∗_𝑗=∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗𝜃_𝑖𝑗∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗. (10) To synthesize 𝐽 meta-analyses, a possible way is to use 𝑤^∗_𝑗=1/∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗 as the weight for meta-analysis 𝑗 and calculate a weighted average of 𝜃_𝑗’s as follows: 𝜃_𝑟=∑^𝐽_𝑗=1𝑤^∗_𝑗𝜃^∗_𝑗∑^𝐽_𝑗=1𝑤^∗_𝑗. (11) Comparison between the estimate above and the one in (4) shows some difference. We can write (4) as the weighted average of meta-analyses summary statistics in the following expression: ∑𝜃=^𝐽𝑗=1𝑤^∗_𝑗𝜃^∗_𝑗∑^𝐽_𝑗=1𝑤^∗_𝑗, (12) where 𝜃^∗_𝑗=∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗𝜃_𝑖𝑗∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗, (13) and 𝑤^∗_𝑗=1/∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗. A close look at 𝜃^∗_𝑗 and 𝜃^∗_𝑗 reveals a slight difference. This is due to different weight estimates, 𝑤^∗_𝑖𝑗 and 𝑤^∗_𝑖𝑗, where the former uses studies in meta-analysis 𝑗 to estimate 𝜏 and the latter pools studies from all meta-analyses to estimate 𝜏. In practice, a well-planned meta-analysis includes a large number of studies, and 𝜏²_𝑗’s and 𝜏² should all be close to the true parameter 𝜏. Subsequently, the difference between 𝜃_𝑟 and 𝜃 can be possibly ignored. Thus, although the weights used in (11) may not be as accurate as the ones obtained by pooling all individual studies, 𝜃_𝑟 is still a statistically sound estimate for 𝜃.

2.2. Combining Effect Sizes from Different Types of Statistics

The summary statistics in meta-analyses may be of different types. When individual studies in a meta-analysis have continuous outcomes, the statistic may appear in the form of the sample mean difference. Sometimes, other studies address the same topic, but the outcome is dichotomized, and the odds ratio is commonly used to summarize the difference between two groups. The Pearson correlation coefficient 𝑟 is frequently used to evaluate the correlation between two continuous variables. When one variable is a continuous outcome and the other variable is dichotomized to indicate the group status, 𝑟 can be used to compare outcomes between two groups. When meta-analyses addressing the same topic use different types of statistics, one should convert these statistics to the same type of summary statistics before synthesizing them.

2.2.1. Synthesizing All Individual Studies

We now discuss how to combine meta-analyses with these types of summary statistics using a two-step procedure. The first step is to convert various types of statistics to the same type. In this paper, we discuss how to convert the log-transformed odds ratio (OR) and the correlation coefficient to the same scale of the sample mean difference. The log-transformed OR can be converted to the sample mean difference using their linear relationship as will be discussed. However, a Taylor expansion is needed to convert the correlation coefficient to the sample mean difference via linear relationship. We focus on the linear relations because the weighted average in a meta-analysis is a linear combination of effect estimates, and preserving the linear nature in the conversion will facilitate the future calculation for synthesizing the meta-analyses.

Suppose meta-analysis 𝑗, 𝑗=1,…,𝐽, is conducted using the standardized sample mean difference 𝜃_𝑗,𝑚 with its variance of 𝜎²_𝑗,𝑚 and meta-analysis 𝑗^′, 𝑗^′=1,…,𝐽^′, has the estimated log-transformed odds ratio 𝜃_𝑗′,OR as the summary effect and its sample variance of 𝜎²_𝑗′,OR.

Chinn gives the detailed discussion on how to convert between the standardized mean difference and the estimated odds ratios from individual studies [15]. Since the log-transformed odds ratio, 𝜃_{𝑖𝑗′,OR}, in study 𝑖 of meta-analysis 𝑗 follows approximately a logistic distribution which differs from a standard normal distribution mainly in the tail area, dividing 𝜃_{𝑖𝑗′,OR} by 𝜋/√3, or 1.81, converts 𝜃_{𝑖𝑗′,OR} to an approximate standardized sample mean difference.

In meta-analysis 𝑗^′′, 𝑗^′′=1,…,𝐽^′′, with correlation coefficients, the coefficient 𝜃_{𝑖𝑗′′,𝑟} from study 𝑖 is converted to Fisher's 𝑧-score 𝜃_{𝑖𝑗′′,𝑧} before the further calculation [14]. The conversion is given by 𝜃_{𝑖𝑗′′,𝑧}=12𝜃ln1+_{𝑖𝑗′′,𝑟}1−𝜃_{𝑖𝑗′′,𝑟}. (14) Since a correlation coefficient 𝜃_𝑟 can be converted to a standardized mean difference 𝜃_𝑚 using 𝜃_𝑚=2𝜃_𝑟/1−𝜃²_𝑟, we can convert Fisher’s 𝑧-score 𝜃_𝑧 to 𝜃_𝑚 using 𝜃_𝑚=𝑒𝜃_𝑧−𝑒⁻𝜃_𝑧. The relationship above between 𝜃_𝑧 and 𝜃_𝑚 is nonlinear. The second order Taylor expansion of 𝜃_𝑚=1+𝜃_𝑧+𝜃²_𝑧2−1−𝜃_𝑧+𝜃²_𝑧2=2𝜃_𝑧 (15) gives the approximate linear relationship between 𝜃_𝑧 and 𝜃_𝑚.

After the effect estimates are converted to the same scale of 𝜃_𝑗,𝑚, they can be used to estimate the overall effect parameter 𝜃 in the following REM model: 𝜃_{𝑖𝑗,𝑚}=𝜃_𝑖𝑗+𝜖_𝑖𝑗,𝜃_{𝑖𝑗′,OR}1.81=𝜃_𝑖𝑗′+𝜖_𝑖𝑗′,2𝜃_{𝑖𝑗′′,𝑧}=𝜃_{𝑖𝑗′′}+𝜖_{𝑖𝑗′′},𝜃_𝑖𝑗=𝜃+𝑢_𝑖𝑗,𝜃_𝑖𝑗′=𝜃+𝑢_𝑖𝑗′,𝜃_{𝑖𝑗′′}=𝜃+𝑢_{𝑖𝑗′′}, (16) where 𝜖_𝑖𝑗∼𝑁(0,𝜎²_𝑖𝑗), 𝜖_𝑖𝑗′∼𝑁(0,𝜎²_𝑖𝑗′), 𝜖_{𝑖𝑗′′}∼𝑁(0,𝜎²_{𝑖𝑗′′}), and 𝑢_𝑖𝑗,𝑢_𝑖𝑗′,𝑢_{𝑖𝑗′′}∼𝑁(0,𝜏²).

In the second step, 𝜃_{𝑖𝑗,𝑚}, 𝜃_𝑖𝑗,OR, and 𝜃_{𝑖𝑗,𝑧} are combined to obtain the overall summary effect estimate 𝜃. Let 𝑤_𝑖𝑗′=1/𝜎²_𝑖𝑗, 𝑤_𝑖𝑗′/𝜎²_𝑖𝑗′, and 𝑤_{𝑖𝑗′′}=1/𝜎²_{𝑖𝑗′′}. Again, the parameter 𝜏² can be estimated by a moment estimator 𝜏² similar to (8) using all effect estimates and their variances: 𝜏²∑=max0,𝑄−^𝐽_𝑗=1𝑘_𝑗+∑^𝐽_𝑗′=1𝑘_𝑗′+∑^𝐽_𝑗′′=1𝑘_𝑗′′−1𝑤−𝑤²,/𝑤 (17) where 𝑤=_𝑖,𝑗𝑤_𝑖𝑗+_𝑖,𝑗′𝑤_𝑖𝑗′+_{𝑖,𝑗′′}𝑤_{𝑖𝑗′′},𝑤²=_𝑖,𝑗𝑤²_𝑖𝑗+_𝑖,𝑗′𝑤²_𝑖𝑗′+_{𝑖,𝑗′′}𝑤²_{𝑖𝑗′′},𝑄=_𝑖,𝑗𝑤_𝑖𝑗𝜃²_{𝑖𝑗,𝑚}+_𝑖,𝑗′𝑤_𝑖𝑗′𝜃_{𝑖𝑗′,OR}1.81²+_{𝑖,𝑗′′}𝑤_{𝑖𝑗′′}2𝜃_{𝑖𝑗′′,𝑟}²−1𝑤_𝑖,𝑗𝑤_𝑖𝑗𝜃_{𝑖𝑗,𝑚}+_𝑖,𝑗′𝑤_𝑖𝑗′𝜃_{𝑖𝑗′,OR}1.81+_{𝑖,𝑗′′}2𝑤_{𝑖𝑗′′}𝜃_{𝑖𝑗′′,𝑟}². (18) The resulting estimate for the summary effect 𝜃 using the REM is given by ∑𝜃=^𝑖,𝑗𝑤^∗_𝑖𝑗𝜃_{𝑖𝑗,𝑚}+∑_𝑖,𝑗′𝑤^∗_𝑖𝑗′𝜃_{𝑖𝑗′,OR}/1.81+∑_{𝑖,𝑗′′}2𝑤^∗_{𝑖𝑗′′}𝜃_{𝑖𝑗′′,𝑟}∑_𝑖,𝑗𝑤^∗_𝑖𝑗+∑_𝑖,𝑗′𝑤^∗_𝑖𝑗′+∑_{𝑖,𝑗′′}𝑤^∗_{𝑖𝑗′′}, (19) where 𝑤^∗_𝑖𝑗′=1/(𝜎²_𝑖𝑗+𝜏²), 𝑤^∗_𝑖𝑗′=1/(𝜎²_𝑖𝑗′+𝜏²), and 𝑤^∗_{𝑖𝑗′′}=1/(𝜎²_{𝑖𝑗′′}+𝜏²). The variance of  𝜃  is given by var(𝜃)=1/(∑_𝑖,𝑗𝑤^∗_𝑖𝑗+∑_𝑖,𝑗′𝑤^∗_𝑖𝑗′+∑_{𝑖,𝑗′′}𝑤^∗_{𝑖𝑗′′}).

2.2.2. Synthesizing Meta-Analyses

Combining meta-analyses with different types of statistics is similar to synthesizing meta-analyses using the same type of statistics except for the appropriate conversion of summary statistics. We let 𝜏_𝑗 be an estimate for 𝜏 in the meta-analysis 𝑗 using the standardized sample mean difference, 𝜏_𝑗′ in the meta-analysis 𝑗^′ using the log-transformed odds ratio, and 𝜏_𝑗′′ in the meta-analysis 𝑗^′′ using the correlation coefficient. These estimates take the similar form as (17) using data from the corresponding meta-analysis. The estimate for the summary effect in each meta-analysis is derived by using these estimates for 𝜏 and the variances of the effect estimates in the weights and is given by 𝜃_𝑗,𝑚=∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗𝜃_{𝑖𝑗,𝑚}∑^𝑘𝑗_𝑖=1𝑤^∗_𝑖𝑗,𝜃_𝑗′,OR=∑^𝑘𝑗′_𝑖=1𝑤^∗_𝑖𝑗′𝜃_{𝑖𝑗′,OR}∑^𝑘𝑗′_𝑖=1𝑤^∗_𝑖𝑗′,𝜃_{𝑗′′,𝑧}=∑^{𝑘𝑗′′}_𝑖=1𝑤^∗_{𝑖𝑗′′}𝜃_{𝑖𝑗′′,𝑧}∑^{𝑘𝑗′′}_𝑖=1𝑤^∗_{𝑖𝑗′′}, (20) where 𝑤^∗_𝑖𝑗=1/(𝜎²_𝑖𝑗+𝜏²_𝑗), 𝑤^∗_𝑖𝑗′=1/(𝜎²_𝑖𝑗′+𝜏²_𝑗′), and 𝑤^∗_{𝑖𝑗′′}=1/(𝜎²_{𝑖𝑗′′}+𝜏²_𝑗′′). To directly synthesize these meta-analysis results, we use a weighted average of converted effect estimates as follows: ∑𝜃=^𝐽𝑗=1𝑤^∗_𝑗𝜃_𝑗,𝑚+∑^𝑗′_𝑗′=1𝑤^∗_𝑗′𝜃_𝑗′,OR/1.81+∑^𝐽′′_𝑗′′=12𝑤^∗_𝑗′′𝜃_𝑗,𝑟∑^𝐽_𝑗=1𝑤^∗_𝑗+∑^𝐽′_𝑗′=1𝑤^∗_𝑗′+∑^𝐽′′_𝑗′′=1𝑤^∗_𝑗′, (21) where 𝑤^∗_𝑗=∑^𝐽_𝑗=1𝑤^∗_𝑖𝑗, 𝑤^∗_𝑗′=∑^𝐽′_𝑗′=1𝑤^∗_𝑖𝑗′, and 𝑤^∗_𝑗′′=∑^𝐽′′_𝑗′′=1𝑤^∗_𝑖𝑗′. The variance of  𝜃 is given by var(𝜃)=1/(∑^𝐽_𝑗=1𝑤^∗_𝑗+∑^𝐽′_𝑗′=1𝑤^∗_𝑗′+∑^𝐽′′_𝑗′′=1𝑤^∗_𝑗′). The weights 𝑤^∗_𝑗, 𝑤^∗_𝑗′, and 𝑤^∗_𝑗′′ are usually calculated from the variances, 𝑃 values, or the confidence intervals found in original meta-analyses being combined. Articles which only report effect estimates do not have sufficient information on the true effect and should be excluded in the synthesis. Following similar lines in previous sections, we can show that the estimated overall effect is approximately the same as the one by combining individual studies.

The effect estimates used in the expression (21) are already converted to the same type. The estimates reported in original meta-analyses may not be in the same form. For example, a meta-analysis using the correlation coefficient is likely to report the correlation coefficient instead of Fisher’s 𝑧-score. Suppose OR_𝑗′ is the estimated odds ratio in meta-analysis 𝑗^′ and 𝑟_𝑗′′ is the estimated correlation coefficient in meta-analysis 𝑗^′′. In terms of these indices and their variances originally reported in the meta-analyses, the overall effect estimate can be written as the following expression: 𝜃=^𝐽_𝑗=1𝑤_𝑗,𝑚𝜃_𝑗,𝑚+1.81^𝐽′_𝑗′=1𝑤_𝑗′,ORlnOR_𝑗′+12^𝐽′′_𝑗′′=1𝑤_{𝑗′′,𝑟}ln1+𝑟_𝑗′′1−𝑟_𝑗′′×^𝐽_𝑗=1𝑤_𝑗,𝑚+1.81^2𝐽′_𝑗′=1𝑤_𝑗′,OR+14^𝐽′′_𝑗′′=1𝑤_{𝑗′′,𝑟}⁻¹, (22) where 𝑤_𝑗,𝑚=1/var(𝜃_𝑗,𝑚), 𝑤_𝑗′,OR=1/var(ln(OR_𝑗′)), and 𝑤_{𝑗′′,𝑟}=1/{var(𝑟_𝑗′′)/(1−𝑟_𝑗²_′′)²}. And its variance is 𝜃=1var∑^𝐽_𝑗=1𝑤_𝑗,𝑚+1.81²∑^𝐽′_𝑗′=1𝑤_𝑗′,OR+1/4∑^𝐽′′_𝑗′′=1𝑤_{𝑗′′,𝑟}. (23) The proposed estimate in (22) is a summary standardized sample mean difference. Its expression represents two steps. The first step is to convert the odds ratios and correlation coefficients to the same scale of the standardized sample mean difference, and the second step is to calculate the overall mean difference from all meta-analyses.

3. Examples

We illustrate the proposed methods in two examples. The first example illustrates that combining two meta-analyses using our methods produces the same overall effect size as conducting a meta-analysis from all includable primary studies based on either the fixed-effects model or the random-effects model. The second example illustrates the process of combining multiple meta-analyses of the same clinical question which have concordant findings. The calculation is conducted using the R package “metafor.”

The first example is based on a meta-analysis by Swift and Callahan [16], which compared treatment outcomes between appropriately and inappropriately matched groups using 26 individual studies [16]. The treatments in these studies include pharmacotherapy, cognitive behavioral therapy, and group therapy. The treatment outcomes range from substance use days to weight loss. All the studies used the Pearson correlation coefficient as the effect estimate, and the estimates and the confidence intervals are presented in Table 1. The summary correlation coefficient estimate using all 26 studies is 𝑟=0.1334 with 95% confidence interval (0.0951,0.1717) based on a fixed-effects model. We conduct two separate meta-analyses based on studies 1–12 and studies 13–26 as cited in the reference section of the review in Swift and Callahan [16]. The summary effect sizes for these two meta-analyses are calculated to be 𝜃₁=0.1774 with its sample variance being var(𝜃₁)=0.00095 and 𝜃₂=0.1041 with its sample variance being var(𝜃₂)=0.00064. It is of interest to investigate whether combining these two effect sizes gives a similar result as conducting the meta-analysis using all 26 individual studies. We use the inverse of the sample variances as the weights and calculate the weighted average of the summary effect sizes. The resulting overall effect size is 𝜃_𝑓=∑²_𝑗=1𝜃_𝑗/var𝜃_𝑗∑²_𝑗=11/var𝜃_𝑗=0.1334, (24) with its variance being var(𝜃_𝑓)=0.00038. The 95% confidence interval is also (0.0951,0.1717). The results are identical to those found using all individual effect sizes indicating the potential utility of this procedure.

Table 1 

Effect estimates with lower limits (LL) and upper limits (UL) of 95% confidence interval (CI).

To further illustrate the technique we repeat the analyses with a random-effects model (REM). We again use the 26 studies from Swift and Callahan [16] and illustrate that the proposed summary effect size obtained under REM may differ slightly from the one achieved from combining all individual studies. Under the REM, the summary correlation coefficient using all 26 studies is 𝜃=0.1597 with 95% confidence interval (0.0941, 0.2253). We repeat the process described above based on studies 1–12 and studies 13–26, respectively. Using the random-effects model, the summary correlation coefficients for these two meta-analyses are 𝜃^∗₁=0.1827 with its variance being var(𝜃^∗₁)=0.0027 and 𝜃^∗₂=0.1370 with its variance being var(𝜃^∗₂)=0.0018. Combining these two effect sizes produces an overall weighted mean correlated coefficient 𝜃_𝑟=0.1553, with its variance being var(𝜃_𝑟)=0.00108. This variance is smaller than those of separate meta-analyses, indicating more accuracy in estimating the overall correlation coefficient by including more individual studies. The 95% confidence interval is (0.0909,0.2197). Although the estimates of the between-studies variance differ slightly across the two meta-analyses, combining the two pseudoanalyses produces a similar finding to that of combining all of the includable individual studies.

To illustrate the method for synthesizing different types of effect estimates, we create a synthetic dataset using the studies from Swift and Callahan [16]. We convert the correlation coefficients to standardized sample mean differences for studies 1–9 and to log-transformed odds ratios for studies 10–17. The estimates are kept unchanged for studies 18–26. The converted estimates and the corresponding confidence intervals are listed in Table 2. The variances of the estimates are obtained using (UL−LL)²/(2×1.96)². We use the fixed-effects model to conduct meta-analyses on these three sets of studies. The summary effect estimates are 𝜃_1,𝑚=0.4355 with var(𝜃_1,𝑚)=0.0059, 𝜃_1,OR=0.3304 with var(𝜃_1,OR)=0.0183, and 𝜃_1,𝑟=0.0711 with var(𝜃_1,𝑟)=0.0011. The overall effect estimate in mean difference is calculated using (22) to be 𝜃=0.2973 with variance var(𝜃)=0.0017. We see that this result is quite close to the summary effect size calculated from combining individual studies at the beginning of the section which is 0.2692 with the variance 0.0016 after converting to the mean effect size. This indicates that the proposed procedure for directly combining results from meta-analyses produces similar results as the ideal but much more tedious procedure which instead combines individual studies subtracted from meta-analyses.

Table 2 

Effect estimates of different types with lower limits (LL) and upper limits (UL) of 95% confidence interval (CI). Summary effect estimates are calculated based on a fixed-effects model for three sets of studies.

The anterior cruciate ligament (ACL) is a commonly reconstructed ligament of the knee. The meta-analyses conducted by Biau et al. [17] and M. C. Forster and I. W. Forster [18] indicate that the graft failure using hamstring autografts is not significantly different compared with bone-patellar tendon-bone autografts. We sought to apply our approach to synthesize these two meta-analyses on the graft choices in the ACL reconstruction. The summary odds ratios of graft fracture in Biau et al. [17] are OR₁=1.33 with the 95% confidence interval (0.73,2.44) and OR₂=1.09 with the 95% confidence interval (0.40,2.96) in M. C. Forster and I. W. Forster [18]. We first convert the odds ratios and their confidence limits to the natural log scale since meta-analyses using odds ratios are commonly conducted using the weighted average of the log-transformed odds ratios. After the conversion, we obtain the log-transformed odds ratios 𝜃_1,OR=0.2852 with the variance 0.9048 and 𝜃_2,OR=0.0862 with the variance 0.3607. The resulting overall estimate of log odds ratio is 𝜃=²_𝑗=1𝜃_𝑗,OR/var𝜃_𝑗,OR∑²_𝑗=11/var𝜃_𝑗,OR=0.2321, (25) with the variance 0.0695. This variance is smaller than either of the meta-analyses due to the inclusion of more studies. The 95% confidence interval of 𝜃 is (−0.2846,0.2321). Converting back to the original scale, the overall estimated odds ratio based on the two meta-analyses is 1.2613 with 95% confidence interval (0.7523, 1.2613). Although this confidence interval does not show significant difference between two graft choices, it is narrower than those in Biau et al. [17] and M. C. Forster and I. W. Forster [18]. This is expected since the overall estimated odds ratio calculated across meta-analyses provides a more accurate indicator than those calculated from separate meta-analyses.

4. Discussion

Despite the advantages of meta-analyses over traditional narrative review methods, these techniques are not without limitations. As is the case with primary studies, multiple meta-analyses often produce conflicting findings even when they examine the same body of literature [7] given the numerous methodological decisions that are made during the review process. These decisions affect the search strategy, coding of studies, moderators examined, inclusion/exclusion criteria, and the reporting of effect sizes, and each has the potential to impact the review findings. Despite the transparency of the method, there is always a certain degree of subjectivity in research synthesis [14]. We have presented methods which can synthesize meta-analysis findings with minimal subjectivity.

Attempting to make statements about the overall findings of a body of systematic review literature raises a number of challenges. Part of the challenge is the lack of consistency in how authors report the findings of meta-analyses [19, 20]. Since no standardized reporting procedures have been adopted, there is a great deal of variability in how meta-analytic findings are reported both within and across academic disciplines. Additionally, many extant meta-analyses do not completely report effect sizes; confidence intervals, variance measures, and precise 𝑃 values are often omitted. This lack of consistency and quality assurance is a major limitation of this body of research and potentially limits the transportability of meta-analysis findings [21]. Like primary studies, meta-analyses on similar constructs may differ from one another in many ways. To address this issue and make meta-analyses more compatible and therefore more appropriate for synthesis and better equipped to inform practice, it is apparent that the research community needs to continue to work towards adopting a set of standardized best practices for conducting and reporting meta-analyses in behavioral health (see, e.g., [22]).

While the examples provided here illustrate that it is possible to calculate summary effect sizes across multiple meta-analyses of the same intervention, our testing of these techniques is still ongoing. A potential barrier to this approach is the lack of independence that results when multiple meta-analyses include many of the same primary studies. Treating these different outcomes as independent can produce incorrect estimates of the variance for the summary effect, but it does not invalidate the calculation of the effect size itself. This can be seen from (4) for combining individual studies. For example, suppose 𝐽 meta-analyses use the exactly same set of studies. Equation (4) becomes 𝜃=𝐽∑^𝑘1_𝑖=1𝑤^∗_𝑖1𝜃_𝑖1𝐽∑^𝑘1_𝑖=1𝑤^∗_𝑖1, (26) which gives the same estimate as the summary effect estimate in either of the meta-analyses. Ideally, the resulting variance of 𝜃 should be the same as the one in either of the meta-analyses. But instead, we have 𝜃=1var𝐽∑^𝑘1_𝑖=1𝑤^∗_𝑖1, (27) which tends to be considerably smaller than the appropriate value of 1/∑^𝑘1_𝑖=1𝑤^∗_𝑖1. This limitation makes the proposed method more appropriate for meta-analyses with no or a limited number of overlapping studies. A possible inflation factor of 𝐽 can be multiplied with the variance in this case to correct for the biased variance estimator. More generally, with 𝑝 overlapping studies in all meta-analyses, the unbiased variance of the summary estimator can be written as 𝜃=1var∑^𝐽_𝑗=1∑^𝑘_𝑖=1𝑤_𝑖𝑗×𝐽²∑^𝑝_𝑖=1𝑤_𝑖1+∑^𝐽_𝑗=1∑^𝑘_𝑖=𝑝+1𝑤_𝑖𝑗𝐽∑^𝑝_𝑖=1𝑤_𝑖1+∑^𝐽_𝑗=1∑^𝑘_𝑖=𝑝+1𝑤_𝑖𝑗=1∑^𝐽_𝑗=1∑^𝑘_𝑖=1𝑤_𝑖𝑗×IF, (28) where the inflation factor IF corrects for the underestimated variance and we have IF≥1. If 𝑤_𝑖𝑗’s can be retrieved from the overlapped studies, the factor can be precisely calculated. Otherwise, the investigator may have to guess an appropriate value for the inflation factor.

While further research is needed to refine the technique proposed in this paper, the findings of these initial validations suggest the utility of this method for combining mean effect sizes from multiple meta-analyses of the same intervention or treatment. The methodology detailed in this paper has several possible applications. The technique has its most utility when several meta-analyses have been conducted on the same treatment and have produced varying results. Even when multiple meta-analyses report similar findings regarding the magnitude and direction of effect sizes, the proposed technique can be used to summarize across the extant meta-analytic evidence base. The technique also offers a solution when meta-analyses have been conducted to update prior research syntheses; rather than ignoring findings predating the inclusion criteria of the updated study, this technique can be used to summarize the full range of available evidence. These and other potential applications make this a particularly appealing technique for researchers and practitioners alike who are faced with the challenge of summarizing the rapidly expanding body of meta-analytic research in medicine and many other scientific disciplines.

Acknowledgments

The authors would like to thank the associate editor and two referees for their constructive comments.