Review Article | Open Access
Margaret M. MacDonell, Lynne A. Haroun, Linda K. Teuschler, Glenn E. Rice, Richard C. Hertzberg, James P. Butler, Young-Soo Chang, Shanna L. Clark, Alan P. Johns, Camarie S. Perry, Shannon S. Garcia, John H. Jacobi, Marcienne A. Scofield, "Cumulative Risk Assessment Toolbox: Methods and Approaches for the Practitioner", Journal of Toxicology, vol. 2013, Article ID 310904, 36 pages, 2013. https://doi.org/10.1155/2013/310904
Cumulative Risk Assessment Toolbox: Methods and Approaches for the Practitioner
The historical approach to assessing health risks of environmental chemicals has been to evaluate them one at a time. In fact, we are exposed every day to a wide variety of chemicals and are increasingly aware of potential health implications. Although considerable progress has been made in the science underlying risk assessments for real-world exposures, implementation has lagged because many practitioners are unaware of methods and tools available to support these analyses. To address this issue, the US Environmental Protection Agency developed a toolbox of cumulative risk resources for contaminated sites, as part of a resource document that was published in 2007. This paper highlights information for nearly 80 resources from the toolbox and provides selected updates, with practical notes for cumulative risk applications. Resources are organized according to the main elements of the assessment process: (1) planning, scoping, and problem formulation; (2) environmental fate and transport; (3) exposure analysis extending to human factors; (4) toxicity analysis; and (5) risk and uncertainty characterization, including presentation of results. In addition to providing online access, plans for the toolbox include addressing nonchemical stressors and applications beyond contaminated sites and further strengthening resource accessibility to support evolving analyses for cumulative risk and sustainable communities.
The public has become increasingly aware of the wide variety of chemicals present—not just in the environmental media to which they are exposed (such as air, water, and soil) but also in the food they eat and the products they use. As access to relevant information continues to grow, notably via the Internet, many communities have voiced concerns about health effects associated with the multiple chemicals in their daily lives. To address these concerns, many organizations have responded with approaches, guidelines, focused workshops, and illustrative applications to better assess cumulative risks. These organizations include the US Environmental Protection Agency (EPA), National Institute for Occupational Safety and Health (NIOSH), Agency for Toxic Substances and Disease Registry (ATSDR), World Health Organization (WHO), California Environmental Protection Agency (Cal/EPA), the Environmental Justice (EJ) community, and professional organizations such as the Society of Toxicology.
Cumulative risk assessment (CRA) explicitly considers the combined fate and effects of multiple contaminants from multiple sources through multiple exposure pathways. The goal of CRA is to address more realistic conditions than those addressed under the classic approach of the 1983 National Research Council (NRC) “red book” on Risk Assessment in the Federal Government , which agencies historically implemented by evaluating one chemical at a time. Reflecting advances in scientific knowledge since that time, which made more detailed and integrated analyses possible, EPA outlined its Framework for Cumulative Risk Assessment  in 2003 that considered joint exposure to multiple chemical, physical, and biological stressors. A number of organizations have published additional cumulative risk reports over the past decade [3–9]. Despite the increase in relevant analyses and reports, the translation of a more fully integrated approach to practice has lagged behind the science. With various groups and individual community members unaware of available tools that could be used to assess cumulative risk, explicit applications have been relatively modest.
The concept of a “cumulative risk toolbox” emerged soon after the EPA framework was published, during the early stages of developing a companion report to support evaluations at contaminated sites. The purpose of this CRA toolbox was to overcome the “awareness” hurdle and serve as a practical resource for cumulative risk assessors and other interested parties. Recognizing that cumulative risk encompasses many different kinds of stressors ranging from chemical and physical to biological and psychological ones, the CRA toolbox compilation that began in 2004 targeted a specific scope, namely, health risks from exposure to multiple anthropogenic chemicals as well as to elevated concentrations of chemicals that occur naturally. The main objectives of the CRA toolbox were to (1) consolidate resources for conducting cumulative risk assessments, (2) provide coverage for the main elements of the assessment process, and (3) offer practical notes to help assessors understand the strengths and limitations of a given approach or model for cumulative risk applications.
The original CRA toolbox was published as an appendix to the 2007 Agency report, Concepts, Methods, and Data Sources for Cumulative Health Risk Assessment of Multiple Chemicals, Exposures, and Effects: A Resource Document . It was recently reevaluated to determine if the descriptions and electronic web links for the individual entries were current or needed to be updated. This paper highlights relevant excerpts from the CRA toolbox with interpretive context for CRAs at contaminated sites, with updates as indicated; it also identifies several additional resources not included in the original CRA toolbox.
The CRA toolbox was developed in three phases: (1) identification of information resources related to CRA, (2) assessment of their relevance to contaminated sites and facilities, and (3) categorization of the relevant resources to facilitate their use in analyses. In searching the scientific literature and other online materials for information related to CRA, the resources pursued included conceptual approaches, models, and other tools related to assessing health risks posed by multiple chemicals in multiple environmental media via multiple exposure routes. In addition, risk analysts from several agencies, universities, and the private sector were informally invited to identify methods and models they had found useful for assessing multiple contaminants, exposures, and effects as part of their environmental projects and programs.
Some information evaluated in the second phase was designed to assess ecological effects, while other material addressed acute and shorter-term human exposures; however, the primary emphasis of the CRA toolbox was on approaches and tools for evaluating health effects from chronic to lifetime exposures. An example of this type of resource is the NIOSH approach for worker protection that accounts for exposures to chemical mixtures . The resources considered less well suited than others to this type of application were screened out at this stage, as were resources that essentially duplicated content that had already been compiled. In the third phase, the resources selected for inclusion were collectively evaluated to identify common themes. The purpose of this last phase was to define a manageable set of categories to serve as the organizational structure of the CRA toolbox. This toolbox was not intended to be comprehensive; rather, it was developed to provide a suite of resources, including guidelines, approaches, and models, that could be applied to assess cumulative health risks associated with contaminated sites. For this reason, three basic documents guided the determination of the CRA toolbox categories: the 1983 NRC risk assessment paradigm , the 1989 EPA Risk Assessment Guidance for Superfund , and the 2003 EPA Framework for Cumulative Risk .
3. Results and Discussion
3.1. Illustrative Information Resources
The online searches and interactions with risk practitioners produced more than 100 methods and tools relevant to CRAs at contaminated sites. Not all explicitly target multiple contaminants, exposures, or effects. That is, many were developed for general risk assessment purposes, but they were found to be either directly useful or adaptable for population-specific CRAs, or the underlying approach was found to offer useful insights for such analyses. Many of the materials addressed one or two components of a CRA, while others addressed multiple elements or the overall concept. Although certain resources clearly consider multiple exposures to multiple chemicals, such as the standard Agency guidance for risk assessment at contaminated sites , only a fraction are explicitly defined as cumulative risk tools. Relatively few specifically focus on exposure groupings or joint toxicity, or on a given population group for CRA, such as children (e.g., see (4.1) in Table 4 and (6.7) in Table 6). Although many focus on contaminant sources or affected environmental media, as has been the historical practice, they can also be applied to more complex cumulative risk analyses. Similarly, various resources can be helpful for evaluating subsets of an overall cumulative risk assessment, such as cumulative hazards, threats, exposures, or impacts, including health impact analyses.
The resources compiled for the CRA toolbox are available on websites managed by a variety of organizations, including (1) federal agencies and institutes such as the EPA, US Nuclear Regulatory Commission, US Department of Energy (DOE), and US Department of Health and Human Services, including NIOSH and ATSDR within the Centers for Disease Control and Prevention (CDC), as well as the National Institutes of Health (NIH), including the National Institute of Environmental Health Sciences (NIEHS); (2) national scientific organizations such as the National Academies, including the National Academy of Sciences and National Research Council; (3) other federal entities such as the National Environmental Justice Advisory Council; (4) national organizations in other countries, including Canada (Health Canada, Environment Canada, and the Quebec worker protection institute IRSST, Institut de Recherche Robert-Sauvé en Santé et en Sécurité du Travail) and The Netherlands (Organization for Applied Scientific Research, Nederlandse Organisatie voor Toegepast Natuurwetenschappelijk Onderzoek (TNO) Nutrition and Food Institute); (5) international bodies, such as the European Commission and WHO; (6) private nonprofit organizations such as the International Institute of Indigenous Resource Management (IIIRM); (7) state agencies, such as the Cal/EPA Office of Environmental Health Hazard Assessment (OEHHA) and Air Resources Board (ARB); (8) industry organizations such as the American Chemistry Council (ACC); (9) scientific groups at universities and national laboratories; and (10) local community advisory boards at contaminated sites and facilities, such as site-specific, restoration, and citizen advisory boards (SSABs, RABs, and CABs).
3.2. CRA Toolbox Categories
The CRA toolbox is divided into five sections as described in Table 1 and illustrated in Figure 1, with some sections further subdivided to identify specific content. (The resource count in Table 1 refers to the number of tools highlighted in tables for each topic.)
3.3. Resource Highlights
The resources highlighted in this section focus on those identified in the original CRA toolbox (which was developed in 2004), with limited updates. Several additional tools that include more recent content are noted in Section 3.5. While several tools apply to multiple categories, they are generally listed with the first CRA toolbox category for which they are particularly useful. (Note that planning and scoping for CRAs continues iteratively throughout the assessment process.) Each section begins with a topical introduction, followed by text bullets highlighting selected resources, and concluding with a table that summarizes the following for a larger number of entries: (1) title, author/organization, and access information (e.g., web link), where available; (2) purpose and scope; (3) remarks on application to CRA. Nearly 80 resources are summarized across the five tables combined, with additional resources highlighted in the accompanying text.
3.3.1. Category 1: Resources for Planning, Scoping, and Problem Formulation (Table 2)
Topics addressed during planning, scoping, and problem formulation include the purpose and breadth of the assessment (considering multiple chemicals, population groups, exposures, and effects), the type of product needed from the assessment to inform a decision, the data to be collected and synthesized, the general assessment approach, and stakeholder involvement. CRAs can involve a very large number of potential combinations of chemicals and interactions inherent to the environmental setting. During this initial and iterative phase of the process, common questions include which chemicals are most likely to contribute significantly to risks, whether they might interact, and what the nature of those interactions could be. The Internet has become a valuable tool for promoting and enhancing stakeholder involvement, and successful programs often combine traditional methods (ranging from one-on-one to town hall meetings and printed newsletters or other information sent in regular mail) with electronic approaches to take advantage of the unique benefits of each. Selected resources that can be used to support planning, scoping, and problem formulation for CRAs, including stakeholder involvement, are presented in Table 2; several specific resources are highlighted below.
Planning and Scoping. In 1997, EPA released its Guidance on Cumulative Risk Assessment, Part 1, Planning and Scoping ((2.1) in Table 2) that presented broad-based approaches that considered (1) multiple endpoints, sources, pathways, and routes of exposure; (2) community-based decision making; (3) flexibility in achieving goals; (4) case-specific responses; (5) all environmental media; and (6) holistic risk reduction. The companion Lessons Learned on the Planning and Scoping of Environmental Risk Assessments ((2.2) in Table 2) followed in 2002, presenting case studies to illustrate organizing principles for CRAs. The following year, the Agency published its Framework for Cumulative Risk Assessment ((2.3) in Table 2) that outlines an umbrella structure for CRAs, identifies key issues, and defines common terms. Neither a procedural guide nor a regulatory requirement, the Framework document summarizes key elements of the CRA process as part of a flexible structure rather than identifying prescriptive protocols. Providing the basic information about important aspects of cumulative risk, that framework continues to serve as a key foundation for CRA reports. One example of a tool based on the framework is EJView ((2.4) in Table 2), a GIS-based module jointly designed by the EPA Offices of Environmental Information (OEI) and Environmental Justice (OEJ). Combining environmental, socioeconomic, and health indicators in statistical tables, this tool was initially developed to evaluate potential EJ issues. For community-based approaches, this and other ranking and prioritization tools can help identify the problems warranting consideration in a CRA. Note that although it is presented here within the planning/problem formulation stage to support front-end scoping, this and other such tools are also very useful for other phases of the CRA process, including risk characterization.
Stakeholder Involvement. A number of tools have been developed to support stakeholder involvement in CRAs, particularly for contaminated sites. Ranging from guidance for EPA’s Superfund and EJ programs to project-specific field activities (see the second section of Table 2, (2.5) through (2.10)), these resources chronicle approaches taken to solicit inputs from multiple stakeholders and incorporate them into the assessment plans. A number of examples from DOE legacy waste sites reflect inputs of long-standing community advisory boards, from the DOE Savannah River Site in Georgia to the DOE Hanford Site in Washington. Practical insights can also be gained from the Risk Analysis, Communication, Evaluation, and Reduction (RACER) project at the DOE Los Alamos National Laboratory (LANL) in New Mexico. Led by the Risk Assessment Corporation (RAC) team, extensive stakeholder involvement has been a hallmark of that effort ((2.8) in Table 2).
Data Quality Objectives. The EPA has developed a series of guidance documents to help ensure that all data collection, including at contaminated sites, is appropriate for the intended use, which is particularly important for CRAs given the typical complexity of these analyses. These documents outline a systematic process for developing performance criteria to collect, evaluate, and use environmental data. Statistical and analytical tools underlie data quality objectives (DQOs), as highlighted in the last section of Table 2 ((2.11) through (2.17)).
3.3.2. Category 2: Resources for Environmental Fate and Transport Analyses (Table 3)
A large number of tools have been developed to assess the environmental fate and transport of chemicals, and many of these can be used to support CRAs, as highlighted below and in Table 3. These tools include computer models available from the EPA Center for Subsurface Modeling Support (CSMoS), as well as resources for physicochemical constants and guidance for determining background concentrations in soil.
Fate and Transport. Risk assessments for contaminated sites and also for urban environments and other settings impacted by multiple pollutant sources commonly simulate the behavior of multiple chemicals in the environment because of the relatively high costs (in terms of both manpower and dollars) to conduct measurements. Hundreds of computer models have been developed to model contaminant fate and transport in the environment. Some are very general and conceptual, while others address specific media characteristics and setting conditions. The use and suitability of individual models vary widely depending on the project objectives and data required, so it is important for the selected model to be appropriate for the given evaluation. CSMoS is a key resource for these tools; the center maintains an online database of publicly available groundwater and vadose zone fate and transport models, a number of which are included in Table 3 (see entries (3.13)–(3.23)). Selected tools are also highlighted below.
Physicochemical Constants. Several online resources provide information on the chemical, physical, and biological properties of substances, including industrial products and byproducts. Resources highlighted in the original toolbox include the EPA ChemBioFinder Database ((3.1) in Table 3) and Soil Screening Guidance ((3.2) in Table 3), which includes an extensive set of environmental and physical constants and parameters that can be used to model the fate and transport of chemicals in soil. The chemical-specific properties used to derive EPA Regional Screening Levels are also available online (see related discussion for Table 5). Information about these properties can be used to predict that chemicals will likely share a similar environmental fate to support exposure groupings for CRAs.
Background Concentrations. Concentrations that represent natural background or ambient conditions are location specific and provide valuable context for assessing chemical fate and transport as well as incremental risk. The EPA has outlined an approach for characterizing background concentrations, including protocols for determining whether site measurements are statistically elevated, in Guidelines for Characterizing Background Chemicals in Soil at Superfund Sites ((3.3) in Table 3). Data on background concentrations of inorganic chemicals can be found in several sources that provide baseline context for community-based environmental risk assessments. Information sources include ATSDR toxicological profiles ((5.4) in Table 5) and technical reports from the US Geological Survey, as illustrated by a report on constituents of ambient surface soil in Chicago that was prepared in cooperation with the City of Chicago . Similar regional data can be found in state-specific resource reports, such as from the Massachusetts Department of Environment  as well as the Texas Commission on Environmental Quality (TCEQ) , regarding background levels of polycyclic aromatic hydrocarbons (PAHs) and other constituents in soil. (Also, see related TCEQ entry (6.5) in Table 6.)
Vapor Intrusion. Vapor intrusion can be an important exposure pathway for multiple chemicals when volatile organic compounds are in the subsurface (e.g., soil and groundwater) and can migrate to indoor air. Contributions from vapor intrusion are commonly combined with estimates from other indoor air pathways (e.g., inhalation of volatiles during showering) to quantify aggregate exposures and risks for single chemicals (e.g., benzene) and cumulative risks for groups of chemicals (e.g., chlorinated solvents). This pathway has historically been evaluated using a model based on the equation published by Johnson and Ettinger in 1991 . The model is a one-dimensional spreadsheet that estimates convective and diffusive transport of chemical vapors to indoor air from sources near a building’s perimeter. Attenuating factors such as biological degradation were not included in the original model, and the source was assumed to be infinite over the exposure duration assessed (e.g., 25 years for a commercial or industrial worker). The EPA provided a detailed description of this earlier vapor intrusion model in its draft guidance issued in 2002, and since that time, the Agency and others have continued to strengthen the modeling approach ((3.11) in Table 3). Following the early Johnson and Ettinger model, several US states have adopted simple equations based on this method to conduct screening evaluations of indoor air ((3.11) in Table 3). The indoor air concentrations calculated by these models across multiple chemicals can be combined to estimate cumulative exposures and corresponding risks.
3.3.3. Category 3: Resources for Exposure Analysis (Table 4)
Many exposure models are well suited to assessing multiple chemicals by multiple routes, although this is generally performed by combining predictions made for individual chemicals. Tools range from relatively straightforward screening models to comprehensive multimedia models, as highlighted below and further illustrated in Table 4. Certain models also support other portions of the risk assessment process. For example, models for subsurface vapor migration described earlier are often tapped for multimedia exposure assessments because they consider both soil and groundwater inputs. In addition, several technical reports identify exposure factors, their bases, and parameters commonly used to estimate cumulative exposures. This category includes resources linked to fate and transport models, in some cases to account for the time lag between release and exposure considering the movement of chemicals from source to receptor. The analysis of changing chemical exposures over time is also an important concept for grouping chemicals, and models that incorporate this factor are included in Table 3 (e.g., see (3.7), (3.14), and (3.15)).
Exposure Factors. Risk assessments commonly rely on exposure models to capture receptor-specific factors that influence chemical intakes. For example, factors that address exposure duration, time involved in certain activities, body weight and surface area, intake rates (e.g., inhalation, or ingestion of food, soil, or water), and many others parameters needed to estimate potential risks from multiple exposures are available from the EPA 2011 Exposure Factors Handbook and 2008 Child-Specific Exposure Factors Handbook (see (4.1) in Table 4). To support the evaluation of potentially vulnerable or susceptible subgroups, EPA’s 1999 report on Sociodemographic Data Used for Identifying Potentially Highly Exposed Populations ((4.2) in Table 4) provides information to help identify subsets of the general population who may be at greater risk for negative health consequences, which can be incorporated into CRAs. An additional valuable resource is the National Human Exposure Assessment Survey (NHEXAS, (4.3) in Table 4), which was developed by EPA in the early 1990s to compile information on human exposures to chemicals at the community and regional scales, with an emphasis on associating these exposures with personal activities. The NHEXAS database is also noted in Table 4.
Multipathway Releases and Exposures. The 3MRA model ((4.4) in Table 4) is a multimedia, multipathway, multireceptor exposure and risk assessment model developed by EPA to assess releases from land-based waste management units. After simulating releases from disposal units, modules model fate and transport through the environment, estimate exposure to receptors, and calculate distributions of risks to receptors. The 3MRA methodology uses a Monte Carlo scheme to quantify uncertainty (e.g., from natural variability or based on selection of representative sites). The Exposure and Fate Assessment Screening Tool (E-FAST, (4.5) in Table 4) is another computer-based model that can provide screening-level estimates for general population, consumer, and environmental exposures to chemicals released to air, surface water, or landfills and those released from consumer products. Potential inhalation, dermal, and ingestion doses resulting from these releases are estimated, with the modeled concentrations and doses designed to reasonably overestimate exposures for use in screening-level assessments. States have also developed tools to assess exposures via the air pathway, such as California’s Air Toxics “Hot Spots” Program that requires stationary air emission sources in the state to report the types and quantities of certain substances routinely released to air and also to estimate potential exposures to surrounding populations. A software package was developed to support these evaluations ((4.10) in Table 4).
Dietary Exposures. The Dietary Exposure Potential Model (DEPM) ((4.12) in Table 4) estimates dietary exposure to multiple chemicals based on data from several national, government-sponsored food intake surveys and chemical residue monitoring programs. The DEPM includes recipes developed specifically for exposure analyses that link consumption survey data for prepared foods to the chemical residue information, which is normally reported for raw food ingredients, to estimate daily dietary exposure. The summary databases are aggregated in a way that allows the analyst to select appropriate demographic factors, such as age/sex groups, geographical regions, ethnic groups, and economic status. The model also includes modules for evaluating chemical exposures from residues, soil, and tap water.
3.3.4. Category 4: Resources for Toxicity Analyses (Table 5)
Resources that can be used to support toxicity analyses for CRAs are highlighted here and summarized in Table 5. Topics include (1) resources for toxicity reference values for various exposure routes and durations; (2) development of toxicity factors, including for whole mixtures; (3) identification of toxicity criteria for similar or surrogate compounds or mixtures to represent a mixture or its components; and (4) joint toxicity of the components of a mixture.
Toxicity Reference Values. The EPA Integrated Risk Information System (IRIS) ((5.7) in Table 5) is a key source of information on chronic toxicity reference values, including reference doses (RfDs), reference concentrations (RfCs), and oral slope factors, unit risks, and corresponding risk-based concentrations  (see (5.7) in Table 5). Chronic toxicity values are also available in IRIS for certain mixtures (such as the RfC for diesel exhaust and cancer toxicity values for polychlorinated biphenyls). Note that if a standard toxicity value is not available in IRIS, a Provisional Peer-Reviewed Toxicity Value (PPRTV; derived for use in the Superfund Program) may be available ((5.8) in Table 5). PPRTVs are derived using the same methods, data sources, and EPA guidance used to derive IRIS values but undergo a comparatively more rapid scientific review . If a relevant PPRTV is not available, some states have also developed selected toxicity values that may be found in summary tables of EPA Regional Screening Levels ((5.11) in Table 5). Recent IRIS assessments commonly include data on effects other than the critical effect used to derive a toxicity value; these effects (sometimes called secondary effects) can be used to at least qualitatively assess joint toxicity (usually via dose or response addition) of combined chemical exposures. The information provided in EPA’s RfD arrays also could be used to support estimates of target organ toxicity doses based on secondary effects . The ATSDR has developed toxicological profiles for many individual chemicals that identify the effects of the given chemical, as well as primary environmental and metabolic transformation products to support grouping by specific target organs and systems. A smaller set of interaction profiles has also been developed that assesses joint toxicity  (see (5.4) in Table 5).
Chemical Mixtures and Pesticides: Common Mode of Action. In 2000, EPA updated its 1986 guidelines for chemical mixtures ((5.1) in Table 5). This supplemental guidance describes risk assessment approaches that depend on the type, nature, and quality of available data, and it includes equations, definitions, discussions of toxicological interactions and pharmacokinetic models, and approaches for assessing whole mixtures, surrogate mixtures, and individual mixture components. The whole-mixture discussion includes the derivation of whole-mixture toxicity values (RfDs, RfCs, cancer slope factors, and inhalation unit risks), as well as consideration of comparative potency and environmental transformations. The component discussion includes dose addition, the hazard index (HI), interaction-based HI, relative potency factors (RPFs), and response addition.
In 2002, EPA published guidance for assessing the cumulative risk of pesticides with a common toxic mechanism to address requirements set forth in the Food Quality Protection Act of 1996  (see (5.2) in Table 5); that guidance was updated in 2006 . Note that common mechanism was interpreted by EPA as common mode of action . The initial guidance considered potential exposures to 30 organophosphate pesticides via food, drinking water, and residential uses, and it applied methods to account for variable exposures per different ages, seasons, and geographic factors.
EPA scientists in the National Center for Environmental Assessment collaborated with colleagues in the National Toxicological Research Center of the Food and Drug Administration to develop additional component approaches for assessing the cumulative risk posed by exposures to multiple chemicals by evaluating three scenarios [22, 23]. They explored a simple scenario in which it was certain that all chemicals being considered shared a common toxic mode of action, so a dose-additive approach could be applied. In the second scenario, modes of action for the chemicals in the mixture were known and could be used to divide the chemicals into independent mode-of-action subclasses; dose addition and response addition were then integrated to assess the risk. In the third scenario, the mode or modes of action were uncertain for the chemicals in the mixture, so a joint dose-response modeling procedure was developed that created a range of risk estimates.
Physiologically Based Models and Chemical Mixtures Toxicology Research. Statistically based methods and computer tools that can model interactions and effects associated with multiple chemicals continue to be developed and refined. A main area of study involves applying physiologically based pharmacokinetic/pharmacodynamic (PBPK/PD) models to chemical mixtures. Reaction network modeling is an example of a computer-based approach that has been used in petroleum engineering to predict chemical reaction rates, products, and outcomes based on various statistical methods (including Monte Carlo-type analysis). A molecular-based model (BioMOL) was designed to use the reaction network modeling approach to predict effects of chemicals in complex biological systems .
Joint toxicology studies have also improved the understanding of potential health effects of chemical mixtures with different modes of action. For example, the TNO Nutrition and Food Research Institute of The Netherlands has evaluated the use of mechanistic models to describe interactions between mixture components expected to act by different modes of action. In a pilot study funded by the American Chemistry Council Long-Range Research Initiative (see (5.10) in Table 5), the TNO team applied PBPK models to assess possible toxicokinetic interactions between compounds in an applied mixture and compared those estimates to empirical dose-response modeling of observed pathological changes in the liver, blood, and kidney. The aim of such research is to develop and refine methods to be applied to other chemical mixtures. Other TNO studies have developed and applied statistical methods combining multivariate data analysis and modeling in in vitro and in vivo studies on various chemical mixtures such as petroleum hydrocarbons, aldehydes, food contaminants, industrial solvents, and mycotoxins [25, 26].
Similarly, NIH/NIEHS has sponsored studies on mixtures toxicology and environmental health, including as part of the National Toxicology Program (NTP), for which related reports and fact sheets are available from the NIEHS website . A search engine on this website can be used to tap research and tools for specific applications, including those related to cumulative risk. NIEHS also publishes Environmental Health Perspectives, a monthly journal that often reports on studies relevant to chemical mixtures, with some issues and supplements entirely dedicated to mixtures. Also, NIH maintains the National Library of Medicine and other databases (see (5.9) in Table 5). In addition to the organizations highlighted in Table 5, others have also been assessing mixtures to support CRAs during the past decade. For example, the Health Canada Toxic Substances Research Initiative (TSRI) assessed cumulative effects of environmental toxics to both human and ecological receptors [28, 29]. Cal/EPA conducted a health assessment in the 1990s that focused on a representative complex mixture, diesel particulate matter (DPM) . The scientific review panel for this study used the analyses of epidemiological data from workers to develop a unit risk estimate for diesel particulates, which was then used to derive an inhalation slope factor. This approach offered insights not only for assessments involving diesel exhaust but also for other chemical mixtures.
Benchmark Dose Software (BMDS) and Categorical Regression (CatReg). The EPA developed the BMDS to fit mathematical models to toxicological dose-response data for a particular toxic effect ((5.6) in Table 5). The user evaluates results of this statistical software to select a benchmark dose (BMD) associated with a predetermined benchmark response (BMR), such as a 10% increase in the incidence of a particular lesion or a 10% decrease in body weight. A goal of the BMD approach is to define a point of departure to derive an RfD or RfC that is more independent of study design than the traditional method based on a single experimental dose, such as the no-observed-adverse-effect level (NOAEL). The HI uses RfDs or RfCs in a formula that is based on dose addition to scale the exposure levels in a mixture, producing an indicator of the extent of concern for toxicity. The BMD values used with dose addition could estimate a BMD for the mixture, allowing the mixture dose to be interpreted in terms of the risk of a particular effect. BMD modeling could also be applied to whole mixture dose-response data.
The categorical regression tool CatReg was developed to conduct dose-response modeling of data on diverse endpoints across multiple toxicological studies by categorizing effects into different severity levels, such as no-effect and adverse-effect levels ((5.3) in Table 5). The EPA has suggested that categorical regression results for a single chemical can yield benchmark doses (e.g., a 10% effective dose or ED10) or risk estimates that reflect the probability of observing a severity level of (nonspecific) response . Once this is done for each component of a mixture, risk assessment methods for chemical mixtures such as the HI or response addition approach can then be applied to yield an indication of risk for the mixture. For CRAs, CatReg can be applied to evaluate grouped chemicals considering multiple effects. Therefore, this tool could be helpful for both the toxicity assessment and risk characterization components of an integrated risk analysis.
Risk-Based Screening Levels. Risk-based screening concentrations have been developed for environmental media (including soil, drinking water, and air) by several organizations, including EPA. For example, EPA Regions 3, 6, and 9 previously developed risk-based concentrations (RBCs), medium-specific screening levels (MSSLs), and preliminary remediation goals (PRGs), respectively; these similar values were subsequently combined to produce regional screening levels (RSLs) for assessing contaminated sites ((5.11) in Table 5). These screening values can be used to narrow the focus of assessments on key contributors to risks; they are based on conservative default assumptions for exposure and environmental parameters, and they reflect toxicity values from IRIS and other sources (e.g., Cal/EPA).
3.3.5. Category 5: Resources to Characterize Risk and Uncertainty and Present Results (Table 6)
Many assumptions are made when assessing health risks from environmental exposures to multiple chemicals. Thus, it is important for the risk estimates and associated uncertainties to be well characterized and clearly presented so this information can be interpreted appropriately to guide sound decisions. This final phase of the CRA process has come to rely on issue-specific summaries and graphical tools to display statistical and spatial information, as highlighted here and in Table 6.
Spatial Analysis and Decision Assistance Tool (SADA). The SADA tool was jointly developed by the EPA, US Nuclear Regulatory Commission, and University of Tennessee as an integrated software package to support human and ecological CRAs (see (6.1) of Table 6). Like many other tools (including the RSLs), the human health module of SADA includes equations from the standard Superfund guidance  and accommodates different land use scenarios and exposure pathways. These can be combined to estimate overall exposures for the selected receptors. This tool emphasizes the spatial distribution of contaminant data, and individual modules cover visualization, geospatial analysis, statistical analysis, sampling design, and decision analysis. Outputs can be tabular or graphical, and they can be used to identify where estimated risks exceed target values. Although the input data for these pathways can be tailored to reflect site-specific conditions, interactions are not considered.
Probabilistic Methods and Tools. Risk assessments commonly present human health risks as single-point estimates (e.g., 1 × 10−5) in accordance with standard guidance for contaminated sites . Such estimates provide little information about the underlying uncertainty or variability. For example, Monte Carlo simulation tools can be used to consider the effect of uncertainty and variability, with results approximating a full range of reasonably possible outcomes that are typically plotted as graphs (e.g., frequency distributions) or tabulated. While these probabilistic approaches have not yet been widely implemented in environmental health risk analyses for contaminated sites, such simulations can help assessors represent uncertainty and variability in the risk results (see (6.7) in Table 6). Extensive research in probabilistic analysis relevant to CRAs has been conducted at a number of universities, including North Carolina State University (C. Frey and colleagues) and the University of Washington (A. Cullen and colleagues). These evolving tools incorporate distributions of parameter values that may be more appropriate for a given assessment than point estimates, and they are particularly well suited for CRAs.
Community-Based Air Pathway Modeling and Other Joint Exposure-Risk Resources. The EPA Region 6 Regional Air Impact Modeling Initiative (RAIMI) was originally designed as a GIS-based system that could tap a number of emissions data sources to assess potential impacts at the community level ((6.3) in Table 6). This system also supports source attribution analyses, and early findings indicated that a small number of sources were responsible for most of the impact. Initially implemented at a pilot scale that focused on risks from direct inhalation exposures, RAMI was subsequently expanded to assess indirect exposures resulting from airborne releases, thus increasing the relevance for additional CRA applications. Tools identified with the exposure assessment phase that consider the spatial scale of various impacts (e.g., (4.7) in Table 4) can also be valuable resources for the risk characterization phase of CRAs. Similarly, disease registries that provide context for exposure assessments (see (4.15) in Table 4) also serve as resources for risk characterization by presenting health data that can be considered in concert with modeled or measured chemical data to assess potential influences of multiple exposures (including population-specific or location-specific patterns) and to calibrate risk models.
3.4. Review Findings
This reevaluation of the original toolbox (as described in EPA’s 2007 report, Concepts, Methods, and Data Sources for Cumulative Health Risk Assessment of Multiple Chemicals, Exposures, and Effects: A Resource Document ) found that only half the web links were still valid, as most websites and models have been updated since the original toolbox was published. However, most information is still available in some form, and only a few web pages no longer exist. The web links presented in this overview of the original toolbox reflect the updated web addresses as of early 2013.
The review of the original toolbox also found that many compilations, approaches, and models have been updated, and additional resources are now available online. For example, a targeted search produced more than twice the number of resources reflected in the original toolbox. Additional organizations represented in this ongoing update of the CRA toolbox include the US Army Corps of Engineers, US Geological Survey, and US Department of Agriculture.
Both new and updated models can be found via a number of EPA (and other) websites, including the Center for Exposure Assessment Modeling (CEAM), National Center for Environmental Assessment (NCEA), and National Exposure Research Laboratory (NERL), as well as the Council for Environmental Regulatory Modeling (CERM). Most of these additional resources focus on the broad topic of exposure, including environmental fate and transport, but some address toxicity and risk characterization resources. Selected examples are provided in the following section.
3.5. Selected CRA Resources in addition to Those Listed in the Original Toolbox
Several cumulative risk meetings held within the last several years have produced compilations in a manner similar to the effort undertaken for the original toolbox in 2004 to support the EPA CRA resource document (which was published in 2007). These include workshops organized by EPA that involved external scientists, as well as an internal EPA workshop that convened cumulative risk experts and EPA Program and Regional project leads in July 2009 to share insights from recent and ongoing applications. The models and other resources compiled from some of these meetings are available in the scientific literature and online [31–33]. Furthermore, the NRC published two studies relevant to cumulative risk [7, 8], one in 2008 that focused on a specific class of chemicals (Phthalates and Cumulative Risk Assessment: The Task Ahead), and the other in 2009 that focused on how current risk analysis approaches could be improved (Science and Decisions: Advancing Risk Assessment). In addition, after an extensive process of collaborative workshops and soliciting input from interested parties, in late 2010 Cal/EPA OEHHA released its report on Cumulative Impacts: Building a Scientific Foundation .
For the exposure category, a key resource not included in the original toolbox is the CDC’s National Health and Nutrition Examination Survey (NHANES) database. This database contains information on the health and nutritional status of adults and children, and it is unique in its incorporation of data from both interviews and physical examinations. NHANES is also a source of information on the distribution of some contaminant concentrations in human tissues drawn from samples collected in the US population. Valuable research findings have been made possible by this unique database, including recent evaluations of cadmium exposures and effects in children that inform risk assessments for contaminated products from overseas . Information about the NHANES program and research publications can be found online via the CDC website .
For the toxicity category, in addition to the recent online availability of the EPA PPRTVs , the IRIS database  now provides additional toxicity information that can be used to assess chemical mixtures. For example, information is included from recent toxicological reviews that evaluate common modes of action and common target organs and systems, which can be considered in evaluating secondary effects and establishing toxicity groupings for CRAs. NCEA has also developed a new database of the toxicity literature for a number of chemicals, called Health and Environmental Research Online (HERO), which provides information underlying recent risk-related EPA analyses and can also be mined to support CRAs .
The International Toxicity Estimates for Risk Assessment (ITER) database is another toxicity resource , which was created and is maintained by the nonprofit scientific organization Toxicology Excellence for Risk Assessment (TERA). Its tabulated data include toxicity values and cancer classifications for more than 600 chemicals, with synopses that explain data differences and web links to the source organizations for further information.
Progress also continues on the international front, spanning each key element of a CRA. In 2009, the WHO released its draft report for public review, Risk Assessment of Combined Exposures to Multiple Chemicals: A WHO/IPCS Framework . Also that year, the European Commission moved forward with its ongoing initiative, NoMiracle (Novel Methods for Integrated Risk Assessment of Cumulative Stressors in Europe), including making a toolbox available to support these analyses .
Only a few of the recent resources have been highlighted here. Many more continue to be developed and refined to support CRAs, as the science evolves to meet the demands of the broader community for more realistic assessments that can guide effective risk management decisions. Resource toolboxes such as this one can support more integrated compilations to help address the need for increased awareness and access to practical approaches for cumulative risk assessors. CRAs that tap resources in this toolbox—which extend from EPA’s cumulative risk framework and resource document to conceptual models, exposure factor handbooks, and toxicity databases, together with census data, specific fate models, community involvement processes, and new visualization tools—are being reflected in the growing literature on this topic (e.g., see [32, 38–44]).
A number of existing methods and tools can be applied to assess cumulative risks for contaminated sites, and progress continues in developing and refining such resources. Most of the tools in the original CRA toolbox have been updated, and others have become available. Nevertheless, explicit CRA applications are not yet widespread as issues related to general awareness and integration of relevant tools persist. Meanwhile, programs continue to evolve that are well suited to CRA approaches, which could help enhance the understanding of, and guide measures to address, combined stressors before they become a problem. Thus, the need exists for a broadly accessible toolbox that can serve as a practical foundation for cumulative risk practitioners across agencies, academia, and the private sector, as well as for interested members of the general public. Plans designed to address this need include the following:(i)Strengthen the online accessibility of CRA toolbox resources; (ii)Provide additional practical context from case studies; and(iii)Incorporate further tools that extend to nonchemical stressors and other applications, to address emerging themes including sustainable communities.
Conflict of Interests
The authors have no direct financial relations with any trademarks mentioned in this paper.
The authors would like to thank their many colleagues who have contributed to this research area over the past decade and beyond, including those whose approaches and tools are reflected in the CRA toolbox. The views expressed in this paper are those of the author(s) and do not necessarily reflect the views or policies of the US Environmental Protection Agency (EPA). Argonne National Laboratory’s work was supported by the EPA under interagency agreement, through US Department of Energy Contract no. DE-AC02-06CH11357.
- National Research Council, “Risk assessment in the Federal Government: managing the process,” Committee on the Institutional Means for Assessment of Risks to Public Health, Commission on Life Sciences, National Academy Press, Washington, DC, USA, 1983, http://www.nap.edu/openbook.php?isbn=0309033497.
- US Environmental Protection Agency, “Framework for cumulative risk assessment,” EPA/630/P-02/001F, Risk Assessment Forum, Washington, DC, USA, 2003, http://www.epa.gov/raf/publications/framework-cra.htm, http://www.epa.gov/raf/publications/pdfs/frmwrk_cum_risk_assmnt.pdf.
- Agency for Toxic Substances and Disease Registry, “Interaction profiles for toxic substances,” Atlanta, GA, USA, 2011, http://www.atsdr.cdc.gov/interactionprofiles/index.asp.
- World Health Organization, “Risk assessment of combined exposure to multiple chemicals,” (Draft report) WHO/IPCS Framework, Geneva, Switzerland, 2009, http://www.who.int/ipcs/methods/harmonization/areas/aggregate/en/, http://www.who.int/ipcs/methods/harmonization/areas/combinedexposure.pdf.
- M. E. Meek, A. R. Boobis, K. M. Crofton, G. Heinemeyer, M. Van Raaij, and C. Vickers, “Risk assessment of combined exposure to multiple chemicals: a WHO/IPCS framework,” Regulatory Toxicology and Pharmacology, vol. 60, no. 2, supplement 1, pp. S1–S14, 2011.
- California Environmental Protection Agency, “Cumulative impacts: building a scientific foundation,” OEHHA, Sacramento, CA, USA, 2010, http://oehha.ca.gov/ej/pdf/CIReport123110.pdf.
- National Research Council, “Phthalates and cumulative risk assessment: the tasks ahead,” Committee on the Health Risks of Phthalates, Board on Environmental Studies and Toxicology, Division on Earth and Life Sciences, The National Academies Press, Washington, DC, USA, 2008, http://books.nap.edu/catalog.php?record_id=12528.
- National Research Council, “Science and decisions: advancing risk assessment,” Committee on Improving Risk Analysis Approaches Used by the U.S. EPA, Board on Environmental Studies and Toxicology, Division on Earth and Life Studies, The National Academies Press, Washington, DC, 2009,http://www.nap.edu/catalog.php?record_id=12209.
- European Commission, “No Miracle (NOvel Methods for Integrated Risk Assessment of CumuLative stressors in Europe),” 2009, http://nomiracle.jrc.ec.europa.eu/default.aspx.
- US Environmental Protection Agency, “Concepts, methods and data sources for cumulative health risk assessment of multiple chemicals, exposures and effects: a resource document,” Final Report, EPA/600/R-06/013F, Office of Research and Development, National Center for Environmental Assessment, Cincinnati, OH, USA, 2007, http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=190187.
- National Institute for Occupational Safety and Health, “Mixed exposures research agenda—a report by the NORA mixed exposures team,” National Occupational Research Agenda (NORA), DHHS (NIOSH) no. 2005-106, 2004, http://www.cdc.gov/niosh/nora/, http://www.cdc.gov/niosh/nora/pastnorapubs.html.
- US Environmental Protection Agency, “Risk assessment guidance for Superfund, vol. 1, human health evaluation manual (part A),” Interim Final (RAGS Part A), EPA/540/1-89/002, Office of Emergency and Remedial Response, Washington, DC, USA, 1989, http://www.epa.gov/oswer/riskassessment/ragsa/index.htm.
- R. T. Kay, T. L. Arnold, W. F. Cannon, D. Graham, E. Morton, and R. Bienert, “Concentrations of polynuclear aromatic hydrocarbons and inorganic constituents in ambient surface soils, Chicago, Illinois: 2001-02,” US Geological Survey Water-Resources Investigations Report 03-4105, USGS, Urbana, IL, USA, Chicago Department of Environment, Chicago, IL, USA, 2003, http://il.water.usgs.gov/pubs/wrir03_4105.pdf.
- Massachusetts Department of Environmental Protection, “Background levels of polycyclic aromatic hydrocarbons and metals in soil,” Technical Update, Office of Research & Standards, Boston, MA, USA, 2002, http://www.mass.gov/dep/cleanup/laws/backtu.pdf.
- Texas Commission on Environmental Quality, “Texas risk reduction program (TRRP),” Austin, TX, USA, 2012, http://www.tceq.texas.gov/remediation/trrp/trrp.html.
- P. C. Johnson and R. A. Ettinger, “Heuristic model for predicting the intrusion rate of contaminant vapors into buildings,” Environmental Science and Technology, vol. 25, no. 8, pp. 1445–1452, 1991.
- US Environmental Protection Agency, “Integrated risk information system (IRIS),” Office of Research and Development, NCEA, Washington, DC, USA, 2012, http://www.epa.gov/iris.
- US Environmental Protection Agency, “Provisional peer reviewed toxicity values for Superfund (PPRTV),” PPRTV Assessments Electronic Library, Office of Superfund Remediation and Technology Innovation, 2012, http://hhpprtv.ornl.gov/.
- M. M. Mumtaz, K. A. Poirier, and J. T. Colman, “Risk assessment for chemical mixtures: fine-tuning the hazard index approach,” Journal of Clean Technology, Environmental Toxicology and Occupational Medicine, vol. 6, no. 2, pp. 189–204, 1997.
- US Environmental Protection Agency, “Guidance on cumulative risk assessment of pesticide chemicals that have a common mechanism of toxicity,” Office of Pesticide Programs, Washington, DC, USA, 2002, http://www.epa.gov/oppfead1/trac/science/cumulative_guidance.pdf.
- US Environmental Protection Agency, “Organophosphorus cumulative risk assessment—2006 Update,” Office of Pesticide Programs, Washington, DC, USA, http://www.epa.gov/pesticides/cumulative/2006-op/op_cra_main.pdf.
- US Environmental Protection Agency, “Developing relative potency factors for pesticide mixtures: biostatistical analyses of joint dose-response,” EPA/ 600/R-03/052, National Center for Environmental Assessment, Cincinnati, OH, USA, in collaboration with US Food and Drug Administration, National Center for Toxicological Research, Jefferson, AR, USA, 2003, http://oaspub.epa.gov/eims/eimscomm.getfile?p_download_id=427398, http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=66273.
- J. J. Chen, Y. J. Chen, G. Rice et al., “Using dose addition to estimate cumulative risks from exposures to multiple chemicals,” Regulatory Toxicology and Pharmacology, vol. 34, no. 1, pp. 35–41, 2001.
- M. T. Klein, G. Hou, R. J. Quann et al., “BioMOL: a computer-assisted biological modeling tool for complex chemical mixtures and biological processes at the molecular level,” Environmental Health Perspectives, vol. 110, supplement 6, pp. 1025–1029, 2002.
- V. J. Feron, F. R. Cassee, J. P. Groten, P. W. van Vliet, and J. A. van Zorge, “International issues on human health effects of exposure to chemical mixtures,” Environmental Health Perspectives, vol. 110, supplement 6, pp. 893–899, 2002.
- D. Jonker, A. P. Freidig, J. P. Groten et al., “Safety evaluation of chemical mixtures and combinations of chemical and non-chemical stressors,” Reviews on Environmental Health, vol. 19, no. 2, pp. 83–139, 2004.
- National Institute of Environmental Health Sciences, “Your environment—your health,” 2012, http://www.niehs.nih.gov/.
- Health Canada, “Environmental and workplace health,” 2012, http://www.hc-sc.gc.ca/ewh-semt/air/out-ext/research-recherche-eng.php.
- K. Krishnan, S. Haddad, M. Béliveau, and R. Tardif, “Physiological modeling and extrapolation of pharmacokinetic interactions from binary to more complex chemical mixtures,” Environmental Health Perspectives, vol. 110, supplement 6, pp. 989–994, 2002.
- California Environmental Protection Agency, “Rulemaking identification of particulate emissions from diesel-fueled engines as a toxic air contaminant,” Air Resources Board, Sacramento, CA, USA, 1998, http://www.arb.ca.gov/regact/diesltac/diesltac.htm.
- Y. A. Sanchez, K. Deener, E. C. Hubal, C. Knowlton, D. Reif, and D. Segal, “Research needs for community-based risk assessment: findings from a multi-disciplinary workshop,” Journal of Exposure Science and Environmental Epidemiology, vol. 20, no. 2, pp. 186–195, 2010.
- T. M. Barzyk, K. C. Conlon, T. Chahine, D. M. Hammond, V. G. Zartarian, and B. D. Schultz, “Tools available to communities for conducting cumulative exposure and risk assessments,” Journal of Exposure Science and Environmental Epidemiology, vol. 20, no. 4, pp. 371–384, 2010.
- M. Medina-Vera, J. M. Van Emon, L. J. Melnyk, K. D. Bradham, S. L. Harper, and J. N. Morgan, “An overview of measurement method tools available to communities for conducting exposure and cumulative risk assessments,” Journal of Exposure Science and Environmental Epidemiology, vol. 20, no. 4, pp. 359–370, 2010.
- P. Ruiz, M. Mumtaz, J. Osterloh, J. Fisher, and B. A. Fowler, “Interpreting NHANES biomonitoring data, cadmium,” Toxicology Letters, vol. 198, no. 1, pp. 44–48, 2010.
- Centers for Disease Control and Prevention, “National health and nutrition examination study (NHANES),” 2012, http://www.cdc.gov/nchs/nhanes.htm.
- US Environmental Protection Agency, “Health and environmental research online (HERO),” National Center for Environmental Assessment, Research Triangle Park, NC, USA, 2012, http://cfpub.epa.gov/ncea/hero/.
- Toxicology Excellence for Risk Assessment, “International toxicity estimates for risk assessment (ITER),” Cincinnati, OH, USA, 2012, http://www.tera.org/iter/.
- D. Payne-Sturges, J. Cohen, R. Castorina, D. A. Axelrod, and T. J. Woodruff, “Evaluating cumulative organophosphorus pesticide body burden of children: a national case study,” Environmental Science & Technology, vol. 43, no. 20, pp. 7924–7930, 2009.
- A. S. Lewis, S. N. Sax, S. C. Wason, and S. L. Campleman, “Non-chemical stressors and cumulative risk assessment: an overview of current initiatives and potential air pollutant interactions,” International Journal of Environmental Research and Public Health, vol. 8, no. 6, pp. 2020–2073, 2011.
- T. Chahine, B. D. Schultz, V. G. Zartarian, J. Xue, S. V. Subramanian, and J. Levy, “Modeling joint exposures and health outcomes for cumulative risk assessment: the case of radon and smoking,” International Journal of Environmental Research and Public Health, vol. 8, no. 9, pp. 3688–3711, 2011.
- D. T. Lobdell, V. Isakov, L. Baxter, J. S. Touma, M. B. Smuts, and H. Özkaynak, “Feasibility of assessing public health impacts of air pollution reduction programs on a local scale: New Haven case study,” Environmental Health Perspectives, vol. 119, no. 4, pp. 487–493, 2011.
- K. M. Ellickson, S. M. Sevcik, S. Burman, S. Pak, F. Kohlasch, and G. C. Pratt, “Cumulative risk assessment and environmental equity in air permitting: interpretation, methods, community participation and implementation of a unique statute,” International Journal of Environmental Research and Public Health, vol. 8, no. 11, pp. 4140–4159, 2012.
- K. Sexton and S. H. Linder, “Cumulative risk assessment for combined health effects from chemical and nonchemical stressors,” American Journal of Public Health, vol. 101, supplement 1, pp. S81–S88, 2011.
- J. J. Salinas, M. Shah, B. Abdelbary, J. L. Gay, and K. Sexton, “Application of a novel method for assessing cumulative risk burden by county,” International Journal of Environmental Research and Public Health, vol. 9, no. 5, pp. 1820–1835, 2012.
Copyright © 2013 Margaret M. MacDonell et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.