A Styrene-<i>alt</i>-Maleic Acid Copolymer Is an Effective Inhibitor of R5 and X4 Human Immunodeficiency Virus Type 1 Infection

Pirrone, Vanessa; Passic, Shendra; Wigdahl, Brian; Rando, Robert F.; Labib, Mohamed; Krebs, Fred C.

doi:https://doi.org/10.1155/2010/548749

BioMed Research International

On this page

Abstract Introduction Materials and Methods Results Discussion Conclusions Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2010 | Article ID 548749 | https://doi.org/10.1155/2010/548749

A Styrene-alt-Maleic Acid Copolymer Is an Effective Inhibitor of R5 and X4 Human Immunodeficiency Virus Type 1 Infection

Vanessa Pirrone,¹Shendra Passic,¹Brian Wigdahl,¹Robert F. Rando,²Mohamed Labib,²and Fred C. Krebs¹

Academic Editor: Robert Blumenthal

Received07 Jan 2010

Accepted11 Mar 2010

Published31 May 2010

Abstract

An alternating copolymer of styrene and maleic acid (alt-PSMA) differs from other polyanionic antiviral agents in that the negative charges of alt-PSMA are provided by carboxylic acid groups instead of sulfate or sulfonate moieties. We hypothesized that alt-PSMA would have activity against human immunodeficiency virus type 1 (HIV-1) comparable to other polyanions, such as the related compound, poly(sodium 4-styrene sulfonate) (PSS). In assays using cell lines and primary immune cells, alt-PSMA was characterized by low cytotoxicity and effective inhibition of infection by HIV-1 BaL and IIIB as well as clinical isolates of subtypes A, B, and C. In mechanism of action assays, in which each compound was added to cells and subsequently removed prior to HIV-1 infection (“washout” assay), alt-PSMA caused no enhancement of infection, while PSS washout increased infection 70% above control levels. These studies demonstrate that alt-PSMA is an effective HIV-1 inhibitor with properties that warrant further investigation.

1. Introduction

Polyanionic molecules were shown to be effective inhibitors of the human immunodeficiency virus type 1 (HIV-1), not long after the recognition of HIV-1 as the viral agent responsible for the acquired immune deficiency syndrome (AIDS). Dextran sulfate and heparin were two of the first sulfated compounds identified as compounds with antiviral activities against HIV-1 [1, 2]. Polyanionic molecules are generally considered to be entry inhibitors, with the activities of these compounds stemming from their abilities to inhibit virus adsorption to the surface of the target cell membrane [3]. The main mechanism of action associated with this family of compounds involves electrostatic interactions with HIV-1 gp120. Interactions between these negatively charged molecules and the positively charged V3 loop of gp120 interfere with receptor-mediated binding and inhibit host cell infection [4–6]. It was also established that their activities were dependent on the level of sulfation, with two sulfate groups per monosaccaride required to achieve maximal anti-HIV-1 activity [3]. These studies demonstrated that the negative charges on these polymeric molecules play important roles in inhibition of HIV-1 infection.

A number of polyanionic compounds and their formulations have been developed as inhibitors of HIV-1 binding and entry [7, 8]. Among these are PRO 2000 (naphthalene 2-sulfonate polymer) [9], PSS [poly(sodium 4-styrene sulfonate] [10], HPMCT (hydroxypropyl methylcellulose trimellitate) [11], and CAP (cellulose acetate phthalate) [4]. These compounds have been evaluated in preclinical studies or have been advanced into clinical trials for their use in microbicides, which are products designed to reduce or eliminate the risk of HIV-1 sexual transmission [7, 12]. Although they have distinct chemical formulae, these compounds have three common structural and chemical characteristics: (i) they are polymeric in nature (high molecular weight); (ii) they possess a level of hydrophobicity associated with incorporated ring structures; (iii) they bear multiple anionic charges along the length of the molecule. These characteristics (particularly the anionic charges) presumably play roles in defining the biological activities of these molecules, specifically their capacities to inhibit infection by HIV-1 and other enveloped viruses and their effects on cellular viability.

We hypothesized that a molecule with similar structural characteristics would have comparable activity against HIV-1. Poly(styrene-alt-maleic acid) is an alternating copolymer (alt-PSMA) comprised of styrene and maleic acid at a ratio. Like PRO 2000, PSS, HPMCT, and CAP, alt-PSMA is characterized by the presence of multiple anionic charges and hydrophobic, aromatic rings (Figure 1(a)). However, in alt-PSMA (120,000 average MW), the hydrophobic phenyl side group, contributed by the styrene unit, is directly attached to the hydrocarbon backbone. In addition, unlike the sulfated or sulfonated molecules PSS (Figure 1(b)), PRO 2000, or carrageenan [13], alt-PSMA derives its anionic charge from two free carboxyl groups (pKa 1.9 and 6.0) of maleic acid, which are also directly attached to the hydrocarbon backbone instead of the aromatic ring.

(a)

(b)

The properties of alt-PSMA prompted an exploration of the antiviral potential of this polyanionic molecule. In vitro assays of compound cytotoxicity and antiviral activity demonstrated that alt-PSMA combines low cytotoxicity with considerable antiviral activity against R5 and X4 viruses of multiple subtypes. Furthermore, in washout assays typically used to explore compound mechanism of action and examine compound residence time after removal from target cells, alt-PSMA had no impact on subsequent HIV-1 infection, whereas PSS caused increased levels of infection. These experiments compare and contrast alt-PSMA with other polyanionic compounds and provide the foundation for the further development of alt-PSMA as an inhibitor of HIV-1 infection.

2. Materials and Methods

2.1. Compounds

alt-PSMA (catalog no. 662631, lot 71K1378) and dextran sulfate (DS; Dextralip 50, catalog no. D8787, lot 71K1378) were purchased from Sigma-Aldrich (St. Louis, MO). Polystyrene sulfonate (PSS, lot DFS-1951) was kindly provided by National Starch and Chemical Company (Chattanooga, TN).

2.2. Cell Line Maintenance and Primary Cell Isolation

P4-R5 MAGI cells (NIH AIDS Research and Reference Reagent Program # 3580) were maintained in Dulbecco’s Modified Eagle’s Media (DMEM) supplemented with 10% fetal bovine serum (FBS), sodium bicarbonate (0.05%), antibiotics (penicillin, streptomycin, and kanamycin at 40 g/mL each), and puromycin (1 g/mL) [14]. TZM-bl cells (NIH AIDS Research and Reference Reagent Program # 8129) were maintained in Dulbecco’s Modified Eagle’s Media (DMEM) supplemented with 10% fetal bovine serum (FBS), sodium bicarbonate (0.05%), and antibiotics (penicillin, streptomycin, and kanamycin at 40 g/mL each) [15]. Human peripheral blood mononuclear cells (PBMCs) were isolated from whole blood (Biological Specialty Corp., Colmar, PA) using Ficoll-hypaque (Amersham Biosciences, Piscataway, NJ) density gradient centrifugation and were subsequently cultured in RPMI supplemented with 10% FBS, sodium bicarbonate (0.05%), antibiotics (penicillin and streptomycin at 90 g/mL each), PHA-P (Sigma-Aldrich; 5 g/mL), and IL-2 (NIH AIDS Research and Reference Reagent Program # 11697; 20 U/mL) [16]. After 48 h, the PBMCs were washed and incubated for an additional 24 h in the absence of PHA-P prior to infection.

2.3. Propagation of Primary HIV-1 Clinical Isolates

PBMCs ( cells stimulated as described above) were treated with polybrene (Sigma-Aldrich) at a concentration of 2 g/mL for 30 min. The cells were then pelleted and 1 mL of virus stock was used to resuspend the cells. Cells and virus were incubated at with 5% CO₂ for 2 h with gentle mixing every 30 min. After the incubation interval, the cells were washed two times with PBMC media. The cells were again pelleted and subjected to a second incubation with virus (1 mL). Following the second incubation, the cells were again washed and resuspended in PBMC media (10 mL) supplemented with 2 g/mL of polybrene (co-culture media), and the volume was equally divided into two T-75 flasks. An additional 25 mL of media was added to each flask. The cultures were assessed for p24 content on days 3 through 7 postinfection, and cells were split whenever the p24 level exceeded 100 ng/mL. On day 7, the cell suspension was clarified by centrifugation twice, initially at 1400 rpm for 10 min at , followed by centrifugation of the supernatant at 2400 rpm for 10 min at . After clarification, the viral suspension was aliquoted and stored at until use. The clinical isolates (all R5 coreceptor phenotypes obtained from the NIH AIDS Research and Reference Reagent Program) included two strains of subtype A (KNH1144, catalog no. 112460; KNH1207, catalog no. 11247) [17], one strain of subtype B (US1, catalog no. 7686) [18–20], and two subtype C strains (93MW965, catalog no. 2913; SM145, catalog no. 7697) [18–20].

2.4. Assessing alt-PSMA Inhibition of Infection by Cell-Free HIV-1

P4-R5 MAGI cells were cultured at a density of cells/well in a 96-well plate 18 h prior to infection. Cells were incubated for 2 h with HIV-1 laboratory strains BaL or IIIB (Advanced Biotechnologies Inc., Columbia, MD) in the absence or presence of alt-PSMA or DS. After 2 h, cells were washed, cultured for an additional 46 h, and subsequently assayed for HIV-1 infection using the Galacto-Star -Galactosidase Reporter Gene Assay System for Mammalian Cells (Applied Biosystems, Bedford, MA). In a variation of this experiment, cell-free virus was preincubated with alt-PSMA or DS for 10 min at , followed by a 1 : 100 dilution of the virus : compound mixture in complete RPMI media and addition to P4-R5 MAGI cells.

2.5. Assessing alt-PSMA Inhibition of Infection by Cell-Associated HIV-1

alt-PSMA was evaluated for its ability to inhibit infection by HIV-1 IIIB-infected Sup-T1 T lymphocytes. Compounds were incubated with HIV-1 IIIB-infected SupT1 cells for 10 min followed by a 1 : 10 dilution in RPMI media and incubation for 2 h with P4-R5 MAGI cells at a density of cells/well in a 12-well plate. In a variation of this experiment, mitomycin-treated HIV-1 IIIB-infected SupT1 cells (0.2 mg/mL mitomycin C for 30 min) were incubated with compounds for 2 min followed by a 1 : 10 dilution in RPMI media and incubation for 2 h with P4-R5 MAGI cells. Cells were then washed and incubated for 46 h and assessed for -galactosidase production as described above.

2.6. Evaluating In Vitro Cytotoxicity of alt-PSMA

P4-R5 MAGI cells were seeded at a density of cells/well in a 96-well plate approximately 18 h prior to experiment. Cells were then exposed to the indicated concentrations of alt-PSMA or DS for 10 min, 2 h, or 24 h. Following the exposure period, cells were washed and assessed for viability using a 3-(4,5-dimethylthiazol-2-yl)-2, 5-diphenyltetrazolium bromide (MTT) assay of viability [21, 22]. For PBMCs, cells were seeded at a density of cells/well in a 96-well plate. Cells were then exposed to the indicated concentrations of alt-PSMA, PSS, or DS for 6 h (to mirror the duration of compound exposure in the antiviral assay). Following the exposure period, cells were washed and assessed for viability using a 3-(4,5-dimethylthiazol-2-yl)-5-(3-carboxymethoxyphenyl)-2-(4-sulfophenyl)-2H-tetrazolium (MTS) assay of viability [23, 24]. CC₅₀ values (concentration at which exposure to the compound resulted in a 50% decrease in cell viability relative to mock-treated cells) were subsequently calculated from the results.

2.7. Demonstrating HIV-1 Inhibition by alt-PSMA in Primary Human Immune Cells

Human PBMCs, stimulated as described above, were seeded at a density of cells/well in a 96-well plate. Cells were then incubated for 6 h with HIV-1 BaL or IIIB in the absence or presence of alt-PSMA, PSS, or DS. After 6 h, cells were washed and subsequently cultured for 3 days, at which time the cells were washed and supplied with new media supplemented with IL-2. The cells were then incubated for an additional 3 days and subsequently assayed for HIV-1 production by determining the level of p24 core antigen in the supernatant using an HIV-1 p24 antigen ELISA assay (ZeptoMetrix, Buffalo, NY). Levels of infection were expressed relative to mock-treated, HIV-1-infected cells.

2.8. Demonstrating alt-PSMA Efficacy against Subtype A, B, and C Clinical Isolates

TZM-bl cells were cultured at a density of cells/well in a 96 well plate 18 h prior to infection. Cells were incubated for 2 h with HIV-1 clinical isolates in the absence or presence of alt-PSMA or PSS. After 2 h, cells were washed, cultured for an additional 46 h, and subsequently assayed for HIV-1 infection using the Galacto-Star -Galactosidase Reporter Gene Assay System for Mammalian Cells (Applied Biosystems, Bedford, MA).

2.9. Evaluating the Effects of Compound Washout on HIV-1 Infection

IC₉₀ concentrations for inhibition of cell-free HIV-1 infection by alt-PSMA, PSS, or DS were determined using procedures as described above, except that the infection duration was shortened to 1 h. In the washout assay, P4-R5 MAGI cells were incubated with IC₉₀ concentrations of compound for 1 h. Cells were then washed extensively, provided with new media without compound, and then infected at 2 h or 4 h after compound removal with HIV-1 BaL or IIIB for 1 h at . As a control, compound, virus, and cells were incubated together for 1 h. Following infection, cells were washed again, provided with media without compound, and assayed 46 h postinfection as described above.

2.10. Data Analyses

Data for all experiments are shown as mean values and calculated standard deviations from two independent assays in which each concentration was examined in triplicate. Calculations of IC₅₀/IC₉₀ concentrations (concentrations that resulted in 50% or 90% decreases in infection relative to mock-treated, HIV-1-infected cells) and CC₅₀ concentrations (concentrations that resulted in a 50% reduction in cell viability relative to mock-exposed cells) were calculated using the Forecast function of Microsoft Excel.

3. Results

3.1. alt-PSMA Is an Effective Inhibitor of Infection by HIV-1 BaL

In the context of microbicide development, the importance of inhibiting infection by R5 viruses is underscored by the prevalence of R5 variants early in infection and the presumed preference for R5 viruses during the process of HIV-1 transmission [8, 25]. Experiments were conducted to determine the ability of alt-PSMA to inhibit infection by HIV-1 BaL, which has an R5 phenotype (Figure 2(a)). alt-PSMA was an effective inhibitor of infection, achieving complete inhibition at 0.1 mg/mL and above and an IC₅₀ value of 0.009 mg/mL during 2 h incubation with virus and P4-R5 MAGI cells. Similar experiments, in which cell-free HIV-1 BaL and compounds were incubated for 10 min prior to introduction to the target P4-R5 MAGI cells, were conducted to determine the direct effect of each compound on virus infectivity. A 10-min preincubation of cell-free virus with alt-PSMA followed by 1 : 100 dilution of the virus : compound mixture prior to addition to cells had no impact on the ability of alt-PSMA to inhibit infection by HIV-1 BaL. These and subsequent experiments also included the control compound DS, which was used as a prototypical polyanionic compound with high antiviral activity and low cytotoxicity. When compared to DS (IC₅₀ value of 0.03 mg/mL), alt-PSMA appeared to be modestly more effective against HIV-1 BaL. Preincubation of DS with cell-free virus had no impact on its antiviral activity.

(a)

(b)

Figure 2

alt-PSMA inhibits infection by HIV-1 strains BaL and IIIB. P4-R5 MAGI cells were incubated for 2 h with HIV-1 strains (a) BaL or (B) IIIB in the absence or presence of alt-PSMA or DS (as described in Section 2). For the preincubation portion of the experiment, compound and virus were first incubated for 10 min prior to dilution and addition to cells. Infectivity remaining was expressed relative to mock-treated, HIV-1-infected cells and graphed against the compound concentration achieved during the 2-h incubation. Each concentration was examined in triplicate in two independent assays.

3.2. Preincubation of alt-PSMA with HIV-1 IIIB Enhances Its Antiviral Activity

Similar experiments were performed using the X4 HIV-1 strain IIIB (Figure 2(b)). Relative to its activity against HIV-1 BaL, alt-PSMA had slightly higher inhibitory activity against HIV-1 IIIB (IC₅₀ value of 0.005 mg/mL). Similar differences in coreceptor usage-dependent activity have been demonstrated using other polyanions, including PSS [26] and HPMCT [11]. Interestingly, a 10-min preincubation of alt-PSMA with cell-free HIV-1 IIIB resulted in an approximately 6-fold increase in antiviral activity (IC₅₀ = 0.0008 mg/mL) relative to its activity in the absence of preincubation (IC₅₀ = 0.005 mg/mL). In contrast, the antiviral activity of DS (IC₅₀ = 0.0006 mg/mL) was unaffected by preincubation (IC₅₀ = 0.0005 mg/mL).

3.3. alt-PSMA Inhibits Infection by HIV-1-Infected T Lymphocytes

Since inhibitors and potential microbicide agents must also be effective against cell-associated infectivity, experiments were conducted to demonstrate the ability of alt-PSMA to inhibit infection by HIV-1-infected cells (Figure 3). Infection of P4-R5 MAGI target cells by HIV-1 IIIB-infected Sup-T1 T lymphocytes (Figure 3(a)) was inhibited in a dose-dependent manner by alt-PSMA (IC₅₀ = 0.03 mg/mL). Inhibition by DS was comparable (IC₅₀ = 0.01 mg/mL). Additionally, to look solely at the ability of alt-PSMA to inhibit viral transfer from infected cells [27], mitomycin C-treated HIV-1 IIIB-infected SupT1 cells were used as the source of infectivity. alt-PSMA again inhibited infection in a dose-dependent manner (IC₅₀ = 0.12 mg/mL) comparable to that of DS (Figure 3(b)).

(a)

(b)

Figure 3

alt-PSMA inhibits infection of P4-R5 MAGI cells by HIV-1-infected cells. (a) HIV-1 strain IIIB-infected Sup-T1 T cells were incubated for 10 min in the absence or presence of alt-PSMA or DS prior to 1 : 10 dilution and incubation with P4-R5 MAGI cells for 2 h (as described in Section 2). (b) Mitomycin C-treated HIV-1 strain IIIB-infected Sup-T1 T cells were incubated for 2 min in the absence or presence of alt-PSMA or DS prior to 1 : 10 dilution and incubation with P4-R5 MAGI cells for 2 h (as described in Section 2). Infectivity remaining was expressed relative to mock-treated, HIV-1-infected cells and graphed against the compound concentration achieved during the 2-h incubation with P4-R5 MAGI cells. Each concentration was examined in triplicate in two independent assays.

3.4. alt-PSMA Is Characterized by Low In Vitro Cytotoxicity

In vitro cytotoxicity experiments were performed to determine the effect of alt-PSMA on cellular viability, to evaluate the potential safety of alt-PSMA, and to demonstrate that compound cytotoxicity had no impact on the results of the antiviral assays. Exposures to alt-PSMA for 10 min or 2 h had no effect on P4-R5 MAGI cell viability at concentrations at or below 3.16 mg/mL (Figure 4(a)). After a 24-h exposure to alt-PSMA, P4-R5 MAGI cell viability was reduced only at the highest concentration (62% viability at 3.16 mg/mL). Similarly, cell viability was reduced by exposure to DS only after a 24-h exposure at the highest concentration examined (Figure 4(b)). These results indicate that (i) the demonstrated antiviral efficacy of alt-PSMA was not affected by changes in target cell viability, since alt-PSMA exposure was not cytotoxic at the concentrations and exposure durations used in the antiviral experiments; (ii) alt-PSMA may have a considerable margin of safety, with an in vitro therapeutic index (CC₅₀/IC₅₀) in excess of 350.

(a)

(b)

3.5. HIV-1 Infection of Primary Human PBMCs Is Inhibited by alt-PSMA

In vitro experiments were also used to demonstrate the ability of alt-PSMA to inhibit HIV-1 infection of primary human immune cell populations (Figure 5). Secondarily, these experiments provided a comparison between alt-PSMA and PSS, which are similar in that both compounds have repeating anionic charges and aromatic rings along their backbones. In experiments in which PBMCs were infected by HIV-1 BaL (Figure 5(a)), inhibition of infection by alt-PSMA (IC₅₀ = 0.02 mg/mL) was concentration-dependent and comparable to levels of inhibition achieved in the presence of PSS (IC₅₀ = 0.04 mg/mL) or DS (IC₅₀ = 0.04 mg/mL). Similar experiments using HIV-1 IIIB (Figure 5(b)) demonstrated that alt-PSMA (IC₅₀ = 0.003 mg/mL), PSS (IC₅₀ = 0.002 mg/mL), and DS (IC₅₀ = 0.005 mg/mL) again had similar capacities for inhibiting HIV-1 infection. Again, all three compounds were approximately one log more effective against HIV-1 IIIB compared to their activities against BaL. In assays of in vitro cytotoxicity using PBMCs, all three compounds were noncytotoxic (Figure 5(c)) at concentrations shown to inhibit HIV-1 infection, with CC₅₀ values of 9.1 mg/mL for alt-PSMA and in excess of 10 mg/mL for PSS and DS. These results indicated that alt-PSMA and PSS, which have similar structural features (anionic charge, aromatic rings), have similar capacities for inhibiting HIV-1 infection. Furthermore, these results demonstrated, at least in the context of this experimental design, that the identity (carboxylic or sulfonic acid) or position (backbone or aromatic ring) of the anionic charge had no apparent effect on antiviral activity and a negligible effect on in vitro cytotoxicity.

(a)

(b)

(c)

Figure 5

alt-PSMA effectively inhibits infection of primary human immune cells by HIV-1 strains BaL and IIIB. PBMCs were incubated for 6 h with HIV-1 strains (a) BaL or (b) IIIB in the absence or presence of alt-PSMA, PSS, or DS (as described in Section 2). Infectivity remaining is expressed relative to mock-treated, HIV-1-infected cells. Each concentration was examined in triplicate in two independent assays. (c) PBMCs were exposed (or mock exposed) to the indicated concentrations of alt-PSMA, PSS, or DS for 6 h. Following the exposure period, cells were washed and assessed for viability using an MTS assay (as described in Section 2). Results illustrated are from two independent assays in which each concentration was examined in triplicate.

3.6. HIV-1 Clinical Isolates of Various Subtypes Are Inhibited by alt-PSMA

To be effective in the context of the global HIV/AIDS epidemic, inhibitors of HIV-1 infection must have antiviral activity against the range of regional HIV-1 subtypes found throughout the world. To this end, alt-PSMA was evaluated for its activities against clinical isolates from subtypes A, B, and C. These isolates included two subtype A strains (KNH1144 and KNH1207), one subtype B virus (US1), and two strains from subtype C (93MW965 and SM145). In antiviral assays using TZM-bl indicator cells, alt-PSMA was an effective inhibitor of all viral subtypes examined, with IC₅₀ concentrations ranging from 0.003 to 0.009 mg/mL (Table 1). Furthermore, alt-PSMA appeared to be more effective against the subtype A and B isolates compared to PSS; PSS IC₅₀ concentrations were approximately 4- to 7-fold higher against these three isolates relative to the IC₅₀ concentrations of alt-PSMA.

3.7. alt-PSMA Washout Does Not Affect HIV-1 Infection, While PSS Washout Enhances Infection

A washout assay, in which antiviral compounds are removed from cells or tissues prior to HIV-1 infection, can be used to explore compound mechanism of action or, in the case of an agent that acts on or in the host cell, examine compound residence time after removal from the culture medium. Preparative assays used to establish IC₅₀ and IC₉₀ concentrations for alt-PSMA, PSS, and DS during a 1-h infection demonstrated that all three compounds were effective HIV-1 inhibitors, with IC₅₀ values against BaL of 0.009 mg/mL, 0.04 mg/mL, and 0.0058 mg/mL and IC₅₀ values against IIIB of 0.003 mg/mL, 0.01 mg/mL, and 0.00052 mg/mL for alt-PSMA, PSS, and DS, respectively. In the washout assays using IC₉₀ concentrations of each compound, co-incubation of compound, cells, and HIV-1 BaL (Figure 6(a)) or IIIB (Figure 6(b)) produced the expected results: approximately 90% inhibition of each virus was achieved. In contrast, removal of alt-PSMA and subsequent infection by either virus at 2 h or 4 h after compound removal had no effect on HIV-1 infection (compared to cells infected in the absence of compound). Since polyanionic compounds have been shown to inhibit HIV-1 infection by interactions with gp120 [5, 6, 28–33], abrogation of the inhibitory effects of PSMA and DS by compound washout was anticipated, since the compounds were not available to bind to gp120 and thus inhibit viral binding and entry.

(a)

(b)

Figure 6

alt-PSMA washout does not increase HIV-1 BaL or IIIB infection. P4-R5 MAGI cells were incubated for 1 h with predetermined IC₉₀ concentrations of alt-PSMA (0.03 mg/mL, BaL; 0.02 mg/mL, IIIB), PSS (0.1 mg/mL, BaL; 2.29 mg/mL, IIIB), or DS (0.031 mg/mL, BaL; 0.0028 mg/mL, IIIB). Following exposure, cells were washed thoroughly, provided with new media for 2 h or 4 h, and subsequently infected with HIV-1 strains BaL (a) or IIIB (b) for 1 h. As a control, compound, virus, and cells were simultaneously incubated for 1 h (labeled “S” on each graph). Levels of infection are expressed relative to mock-treated, HIV-1 infected cells. Each concentration was examined in triplicate in two independent assays. The asterisk (*) indicates a significant difference compared to infected, mock-exposed control cells ().

Unexpectedly, PSS washout caused increases in HIV-1 infection dependent on time of infection and coreceptor usage by the infecting virus. HIV-1 BaL infection after PSS washout was increased by approximately 39% over controls at the 2-h time point but unaffected by 4-h postwashout. In contrast, HIV-1 IIIB infection was persistently enhanced at both 2 h (55% increase) and 4 h (67% increase) after PSS removal. These results suggest functional interactions between PSS and host cells that serve to enhance HIV-1 infection and/or replication. In addition, these experiments provide a clear functional distinction between alt-PSMA and PSS despite their structural similarities.

4. Discussion

These in vitro studies confirmed the opening hypothesis, which stated that alt-PSMA would be an effective inhibitor of HIV-1 infection as a consequence of its gross structural similarities to the anti-HIV-1 compound PSS [10]. alt-PSMA was shown to have equipotent activities against laboratory-adapted R5 and X4 HIV-1 strains, activity against cell-free and cell-associated virus, demonstrable antiviral activity in the context of HIV-1 infection of primary human immune cells, and activity against a range of HIV-1 subtypes. The activity of alt-PSMA against HIV-1, which we first reported at the 2005 International Meeting of the Institute of Human Virology [34], was recently confirmed in structure-activity studies of poly(styrene-alt-maleic anhydride) derivatives [35]. Presumably, alt-PSMA inhibits HIV-1 attachment and entry in a manner comparable to PSS [26]. The specific mechanism of action of alt-PSMA, like polyanionic dextran sulfate, likely involves electrostatic interactions with the V3 loop and coreceptor binding site [6].

Demonstrated differences, however, between alt-PSMA and other polyanionic compounds, particularly in the compound washout experiments, suggested that subtle structural differences between these compounds may be functionally important under specific conditions. Despite general similarities between members of the polyanionic compound family, alt-PSMA is distinguished from PSS and other polyanionic molecules by three charge-related characteristics. First, the anionic charges of alt-PSMA are provided by carboxylic groups instead of the sulfonate side groups found on PSS (charge identity). Second, the negatively charged carboxylic groups of alt-PSMA are located on the backbone and are separated from the hydrophobic phenyl side chains of the styrene moiety (charge location). In contrast, the negatively charged sulfonate group of PSS is itself a side group of the styrene repeat unit (Figure 1). Similarly, the sulfonate group of PRO 2000, which provides its anionic charge, is a side chain of the naphthalene ring that is part of the polymeric structure [9]. Third, alt-PSMA includes two anionic charges per repeat instead of the single anionic side group found in each repeat of PSS (charge density). As previously shown, differences in the number of charges per repeat unit, which affect the linear charge density of the molecule, can impact antiviral activity [3]. These distinctions in charge identity, location, and density may be important factors in differentiating alt-PSMA from other polyanionic molecules. The present studies provide the starting point for future structure-function experiments that will examine the effects of charge identity, placement, and density on compound activity and will provide new insights into the mechanisms that underlie the efficacy and safety of this family of compounds.

Results of these investigations also indicated that variations in antiviral efficacy were evident not only as a consequence of differences in inhibitors, but also because of differences between infecting viruses and between target cell types. For example, the differential effect of virus-compound preincubation (Figures 2(a) and 2(b)) on antiviral activity indicated both compound- and virus-specific differences in inhibitor retention on cell-free virions. Specifically, enhanced antiviral activity against HIV-1 IIIB but not BaL following alt-PSMA preincubation with virus may be attributed to the more basic nature of X4 gp120 molecules [6], which would theoretically promote increased compound retention on the virus surface. Furthermore, because of fundamental differences in backbone structure between alt-PSMA and DS, this mechanism of electrostatic retention of DS may not be possible. In addition, assays using PBMCs as target cells (Figure 5) demonstrated that alt-PSMA, PSS, and DS were more effective inhibitors of HIV-1 IIIB (X4) relative to their activities against the R5 strain BaL. Similar differences in antiviral activity were previously reported in assays involving the polyanions PSS [26] and HPMCT [11]. Differences in antiviral activity related to coreceptor usage could be attributed to the more basic nature of the X4 gp120 V3 region [6], which may promote a greater electrostatic affinity for anionic inhibitors and, thus, greater antiviral efficacy. However, this effect may be offset by other factors (such as host cell type), since alt-PSMA had equipotent activity against BaL and IIIB in experiments using P4-R5 MAGI cells as hosts for infection. In addition, differences in efficacy against viruses that differ in subtype but not in coreceptor usage (Table 1) suggested that other genotypic or phenotypic differences between viruses affect antiviral efficacy.

Assays involving compound washout have revealed a previously unknown activity of selected polyanionic molecules. The unexpected enhancement of HIV-1 infection by PSS suggested an activity that involves perturbation of the host cell, since, in this type of assay, only the target cells are exposed to significant concentrations of the compound. This activity appeared to compound-specific, since neither alt-PSMA nor DS produced this effect. In the context of the experimental design, two possible mechanisms of PSS action are suggested. First, low levels of PSS may be retained on the cell surface despite extensive cell washing. This explanation appears to be consistent with a previous report of enhanced HIV-1 infection at low concentrations of the polyanionic compound cellulose sulfate [36]. At low cellulose sulfate concentrations that are insufficient to provide antiviral activity, mechanisms that enhance HIV-1 infection might be unmasked. In a similar fashion, the removal of PSS from the media and the retention of small amounts of PSS on the cell surface may have caused enhanced HIV-1 infection. However, the mechanisms at work in these situations may be similar but not identical, since the magnitude of enhancement in the washout assays was considerably higher (Figure 6 and Pirrone and Krebs, manuscript in preparation) than increases in infection observed at low compound concentrations [36]. In either case, low concentrations of compound on the cell surface may effect changes on the cell surface that facilitate the binding and entry steps of the HIV-1 replication cycle.

Alternatively, enhanced HIV-1 infection may be the result of intracellular changes initiated by interactions between PSS and the cell surface. For example, incubation with PSS may activate signaling pathways that affect HIV-1 gene expression. Cellular activation mediated by transient exposure to PSS would persist for several hours and would not require the continued presence of PSS. Interestingly, there is precedence for this type of activity among polyanionic HIV-1 inhibitors. Previous studies have shown that activation of macrophages by -carrageenan, which is used to induce nonspecific inflammation in murine models of footpad edema and pleurisy, was dependent on toll-like receptor 4 (TLR4) and the downstream factor MyD88 [37]. Cellular signaling pathways, such as those activated by TLR stimulation, may stimulate HIV-1 gene expression through NF-B-mediated activation of the HIV-1 long terminal repeat (LTR) [38].

Enhancement of HIV-1 infection by select polyanionic compounds may be relevant to microbicide clinical trials, particularly to the failure of Ushercell (cellulose sulfate) in recent trials [39, 40]. Because the cervicovaginal environment is open, microbicide products can leak out over time. For example, in clinical trials involving Ushercell, compound leakage was reported by female users, and product leakage had a negative effect on product acceptability [41, 42]. In a scenario in which product leakage has sufficiently reduced the concentration of the active compound (i.e., cellulose sulfate) within the cervicovaginal environment, HIV-1 transmission may be enhanced rather than inhibited. This effect could explain the increased risk of HIV-1 infection associated with Ushercell use in one of two clinical trials [39, 40].

To date, the clinical failures of cellulose sulfate, carrageenan, and PRO 2000, despite clear demonstrations of in vitro efficacy and in vivo safety, have not been satisfactorily explained. Two reports that presented possible explanations for the failures of two of these compounds [36, 43] were published after their respective clinical trials had been initiated. In light of the results of washout experiments described above, our own studies were expanded to include other polyanionic HIV-1 inhibitors. We demonstrated using cell lines as well as primary human immune cells that other polyanionic molecules, including cellulose sulfate and carrageenan, also enhanced HIV-1 infection if the compounds were removed prior to the introduction of virus (Pirrone and Krebs, manuscript in preparation) [44]. In these experiments, demonstrated levels of enhancement in primary immune cells equaled or exceeded levels of enhancement achieved in the HeLa-based P4-R5 MAGI indicator cell line. We also demonstrated that enhancement of HIV-1 infection was dependent on a number of variables, including compound identity and concentration, time of application with respect to HIV-1 infection, and coreceptor usage.

5. Conclusions

In light of the clinical failures of polyanionic molecules such as dextran sulfate [45], cellulose sulfate [40, 46], carrageenan [47], and PRO 2000 [48, 49], a shift away from any further investigations focused on polyanionic HIV-1 inhibitors as drug or microbicide candidates might be perceived as an appropriate course of action. However, the benefits of continued studies of these compounds should not be overlooked. Studies of existing polyanion HIV-1 inhibitors may contribute to a better understanding of the clinical failures of these compounds and provide much-needed links between preclinical investigations and clinical trials. In a similar manner, studies of new and effective polyanionic inhibitors, such as alt-PSMA, may further our understanding of these agents. Finally, continued studies may provide useful information about molecular structures and charge characteristics that should be incorporated into future generations of HIV-1 inhibitors destined for systemic or microbicide use.

Acknowledgments

These studies were supported by Public Health Service Grant U19 AI076965. The following reagents were obtained through the AIDS Research and Reference Reagent Program, Division of AIDS, NIAID, NIH: P4-R5 MAGI cells, human rIL-2 (Dr. Maurice Gately, Hoffmann-La Roche, Inc.), TZM-bl (Dr. John C. Kappes, Dr. Xiaoyun Wu, and Tranzyme. Inc.), HIV-1 KNH1144 and HIV-1 KNH1207 (Dr. Victoria Polonis), HIV-1 93MW965 (Dr. Robert Bollinger and the UNAIDS Network for HIV), and HIV-1 SM145 and HIV-1 US1 (Dr. Nelson Michael). The authors would like to acknowledge Dr. Michael Nonnemacher for his contributions during the preparation of this manuscript.

References

H. Mitsuya, D. J. Looney, S. Kuno, R. Ueno, F. Wong-Staal, and S. Broder, “Dextran sulfate suppression of viruses in the HIV family: inhibition of virion binding to CD4⁺ cells,” Science, vol. 240, no. 4852, pp. 646–649, 1988.
View at: Google Scholar
M. Ito, M. Baba, A. Sato, R. Pauwels, E. De Clercq, and S. Shigeta, “Inhibitory effect of dextran sulfate and heparin on the replication of human immunodeficiency virus (HIV) in vitro,” Antiviral Research, vol. 7, no. 6, pp. 361–367, 1987.
View at: Google Scholar
M. Baba, R. Pauwels, J. Balzarini, J. Arnout, J. Desmyter, and E. De Clercq, “Mechanism of inhibitory effect of dextran sulfate and heparin on replication of human immunodeficiency virus in vitro,” Proceedings of the National Academy of Sciences of the United States of America, vol. 85, no. 16, pp. 6132–6136, 1988.
View at: Google Scholar
A. R. Neurath, N. Strick, Y. Y. Li, and A. K. Debnath, “Cellulose acetate phthalate, a common pharmaceutical excipient, inactivates HIV-1 and blocks the coreceptor binding site on the virus envelope glycoprotein gp120,” BMC Infectious Diseases, vol. 1, no. 1, article 17, 2001.
View at: Google Scholar
L. N. Callahan, M. Phelan, M. Mallinson, and M. A. Norcross, “Dextran sulfate blocks antibody binding to the principal neutralizing domain of human immunodeficiency virus type 1 without interferon with gp120-CD4 interactions,” Journal of Virology, vol. 65, no. 3, pp. 1543–1550, 1991.
View at: Google Scholar
M. Moulard, H. Lortat-Jacob, I. Mondor et al., “Selective interactions of polyanions with basic surfaces on human immunodeficiency virus type 1 gp120,” Journal of Virology, vol. 74, no. 4, pp. 1948–1960, 2000.
View at: Publisher Site | Google Scholar
J. Balzarini and L. Van Damme, “Microbicide drug candidates to prevent HIV infection,” The Lancet, vol. 369, no. 9563, pp. 787–797, 2007.
View at: Google Scholar
M. M. Lederman, R. E. Offord, and O. Hartley, “Microbicides and other topical strategies to prevent vaginal transmission of HIV,” Nature Reviews Immunology, vol. 6, no. 5, pp. 371–382, 2006.
View at: Publisher Site | Google Scholar
S. Rusconi, M. Moonis, D. P. Merrill et al., “Naphthalene sulfonate polymers with CD4-blocking and anti-human immunodeficiency virus type 1 activities,” Antimicrobial Agents and Chemotherapy, vol. 40, no. 1, pp. 234–236, 1996.
View at: Google Scholar
P. Mohan, D. Schols, M. Baba, and E. De Clercq, “Sulfonic acid polymers as a new class of human immunodeficiency virus inhibitors,” Antiviral Research, vol. 18, no. 2, pp. 139–150, 1992.
View at: Publisher Site | Google Scholar
R. F. Rando, S. Obara, M. C. Osterling et al., “Critical design features of phenyl carboxylate-containing polymer microbicides,” Antimicrobial Agents and Chemotherapy, vol. 50, no. 9, pp. 3081–3089, 2006.
View at: Publisher Site | Google Scholar
P. J. Klasse, R. Shattock, and J. P. Moore, “Antiretroviral drug-based microbicides to prevent HIV-1 sexual transmission,” Annual Review of Medicine, vol. 59, pp. 455–471, 2008.
View at: Publisher Site | Google Scholar
M. Konlee, “Sulfated polysaccharides (chondroitin sulfate and carrageenan) plus glucosamine sulfate are potent inhibitors of HIV,” Positive Health News, no. 17, pp. 4–7, 1998.
View at: Google Scholar
P. Charneau, G. Mirambeau, P. Roux, S. Paulous, H. Buc, and F. Clavel, “HIV-1 reverse transcription. A termination step at the center of the genome,” Journal of Molecular Biology, vol. 241, no. 5, pp. 651–662, 1994.
View at: Publisher Site | Google Scholar
E. J. Platt, K. Wehrly, S. E. Kuhmann, B. Chesebro, and D. Kabat, “Effects of CCR5 and CD4 cell surface concentrations on infections by macrophagetropic isolates of human immunodeficiency virus type 1,” Journal of Virology, vol. 72, no. 4, pp. 2855–2864, 1998.
View at: Google Scholar
H.-W. Lahm and S. Stein, “Characterization of recombinant human interleukin-2 with micromethods,” Journal of Chromatography, vol. 326, pp. 357–361, 1985.
View at: Google Scholar
B. K. Brown, J. M. Darden, S. Tovanabutra et al., “Biologic and genetic characterization of a panel of 60 human immunodeficiency virus type 1 isolates, representing clades A, B, C, D, CRF01_AE, and CRF02_AG, for the development and assessment of candidate vaccines,” Journal of Virology, vol. 79, no. 10, pp. 6089–6101, 2005.
View at: Publisher Site | Google Scholar
L. L. Jagodzinski, D. L. Wiggins, J. L. McManis et al., “Use of calibrated viral load standards for group M subtypes of human immunodeficiency virus type 1 to assess the performance of viral RNA quantitation tests,” Journal of Clinical Microbiology, vol. 38, no. 3, pp. 1247–1249, 2000.
View at: Google Scholar
N. L. Michael, S. A. Herman, S. Kwok et al., “Development of calibrated viral load standards for group M subtypes of human immunodeficiency virus type 1 and performance of an improved AMPLICOR HIV-1 MONITOR test with isolates of diverse subtypes,” Journal of Clinical Microbiology, vol. 37, no. 8, pp. 2557–2563, 1999.
View at: Google Scholar
M. Vahey, M. E. Nau, S. Barrick et al., “Performance of the Affymetrix GeneChip HIV PRT 440 platform for antiretroviral drug resistance genotyping of human immunodeficiency virus type 1 clades and viral isolates with length polymorphisms,” Journal of Clinical Microbiology, vol. 37, no. 8, pp. 2533–2537, 1999.
View at: Google Scholar
F. C. Krebs, S. R. Miller, D. Malamud, M. K. Howett, and B. Wigdahl, “Inactivation of human immunodeficiency virus type 1 by nonoxynol-9, C31G, or an alkyl sulfate, sodium dodecyl sulfate,” Antiviral Research, vol. 43, no. 3, pp. 157–173, 1999.
View at: Google Scholar
T. Mosmann, “Rapid colorimetric assay for cellular growth and survival: application to proliferation and cytotoxicity assays,” Journal of Immunological Methods, vol. 65, no. 1-2, pp. 55–63, 1983.
View at: Google Scholar
A. H. Cory, T. C. Owen, J. A. Barltrop, and J. G. Cory, “Use of an aqueous soluble tetrazolium/formazan assay for cell growth assays in culture,” Cancer Communications, vol. 3, no. 7, pp. 207–212, 1991.
View at: Google Scholar
B. J. Catalone, S. R. Miller, M. L. Ferguson et al., “Toxicity, inflammation, and anti-human immunodeficiency virus type 1 activity following exposure to chemical moieties of C31G,” Biomedicine and Pharmacotherapy, vol. 59, no. 8, pp. 430–437, 2005.
View at: Publisher Site | Google Scholar
J. P. Moore, S. G. Kitchen, P. Pugach, and J. A. Zack, “The CCR5 and CXCR4 coreceptors—central to understanding the transmission and pathogenesis of human immunodeficiency virus type 1 infection,” AIDS Research and Human Retroviruses, vol. 20, no. 1, pp. 111–126, 2004.
View at: Publisher Site | Google Scholar
I. A. Scordi-Bello, A. Mosoian, C. He et al., “Candidate sulfonated and sulfated topical microbicides: comparison of anti-human immunodeficiency virus activities and mechanisms of action,” Antimicrobial Agents and Chemotherapy, vol. 49, no. 9, pp. 3607–3615, 2005.
View at: Publisher Site | Google Scholar
X. Tan, R. Pearce-Pratt, and D. M. Phillips, “Productive infection of a cervical epithelial cell line with human immunodeficiency virus: implications for sexual transmission,” Journal of Virology, vol. 67, no. 11, pp. 6447–6452, 1993.
View at: Google Scholar
O. Bagasra and H. W. Lischner, “Activity of dextran sulfate and other polyanionic polysaccharides against human immunodeficiency virus,” Journal of Infectious Diseases, vol. 158, no. 5, pp. 1084–1087, 1988.
View at: Google Scholar
A. de Parseval, M. D. Bobardt, A. Chatterji et al., “A highly conserved arginine in gp120 governs HIV-1 binding to both syndecans and CCR5 via sulfated motifs,” Journal of Biological Chemistry, vol. 280, no. 47, pp. 39493–39504, 2005.
View at: Publisher Site | Google Scholar
H. A. Harrop, D. R. Coombe, and C. C. Rider, “Heparin specifically inhibits binding of V3 loop antibodies to HIV-1 gp120, an effect potentiated by CD4 binding,” AIDS, vol. 8, no. 2, pp. 183–192, 1994.
View at: Google Scholar
H. A. Harrop and C. C. Rider, “Heparin and its derivatives bind to HIV-1 recombinant envelope glycoproteins, rather than to recombinant HIV-1 receptor, CD4,” Glycobiology, vol. 8, no. 2, pp. 131–137, 1998.
View at: Publisher Site | Google Scholar
M. O. McClure, J. P. Moore, D. F. Blanc et al., “Investigations into the mechanism by which sulfated polysaccharides inhibit HIV infection in vitro,” AIDS Research and Human Retroviruses, vol. 8, no. 1, pp. 19–26, 1992.
View at: Google Scholar
R. R. Vives, A. Imberty, Q. J. Sattentau, and H. Lortat-Jacob, “Heparan sulfate targets the HIV-1 envelope glycoprotein gp120 coreceptor binding site,” Journal of Biological Chemistry, vol. 280, no. 22, pp. 21353–21357, 2005.
View at: Publisher Site | Google Scholar
J. Qian, S. Miller, M. L. Ferguson et al., “Use of the synthetic copolymer PSMA as a component in a combination microbicide active against HIV-1,” Retrovirology, vol. 2, supplement 1, p. S96, 2005.
View at: Google Scholar
W. Fang, Y. Cai, X. Chen et al., “Poly(styrene-alt-maleic anhydride) derivatives as potent anti-HIV microbicide candidates,” Bioorganic and Medicinal Chemistry Letters, vol. 19, no. 7, pp. 1903–1907, 2009.
View at: Publisher Site | Google Scholar
W. Tao, C. Richards, and D. Hamer, “Enhancement of HIV infection by cellulose sulfate,” AIDS Research and Human Retroviruses, vol. 24, no. 7, pp. 925–929, 2008.
View at: Publisher Site | Google Scholar
R. F. Tsuji, K. Hoshino, Y. Noro et al., “Suppression of allergic reaction by $?$ -carrageenan: toll-like receptor 4/MyD88-dependent and -independent modulation of immunity,” Clinical and Experimental Allergy, vol. 33, no. 2, pp. 249–258, 2003.
View at: Publisher Site | Google Scholar
R. J. Pomerantz, M. B. Feinberg, D. Trono, and D. Baltimore, “Lipopolysaccharide is a potent monocyte/macrophage-specific stimulator of human immunodeficiency virus type 1 expression,” Journal of Experimental Medicine, vol. 172, no. 1, pp. 253–261, 1990.
View at: Publisher Site | Google Scholar
V. Halpern, F. Ogunsola, O. Obunge et al., “Effectiveness of cellulose sulfate vaginal gel for the prevention of HIV infection: results of a phase III trial in Nigeria,” PLoS ONE, vol. 3, no. 11, article e3784, 2008.
View at: Publisher Site | Google Scholar
L. Van Damme, R. Govinden, F. M. Mirembe et al., “Lack of effectiveness of cellulose sulfate gel for the prevention of vaginal HIV transmission,” The New England Journal of Medicine, vol. 359, no. 5, pp. 463–472, 2008.
View at: Google Scholar
I. M. Malonza, F. Mirembe, C. Nakabiito et al., “Expanded Phase I safety and acceptability study of 6% cellulose sulfate vaginal gel,” AIDS, vol. 19, no. 18, pp. 2157–2163, 2005.
View at: Google Scholar
C. Mauck, D. H. Weiner, S. Ballagh et al., “Single and multiple exposure tolerance study of cellulose sulfate gel: a Phase I safety and colposcopy study,” Contraception, vol. 64, no. 6, pp. 383–391, 2001.
View at: Publisher Site | Google Scholar
A. R. Neurath, N. Strick, and Y. Y. Li, “Role of seminal plasma in the anti-HIV-1 activity of candidate microbicides,” BMC Infectious Diseases, vol. 6, article 150, 2006.
View at: Google Scholar
V. Pirrone, S. R. Passic, B. Wigdahl, and F. C. Krebs, “Clinical failures of select polyanionic microbicide candidates may be predicted by in vitro enhancement of HIV-1 infection,” in Proceedings of the 22nd International Conference on Antiviral Research, Miami, Fla, USA, 2009.
View at: Google Scholar
C. Flexner, P. A. Barditch-Crovo, D. M. Kornhauser et al., “Pharmacokinetics, toxicity, and activity of intravenous dextran sulfate in human immunodeficiency virus infection,” Antimicrobial Agents and Chemotherapy, vol. 35, no. 12, pp. 2544–2550, 1991.
View at: Google Scholar
J. Horwood, “Cellulose sulphate microbicide trial halted,” The Lancet Infectious Diseases, vol. 7, no. 3, p. 183, 2007.
View at: Google Scholar
J. Cohen, “AIDS research: microbicide fails to protect against HIV,” Science, vol. 319, no. 5866, pp. 1026–1027, 2008.
View at: Google Scholar
J. Cohen, “HIV Outwits Yet Another Microbicide,” ScienceNOW Daily News, December 2009, http://news.sciencemag.org/sciencenow/2009/12/14-01.html.
View at: Google Scholar
M. Warren and K. Marshall, “AVAC says MDP 301 Microbicide Trial results Disappointing, But Researchers and 9,400 Trial Volunteers Deserve Praise for Successful Trial,” AVAC Global Advocacy for HIV Prevention, December 2009.
View at: Google Scholar

Copyright

Copyright © 2010 Vanessa Pirrone et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

2006

Downloads

942

Citations