Abstract
Background. Altmetrics analyze the visibility of articles in social media and estimate their impact on the general population. We performed an altmetric analysis of articles on central nervous system inflammatory demyelinating disease (CIDD) and investigated its correlation with citation analysis. Methods. Articles in the 91 journals comprising the “clinical neurology,” “neuroscience,” and “medicine, general, and internal” Web of Science categories were searched for their relevance to the CIDD topic. The Altmetric Explorer database was used to determine the Altmetric.com Attention Score (AAS) values of the selected articles. The papers with the top 100 AAS values were characterized. Results. Articles most frequently mentioned online were primarily published after 2014 and were published in journals with high impact factors. All articles except one were dealt with the issue of multiple sclerosis. Most were original articles, but editorials were also common. Novel treatments and risk factors are the most frequent topics. The AAS was weakly correlated with journal impact factors; however, no link was found between the AAS and the number of citations. Conclusions. We present the top 100 most frequently mentioned CIDD articles in online media using an altmetric approach. Altmetrics can rapidly offer alternative information on the impact of research based on a broader audience and can complement traditional metrics.
1. Introduction
Traditionally, the impact of each scientific article is indirectly measured by its citation count or the impact factor of the journal publishing it. The impact factor is calculated as the average citation number of the articles that were published in that journal during the preceding two years (number of total citations/number of total citable articles). However, accessibility to websites is dramatically increasing worldwide, and the number of social media users has reached almost 2.46 billion in 2017 [1]. One-third of people worldwide are anticipated to use social media in 2021 [1]. Because of these shifts, the channels and audiences for science communication are inexorably increasing. Altmetrics, also called alternative metrics, capture the online visibility of scholarly material and indirectly present its influence on social networks [2]. Altmetrics quantify the frequency of mentions in online channels including news outlets, science blogs, Twitter, Facebook, Sina Weibo, Wikipedia, public policy documents, peer review platforms, research highlights on Faculty of 1000, reference manager such as Mendeley, and multimedia (YouTube and Q&A) [3]. The Altmetric.com Attention Score (AAS), one of the commonly used altmetric tools, is calculated depending on the intensity, types (e.g., retweets are scored lower than original tweets), and sources of attention (e.g., articles shared through news outlets are weighted 8, but those shared on Twitter are weighted 1) [3]. A high AAS, as calculated by this method, represents noteworthy interest in an article within social media outlets [4]. Altmetrics have certain advantages over traditional metrics: the former can measure the effect of scientific works outside the academic community and can estimate the impact of discoveries more quickly than citation analysis [4].
Multiple sclerosis (MS), the most representative form of central nervous system inflammatory demyelinating disease (CIDD), has been an important issue in the health care community because of its relatively high prevalence, chronicity, and massive social burden. MS typically develops in young adults aged 20–50 years, and its incidence peaks in the fourth decade of life [5]. These young adult patients are especially familiar with Internet use, and they actively acquire and spread information on the disease by social media or other digital outlets [6]. There are numerous official and unofficial social networks, web pages, and blogs on the topic of MS [7].
Here, our purpose is to identify the most influential articles in the field of CIDD in online media and to investigate the characteristics of those articles. We hypothesized that traditional metrics and altmetrics were not necessarily related. We compared the ASS to a traditional citation analysis to find the relationship between the two metrics.
2. Materials and Methods
2.1. Article Selection and Analysis
We searched journals in the categories of “clinical neurology,” “neuroscience,” and “medicine, general, and internal” in the 2016 edition of Web of Science (Thomson Reuters, New York, NY) to further identify the CIDD-related articles that were most commonly mentioned in digital media. A total of 91 journals were selected, and articles published in those journals were subjected to further analysis. We used the AAS as a weighted measure of the attention received by each article in digital media [3]. The AAS of articles in selected journals were searched separately in the Altmetric Explorer database (Altmetric LLP, London, UK) on May 25, 2018. All articles were compiled as a single database and rearranged in order of descending AAS. To retrieve CIDD-related articles, we chose articles with the following terms in their titles: “multiple sclerosis,” “clinically isolated syndrome,” “neuromyelitis optica,” “neuromyelitis optica spectrum disorder (NMOSD),” “optic neuritis,” “myelitis,” “demyelinating disease,” “Devic’s disease,” “aquaporin-4 antibody,” “neuromyelitis optica-immunoglobulin G,” “Schilder’s disease,” “Schilder’s diffuse sclerosis,” “diffuse myelinoclastic sclerosis,” “Balo concentric sclerosis,” “Marburg multiple sclerosis,” “solitary sclerosis,” “acute disseminated encephalomyelitis,” or “acute hemorrhagic leukoencephalitis.” After reviewing the original texts to confirm their relevance to CIDD, we identified the top 100 articles by AAS. To avoid the risk of overestimating public attention because of single articles dealing with many different diseases, we excluded articles that dealt with multiple diseases (e.g., an article that evaluated the effect of living near a major road on the risk of multiple sclerosis, dementia, and Parkinson’s disease had the highest AAS in the present study, but it was excluded from further evaluation) [8]. Articles were reviewed for the selection process regardless of language, scholarly identifier, or document type.
These mixed methods study utilized quantitative analyses, qualitative assessment, and an extensive review of literature. The following information was extracted from the top 100 most highly mentioned CIDD-related articles in online media: article title, year of publication (if electronic publishing was opened ahead of formal publication, the year of electronic publishing was selected for further analysis), published journal, journal impact factor, country of origin, type of article, main topics, subject of article (e.g., MS, NMOSD, and myelitis), AAS, and number of citations. Country of origin was defined by the affiliation of the first author. If the first author had more than one affiliation, then the affiliation of the corresponding author was used to determine the country of origin. The journal impact factor was extracted from the 2017 edition of Web of Science. The traditional citation count for each article was obtained from Scopus and compared to the AAS.
The study was exempted from approval by an ethics committee because it was a bibliometric analysis.
2.2. Statistical Analysis
All statics were analyzed using SPSS Statistics 20 (SPSS, Chicago, IL, USA). Pearson’s correlation analysis was performed to assess the relationship between AAS and journal impact factor (or citation count). was regarded as statistically significant.
3. Results
We identified 100 articles with the highest altmetric scores (Table 1). AAS values were in the range of 113 to 1302 (median 214.5). The median journal impact factor of the articles with the top 100 altmetric scores was 7.690 (range 2.45–79.26). The median number of citations per article was 21 (range 0–765). Altmetric.com has tracked attention paid to articles mentioned in each source since 2011. The identified articles were published between 1990 and 2018, and 85% of the articles were published after 2014 (Figure 1). Articles on the top 100 list originated from 15 different countries (Table 2). Approximately 60% of the top 100 articles originated in Northern America. The remaining high-ranked countries of origin were on the European continent. Interestingly, the top 100 articles discussed online were published mostly in journals with high impact factors (Table 3). Among 23 journals, Neurology (n = 31) has the most articles ranked in the top 100, followed in order, by the Journal of the American Medical Association Neurology (n = 14), The Lancet (n = 11), the Journal of Neurology, Neurosurgery and Psychiatry (n = 10), and The New England Journal of Medicine (n = 8). Except for one article explaining clinical biomarkers to differentiate inflammatory myelopathy from other causes, the subject of all the articles was MS. No articles on the subject of NMOSD were in the top 100. However, two NMOSD articles had notably high AAS values. One article suggesting 2015 NMOSD diagnostic criteria ranked 112 (AAS = 98), and an article regarding the clinical course and therapeutic efficiency of NMOSD ranked 125 (AAS = 88) [9, 10].
Original articles were the most frequent type among the top 100 highly mentioned CIDD papers, and 60% of the original articles were clinical observational studies (Table 4). Approximately 10% of highly mentioned articles were reviews, meta-analyses, or articles suggesting practical guidelines or diagnostic criteria. Editorial articles were also included in the list of high-ranking publications (Table 4). Most of these editorial articles summarized notable recent studies and provided expert opinions regarding new treatments (including B-cell depletion treatments such as ocrelizumab and alemtuzumab, BCG vaccines, hematopoietic stem-cell transplantation, and percutaneous transluminal venous angioplasty) and risk factors (such as coffee consumption, chronic cerebrospinal venous insufficiency, diet, vitamin D, and smoking). The main issues addressed by the top 100 articles were mostly related to disease treatment, and many of them were comparative clinical trials (Table 5). The specific treatments receiving the most attention are presented in Table 6. One article was a retrospective cohort study presenting the complication of malignancy after mitoxantrone treatment in MS. The second most frequent class of topics consisted of the risk factors for MS development and aggravation (Table 6). Notably, several high-ranking articles concerned quality-of-life or economic issues.
Regarding the correlation between AAS and journal impact factor among the top 100 articles, a weak positive correlation was revealed (r = 0.2474; ) (Figure 2). However, there was no significant correlation between AAS and citation counts.
4. Discussion
We performed an altmetric analysis of CIDD articles to examine their influences on not only academia but also the general public. We thus obtained a list of the 100 articles in the field of CIDD that were most frequently mentioned in web media according to weighted AAS. These results show trends in scientific research and social attention on a shorter time lag than traditional citation analysis. Among our top 100 articles, AAS showed correlation with journal impact factor but not with citation count, which means that altmetrics are correlated with traditional metrics to some extent but do not convey identical information.
Traditional citation analysis has a long history as a tool for measuring the impact of an article, authors, and contributed institutes/countries. Citation analysis offers a “best-seller list” of the specific diseases that receive the most attention in academia. However, this form of analysis has several limitations. In practice, citations of an article start 1–2 years after its initial publication and peak between 3 and 10 years [11]. After considerable periods, some important articles cease to be frequently cited because their substance has been absorbed into the current body of common knowledge, the phenomenon called “obliteration by incorporation” [12]. For this reason, traditional analysis best captures the actual impact of a paper 10 to 20 years after publication [13, 14]. Altmetrics are a new class of metrics that can indicate the impact of an article in a more rapid and responsive way than citation analysis. Previous results comparing altmetrics and citation analysis have shown that new web-based metrics are especially sensitive to the latest news [15, 16]. Reflecting this characteristic, most of the top 100 CIDD articles in our study were published after 2014, and a plurality of articles were published in 2016. Furthermore, when we considered the relationship between citation number and AAS, we could not find any correlation. A substantial number of articles on the top 100 list were cited fewer than 10 times, which might be due to the scope of the audience, the document types, and the time since publication. Our results support the idea that citation analysis and altmetrics do not measure the same construct; they are complementary rather than substitutable. The absence of a strong correlation between new metrics and citation analysis was consistently observed in other studies [17–23]. Only Mendeley, an online reference manager, showed a moderate positive link with citation analysis [19, 20].
In our correlation analysis, AAS showed a weak positive correlation with a journal impact factor. In addition, the top 5 most represented journals, which included 74% of the listed articles, have impact factors of more than 7. This result may have occurred because articles published in higher impact journals are generally more interesting in their own right or because of the fame and wide reach of these journals. Several factors are known to influence AAS, such as journal impact factor, article length, number of collaborators or references, presence of a press release, free accessibility, document type, and funding [16, 24]. Several factors, such as reference count and collaborative practices, positively affect both citations and altmetrics [16]. However, the impacts of other factors are not identical [16]. Longer papers typically receive more citations, but the opposite trend can be seen on social media platforms [16]. Editorial and news documents, which are rarely cited, are frequently mentioned document types on Twitter [16]. Simple declarative titles that present key conclusions are associated with high AAS values [25–27]. Our study also showed substantial inclusion of editorial articles, which distill topics to their essentials, and are easier to understand than research papers. Brevity and understandability are important factors in the dissemination of articles on websites [25].
The most frequently mentioned topics were associated with novel therapeutics and risk factors (especially correctable environmental factors). Articles concerning disease mechanisms, clinical characteristics, and diagnosis also accounted for a portion of the list. However, compared with traditional citation analysis, altmetrics show increased attention to treatment and decreased attention to diagnosis and disease mechanisms [28]. Similarly, articles that address socioeconomic and quality-of-life issues of CIDD patients are also highly ranked in terms of altmetrics, an outcome that is seldom observed in citation analysis [28]. Other researchers also reported that research papers relating to humanities and social science are most frequently found on online media, in contrast to the pattern in conventional media [16]. Our findings indirectly represent the difference between audience and user groups (academic-based versus a more inclusive public audience) and their interest in conventional and social media. The top 100 articles mostly originated in North America and Europe. These results may have occurred because many core studies were performed in those areas. Another influence might be the large volume of attention paid to MS in Northern America and Europe because of its relatively high prevalence and because of widespread Internet use [1, 5].
Several aspects of this study should be considered. First, altmetrics are new methods of bibliometrics, and their value was uncertain until now. Several inherent limitations of altmetrics have been discussed recently [4, 21, 29]. Social media are at risk of overestimating the impact of research with sensational outcomes [21, 29]. Articles with negative findings or attractive headlines are likely to have high AAS values [21]. Some scientists have reported that metrics from online platforms do not accurately reflect the quality research [30]. AAS scores cannot distinguish the depth of online mentions although the algorithm is adjusted to weight scores by the source and type of attention [22]. Furthermore, due to the anonymity of social media, the demographics of the users referencing the articles (scientists or the lay public) and the validity of the data are often not certain [21, 29]. As in traditional citation analysis, altmetrics may be influenced by temporal bias [12]. Second, we used only altmetric data offered by Altmetric.com in this study. Social media metrics in each data aggregator can differ due to methodological (e.g., selections on the data collection) and reporting choices (e.g., data aggregation approaches) and have both advantages and disadvantages [31]. Other tools such as ALS-PLoS, ImpactStory, and Plum Analytics can also be used to track altmetrics data. Third, we did not perform subgroup analyses based on the attention source subtype. Further studies focusing on difference in the social media subset are expected.
5. Conclusion
We identified the 100 research outputs most discussed on online platforms shared by a broad audience of scientists and laypeople. There was some discordance between traditional citation analysis and altmetrics analysis of CIDD articles. Data from altmetrics may not accurately represent scientific quality or importance, but they can reflect the dissemination of research through the general population, including the scientific community, on the web. The influence of online media will inevitably increase in the future. Altmetrics can provide a broad, real-time web-based reflection of research and can complement traditional analysis as a tool to evaluate the impact of studies. Further research will be needed to assess the validity of altmetrics and explore the factors that influence them.
Abbreviations
| AAS: | Altmetric.com Attention Score |
| CIDD: | Central nervous system inflammatory demyelinating disease |
| CSF: | Cerebrospinal fluid |
| IF: | Impact factor |
| MS: | Multiple sclerosis |
| NMOSD: | Neuromyelitis optica spectrum disorder |
| No: | Number. |
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest.