Crowd Sourcing Literature Reviews

by Francis Hane, PhD on 14 April 2017

Recently, the Journal of Alzheimer’s Disease (JAD) solicited their readership to vote on the most influential research articles in the field of Alzheimer’s disease (AD) in the last 5 years. The response to this exercise was overwhelming: approximately 300 articles were voted on to determine which articles were deemed the “top papers”. Readers voted on the articles and 50 articles were selected (http://www.j-alz.com/top50).

This exercise inspired us to write a comprehensive review of the field of recent AD research [1–3], a so-called “greatest hits” of AD research. We used the 300 nominated papers as a basis for our review series. We believe that this method of reviewing the literature is somewhat novel. Traditionally, academic reviews consist of a small number of authors deciding what articles should be included in the review, the review typically being of very narrow breadth, but considerable depth [4,5]. This traditional approach has a number of advantages and disadvantages. Notably, it makes logical sense for an “expert” to be the arbiter of the quality of certain research articles to include, especially in a field as controversial as AD. Secondly, the traditional approach allows for the reader to rapidly get up to speed on a given topic within the vast field of AD research. However, the traditional review approach suffers from a couple of notable drawbacks. Firstly, it is subject to the biases of the authors; for example, an author who has spent her career advancing the amyloid hypothesis may be loath to give sufficient credence to alternative hypotheses. Secondly, such a deep and narrow focus may result in a review resulting in “not seeing the forest for the trees”, so to speak. As we discuss in our review series, there is simply no longer any evidence that AD is caused by one specific factor. Rather, AD is caused by a myriad of factors all interacting with one another, their interplay leading to the symptoms and pathological hallmarks that we refer to as AD. In our opinion, a lot of effort has been wasted in the past arguing the amyloid versus tau hypothesis as opposed to investigating the complex interplay between these proteins and other factors such as inflammatory responses, as is now routinely done. We attribute this phenomenon to the silos formed in each research field. It is very easy to attend an international Alzheimer’s disease conference and interact with only researchers investigating, for example, AD pathology, with the topic of, say, AD imaging, being as remote as an entirely different pathology.

By contrast, our approach seeks to mitigate the limitations of traditional reviews. The selection of articles, and even topics, for our review was largely independent of our own opinions about the most promising AD research. This “crowd-sourced” approach relied on the broad expertise available in the AD research community to select topics and articles for review. Of course, we did have to select articles that provide background information in order to introduce more advanced articles. Our approach largely removed the biases that we hold from the selection process. (We are molecular biophysicists with expertise in amyloid-β.) Additionally, our approach allowed for a broad overview of contemporary AD research topics. In our review series, we noted a few topics that the readership seems to find especially promising (i.e., “hot”): amyloid-tau synergies, the prion hypothesis of AD, the effect of AD on the brain connectome, and advanced brain imaging. Additionally, contemporary diagnostic guidelines and the results of the latest clinical trials are always of interest to the clinician. These active areas of interest are of importance to both new and senior investigators as they allow investigators to tailor their research to the contemporary unsolved questions in the field of AD research.

In addition to the advantages of our approach discussed above, our approach does suffer some limitations. Because of the breadth and the intent of our methodology, our reviews do not provide the depth that some other more contemporary reviews of a specific sub-field of AD may provide [6,7]. Additionally, we relied on the community of AD researchers to vote on papers that were mentioned. Sometimes crowds are wrong [8,9]. There is a possibility that some articles were nominated and included because they were “sexy” but lacked the scientific rigour that an experienced researcher could ferret out.

Our approach is not a replacement to traditional academic review authorship, but does demonstrate the power of collaboration when large groups of knowledgeable scientists converge allowing us to see both the “forest” and the “trees”.

Given this novel approach, it raises the question, are there any other alternative methods of review of scientific literature that may prove to be effective in the transmission of knowledge from author to reader?

Some time ago, I had idea of harnessing artificial intelligence (AI) as a method of producing individualized academic reviews. (If you’re into AI feel free to contact me about collaboration.) This “artificial” type of review may not be published, but instead, each reader could determine for themselves what topics they wanted covered in the review. An interface would work something like this: a user would type “Alzheimer’s disease” into a field in the interface. The interface would return a list of popular topics within AD that could be checked by the user. The user may be interested in, let’s say, inflammation, amyloid, and treatment, and select those topics on the interface. The program would than search the literature and select the most influential papers discussing those topics. (A weighting algorithm would weight papers to select by citations and recency.) Now here would be the difficult part: the program would have to mine the papers selected for the relevant data contained therein and summarize and organize the raw data into cohesive sentences and paragraphs. Voila! An instant, personalized, review covering only the topics you want to know. In theory, this could render the academic review obsolete.

In the meantime, reviews will continue to provide investigators with the most pertinent information at their fingertips.

References
[1] Hane FT, Lee BY, Leonenko Z (2017) Recent progress in Alzheimer’s disease research, part 1 : pathology. J Alzheimers Dis 57, 1–28.
[2] Robinson M, Lee BY, Hane FT (2017) Recent progess in Alzheimer’s disease research, part 2: genetics and epidemiology. J Alzheimers Dis 57, 317–330.
[3] Hane FT, Robinson M, Lee BY, Bai O, Leonenko Z, Albert MS (2017) Recent progress in Alzheimer’s disease research, part 3: diagnosis and treatment. J Alzheimers Dis 57, 645-665.
[4] Drolle E, Hane F, Lee B, Leonenko Z (2014) Atomic force microscopy to study molecular mechanisms of amyloid fibril formation and toxicity in Alzheimer’s disease. Drug Metab Rev 46, 207–223.
[5] Hane F, Leonenko Z (2014) Effect of metals on kinetic pathways of amyloid-β aggregation. Biomolecules 4, 101–116.
[6] Heppner FL, Ransohoff RM, Becher B (2015) Immune attack: the role of inflammation in Alzheimer disease. Nat Rev Neurosci 16, 358–372.
[7] Nasica-labouze J, Nguyen PH, Sterpone F, Berthoumieu O, Buchete N, Simone A De, Doig AJ, Faller P, Garcia A, Laio A, Li MS, Melchionna S, Mousseau N, Mu Y, Paravastu A, Pasquali S, Rosenman DJ, Strodel B, Tarus B, Viles JH, Zhang T, Wang C, Derreumaux P (2015) Amyloid β protein and Alzheimer’s disease : when computer simulations complement experimental studies. Chem Rev 115, 3518–3563.
[8] Flegenheimer M, Barbaro M (2016) Donald Trump is elected president. New York Times A1.
[9] Wolfe-Simon F, Switzer Blum J, Kulp TR, Gordon GW, Hoeft SE, Pett-Ridge J, Stolz JF, Webb SM, Weber PK, Davies PCW, Anbar AD, Oremland RS, Hille R, Lane TW, Morel FM, Wolfe-Simon F, Davies PCW, Anbar AD, Rosen BP, Baer CD, Edwards JO, Rieger PH, Oremland RS, Stolz JF, Hollibaugh JT, Blum JS, Bindi AB, Buzzelli J, Stolz JF, Oremland RS, Takeuchi M, Makino W, Cotner J, Sterner R, Elser J, Mandelstam J, Smith PG, Pickering IJ, Holbrook S, Dickerson R, Kim SH, Oremland RS, Stolz JF, Quillaguaman J, Delgado O, Mattiasson B, Hatti-Kaul R (2011) A bacterium that can grow by using arsenic instead of phosphorus. Science 332, 1163–1166.

Last comment on 14 April 2017 by Francis Hane, PhD

Comment Icon |
Bookmark Icon Bookmark Recommend Icon Recommend Follow Icon Follow
| Bookmark | Recommend | Follow

Comments

Francis,
I very much liked reading your ideas regarding the use of AI to help in the process of writing a review article. In an AD bibliometric analysis last year (https://rd.springer.com/article/10.1007/s11192-016-2119-7 -- see figure 5), some colleagues and I and used a simple form of AI to select the forty most salient articles in the literature describing the use of transgenic mice to investigate the role of amyloid beta-protein precursor in AD. Below is the list of forty papers that the citation-network-driven algorithm produced.

Something interesting that came out of the analysis was the fact that a retracted paper (Kawabata et al. [1991]) was included in the forty recommended papers. As you can imagine it caused a debate among the coauthors regarding whether certain retractions can still be viewed as important milestones in the history of a given subfield of science.

I would be curious to year thoughts regarding the list of forty. Are there any oblivious false positives or any papers which were left off the list which clearly should have been included? Any advice on general rules to apply before running the algorithm (e.g. eliminate all retracted papers).

Regards,
Aaron

Year Published

Authors

Title

Journal

Paper Type

1991

KAWABATA, S; HIGGINS, GA; GORDON, JW

AMYLOID PLAQUES, NEUROFIBRILLARY TANGLES AND NEURONAL LOSS IN BRAINS OF TRANSGENIC MICE OVEREXPRESSING A C-TERMINAL FRAGMENT OF HUMAN AMYLOID PRECURSOR PROTEIN (RETRACTED ARTICLE. SEE VOL 356, PG 265, 1992)

NATURE

Article

1991

WIRAK, DO; BAYNEY, R; KUNDEL, CA; LEE, A; SCANGOS, GA; TRAPP, BD; UNTERBECK, AJ

REGULATORY REGION OF HUMAN AMYLOID PRECURSOR PROTEIN (APP) GENE PROMOTES NEURON-SPECIFIC GENE-EXPRESSION IN THE CNS OF TRANSGENIC MICE

EMBO J

Article

1991

QUON, D; WANG, Y; CATALANO, R; SCARDINA, JM; MURAKAMI, K; CORDELL, B

FORMATION OF BETA-AMYLOID PROTEIN DEPOSITS IN BRAINS OF TRANSGENIC MICE

NATURE

Article

1991

WIRAK, DO; BAYNEY, R; RAMABHADRAN, TV; FRACASSO, RP; HART, JT; HAUER, PE; HSIAU, P; PEKAR, SK; SCANGOS, GA; TRAPP, BD; UNTERBECK, AJ

DEPOSITS OF AMYLOID-BETA PROTEIN IN THE CENTRAL-NERVOUS-SYSTEM OF TRANSGENIC MICE

SCIENCE

Article

1994

HIGGINS, LS; HOLTZMAN, DM; RABIN, J; MOBLEY, WC; CORDELL, B

TRANSGENIC MOUSE-BRAIN HISTOPATHOLOGY RESEMBLES EARLY ALZHEIMERS-DISEASE

ANN NEUROL

Article

1995

GAMES, D; ADAMS, D; ALESSANDRINI, R; BARBOUR, R; BERTHELETTE, P; BLACKWELL, C; CARR, T; CLEMENS, J; DONALDSON, T; GILLESPIE, F; GUIDO, T; HAGOPIAN, S; JOHNSONWOOD, K; KHAN, K; LEE, M; LEIBOWITZ, P; LIEBERBURG, I; LITTLE, S; MASLIAH, E; MCCONLOGUE, L; MONTOYAZAVALA, M; MUCKE, L; PAGANINI, L; PENNIMAN, E; POWER, M; SCHENK, D; SEUBERT, P; SNYDER, B; SORIANO, F; TAN, H; VITALE, J; WADSWORTH, S; WOLOZIN, B; ZHAO, J

ALZHEIMER-TYPE NEUROPATHOLOGY IN TRANSGENIC MICE OVEREXPRESSING V717F BETA-AMYLOID PRECURSOR PROTEIN

NATURE

Article

1995

HSIAO, KK; BORCHELT, DR; OLSON, K; JOHANNSDOTTIR, R; KITT, C; YUNIS, W; XU, S; ECKMAN, C; YOUNKIN, S; PRICE, D; IADECOLA, C; CLARK, HB; CARLSON, G

AGE-RELATED CNS DISORDER AND EARLY DEATH IN TRANSGENIC FVB/N MICE OVEREXPRESSING ALZHEIMER AMYLOID PRECURSOR PROTEINS

NEURON

Article

1995

ROCKENSTEIN, EM; MCCONLOGUE, L; TAN, H; POWER, M; MASLIAH, E; MUCKE, L

LEVELS AND ALTERNATIVE SPLICING OF AMYLOID-BETA PROTEIN-PRECURSOR (APP) TRANSCRIPTS IN BRAINS OF APP TRANSGENIC MICE AND HUMANS WITH ALZHEIMERS-DISEASE

J BIOL CHEM

Article

1995

LAFERLA, FM; TINKLE, BT; BIEBERICH, CJ; HAUDENSCHILD, CC; JAY, G

THE ALZHEIMERS A-BETA PEPTIDE INDUCES NEURODEGENERATION AND APOPTOTIC CELL-DEATH IN TRANSGENIC MICE

NAT GENET

Article

1996

Hsiao, K; Chapman, P; Nilsen, S; Eckman, C; Harigaya, Y; Younkin, S; Yang, FS; Cole, G

Correlative memory deficits, A beta elevation, and amyloid plaques in transgenic mice

SCIENCE

Article

1996

Masliah, E; Sisk, A; Mallory, M; Mucke, L; Schenk, D; Games, D

Comparison of neurodegenerative pathology in transgenic mice overexpressing V717F beta-amyloid precursor protein and Alzheimer's disease

J NEUROSCI

Article

1997

Irizarry, MC; Soriano, F; McNamara, M; Page, KJ; Schenk, D; Games, D; Hyman, BT

A beta deposition is associated with neuropil changes, but not with overt neuronal loss in the human amyloid precursor protein V717F (PDAPP) transgenic mouse

J NEUROSCI

Article

1997

JohnsonWood, K; Lee, M; Motter, R; Hu, K; Gordon, G; Barbour, R; Khan, K; Gordon, M; Tan, H; Games, D; Lieberburg, I; Schenk, D; Seubert, P; McConlogue, L

Amyloid precursor protein processing and A beta(42) deposition in a transgenic mouse model of Alzheimer disease

P NATL ACAD SCI USA

Article

1998

Calhoun, ME; Wiederhold, KH; Abramowski, D; Phinney, AL; Probst, A; Sturchler-Pierrat, C; Staufenbiel, M; Sommer, B; Jucker, M

Neuron loss in APP transgenic mice

NATURE

Letter

1999

Hsia, AY; Masliah, E; McConlogue, L; Yu, GQ; Tatsuno, G; Hu, K; Kholodenko, D; Malenka, RC; Nicoll, RA; Mucke, L

Plaque-independent disruption of neural circuits in Alzheimer's disease mouse models

P NATL ACAD SCI USA

Article

2000

Mucke, L; Masliah, E; Yu, GQ; Mallory, M; Rockenstein, EM; Tatsuno, G; Hu, K; Kholodenko, D; Johnson-Wood, K; McConlogue, L

High-level neuronal expression of A beta(1-42) in wild-type human amyloid protein precursor transgenic mice: Synaptotoxicity without plaque formation

J NEUROSCI

Article

2000

Morgan, D; Diamond, DM; Gottschall, PE; Ugen, KE; Dickey, C; Hardy, J; Duff, K; Jantzen, P; DiCarlo, G; Wilcock, D; Connor, K; Hatcher, J; Hope, C; Gordon, M; Arendash, GW

A beta peptide vaccination prevents memory loss in an animal model of Alzheimer's disease

NATURE

Article

2001

Koistinaho, M; Ort, M; Cimadevilla, JM; Vondrous, R; Cordell, B; Koistinaho, J; Bures, J; Higgins, LS

Specific spatial learning deficits become severe with age in beta-amyloid precursor protein transgenic mice that harbor diffuse beta-amyloid deposits but do not form plaques

P NATL ACAD SCI USA

Article

2002

Selkoe, DJ

Alzheimer's disease is a synaptic failure

SCIENCE

Article

2002

Dodart, JC; Bales, KR; Gannon, KS; Greene, SJ; DeMattos, RB; Mathis, C; DeLong, CA; Wu, S; Wu, X; Holtzman, DM; Paul, SM

Immunization reverses memory deficits without reducing brain A beta burden in Alzheimer's disease model

NAT NEUROSCI

Article

2003

Oddo, S; Caccamo, A; Shepherd, JD; Murphy, MP; Golde, TE; Kayed, R; Metherate, R; Mattson, MP; Akbari, Y; LaFerla, FM

Triple-transgenic model of Alzheimer's disease with plaques and tangles: Intracellular A beta and synaptic dysfunction

NEURON

Article

2003

Kamenetz, F; Tomita, T; Hsieh, H; Seabrook, G; Borchelt, D; Iwatsubo, T; Sisodia, S; Malinow, R

APP processing and synaptic function

NEURON

Article

2004

Ohno, M; Sametsky, EA; Younkin, LH; Oakley, H; Younkin, SG; Citron, M; Vassar, R; Disterhoft, JF

BACE1 deficiency rescues memory deficits and cholinergic dysfunction in a mouse model of Alzheimer's disease

NEURON

Article

2005

Kobayashi, DT; Chen, KS

Behavioral phenotypes of amyloid-based genetically modified mouse models of Alzheimer's disease

GENES BRAIN BEHAV

Review

2005

Chin, J; Palop, JJ; Puolivali, J; Massaro, C; Bien-Ly, N; Gerstein, H; Scearce-Levie, K; Masliah, E; Mucke, L

Fyn kinase induces synaptic and cognitive impairments in a Transgenic mouse model of Alzheimer's disease

J NEUROSCI

Article

2006

Esposito, L; Raber, J; Kekonius, L; Yan, FR; Yu, GQ; Bien-Ly, N; Puolivali, J; Scearce-Levie, K; Masliah, E; Mucke, L

Reduction in mitochondrial superoxide dismutase modulates Alzheimer's disease-like pathology and accelerates the onset of behavioral changes in human amyloid precursor protein transgenic mice

J NEUROSCI

Review

2007

Palop, JJ; Chin, J; Roberson, ED; Wang, J; Thwin, MT; Bien-Ly, N; Yoo, J; Ho, KO; Yu, GQ; Kreitzer, A; Finkbeiner, S; Noebels, JL; Mucke, L

Aberrant excitatory neuronal activity and compensatory remodeling of inhibitory hippocampal circuits in mouse models of Alzheimer's disease

NEURON

Article

2007

Cheng, IH; Scearce-Levie, K; Legleiter, J; Palop, JJ; Gerstein, H; Bien-Ly, N; Puolivali, J; Lesne, S; Ashe, KH; Muchowski, PJ; Mucke, L

Accelerating amyloid-beta fibrillization reduces oligomer levels and functional deficits in Alzheimer disease mouse models

J BIOL CHEM

Article

2008

Meilandt, WJ; Yu, GQ; Chin, J; Roberson, ED; Palop, JJ; Wu, T; Scearce-Levie, K; Mucke, L

Enkephalin elevations contribute to neuronal and behavioral impairments in a transgenic mouse model of Alzheimer's disease

J NEUROSCI

Article

2009

Palop, JJ; Mucke, L

Epilepsy and Cognitive Impairments in Alzheimer Disease

ARCH NEUROL-CHICAGO

Review

2010

Harris, JA; Devidze, N; Halabisky, B; Lo, I; Thwin, MT; Yu, GQ; Bredesen, DE; Masliah, E; Mucke, L

Many Neuronal and Behavioral Impairments in Transgenic Mouse Models of Alzheimer's Disease Are Independent of Caspase Cleavage of the Amyloid Precursor Protein

J NEUROSCI

Article

2010

Harris, JA; Devidze, N; Verret, L; Ho, K; Halabisky, B; Thwin, MT; Kim, D; Hamto, P; Lo, I; Yu, GQ; Palop, JJ; Masliah, E; Mucke, L

Transsynaptic Progression of Amyloid-beta-Induced Neuronal Dysfunction within the Entorhinal-Hippocampal Network

NEURON

Article

2011

Bero, AW; Yan, P; Roh, JH; Cirrito, JR; Stewart, FR; Raichle, ME; Lee, JM; Holtzman, DM

Neuronal activity regulates the regional vulnerability to amyloid-beta deposition

NAT NEUROSCI

Article

2012

Sanchez, PE; Zhu, L; Verret, L; Vossel, KA; Orr, AG; Cirrito, JR; Devidze, N; Ho, K; Yu, GQ; Palop, JJ; Mucke, L

Levetiracetam suppresses neuronal network dysfunction and reverses synaptic and cognitive deficits in an Alzheimer's disease model

P NATL ACAD SCI USA

Article

2012

Verret, L; Mann, EO; Hang, GB; Barth, AMI; Cobos, I; Ho, K; Devidze, N; Masliah, E; Kreitzer, AC; Mody, I; Mucke, L; Palop, JJ

Inhibitory Interneuron Deficit Links Altered Network Activity and Cognitive Dysfunction in Alzheimer Model

CELL

Article

2013

Corbett, BF; Leiser, SC; Ling, HP; Nagy, R; Breysse, N; Zhang, XH; Hazra, A; Wood, A; Pangalos, MN; Reinhart, PH; Chin, J

Sodium Channel Cleavage Is Associated with Aberrant Neuronal Activity and Cognitive Deficits in a Mouse Model of Alzheimer's Disease

J NEUROSCI

Article

2014

Tai, LM; Koster, KP; Luo, J; Lee, SH; Wang, YT; Collins, NC; Ben Aissa, M; Thatcher, GRJ; LaDu, MJ

Amyloid-beta Pathology and APOE Genotype Modulate Retinoid X Receptor Agonist Activity in Vivo

J BIOL CHEM

Article

2014

Born, HA; Kim, JY; Savjani, RR; Das, P; Dabaghian, YA; Guo, QX; Yoo, JW; Schuler, DR; Cirrito, JR; Zheng, H; Golde, TE; Noebels, JL; Jankowsky, JL

Genetic Suppression of Transgenic APP Rescues Hypersynchronous Network Activity in a Mouse Model of Alzeimer's Disease

J NEUROSCI

Article

2014

Fowler, SW; Chiang, ACA; Savjani, RR; Larson, ME; Sherman, MA; Schuler, DR; Cirrito, JR; Lesne, SE; Jankowsky, JL

Genetic Modulation of Soluble A beta Rescues Cognitive and Synaptic Impairment in a Mouse Model of Alzheimer's Disease

J NEUROSCI

Article

2015

Liu, DS; Pan, XD; Zhang, J; Shen, H; Collins, NC; Cole, AM; Koster, KP; Ben Aissa, M; Dai, XM; Zhou, M; Tai, LM; Zhu, YG; LaDu, MJ; Chen, XC

APOE4 enhances age-dependent decline in cognitive function by down-regulating an NMDA receptor pathway in EFAD-Tg mice

MOL NEURODEGENER

Article

 

Hi Aaron,

The papers you listed are certainly landmarks. The problem I see is that in a field as broad as AD research, is it fair to try to reduce the entire field to the 40 top papers? Also, it is possible that some fields of AD (APOE genetics) tend to procude higher rates of citations than, let's say, membrane-amyloid interactions. In a field this broad, you'd almost have to look at the top 40 papers dealing with genetics, AB, tau, neuro imaging, etc.

WRT retracted papers, that is a very difficult question to answer. Some papers are retracted because a junior author fabricated results. Should the entire paper be thrown out because, let's say, one figure of Western Blots was altered, but the remainder of the data is sound? I would tend to err on the side of caution and leave the whole thing out. Where there's smoke, there's fire. I respect if others disagree though.

I ran into a someone similar problem in preparing part 1 of this review. The paper by Cramer et al. (Cramer, Science, 2012) dealing with bexarotene to clear AB, contained results that other groups had difficulty in replicating. I included the Cramer paper in the review but also made a note that numerous other groups could not replicate the results in their entirety. This lets the reader decide for themselves how they wish to weight the evidence.

Pages