Contents

Find and evaluate data sources, published research & Literature review strategies

Contents

Find and evaluate data sources, published research & Literature review strategies

Warning

These pages are currently under construction and will be updated continuously. Please visit these pages again in the next few weeks for further information.

There are nowadays various ways to find and organize literature and data online. While a simple google scholar search may seem sufficient to quickly get an idea about a specfic topic, this strategy becomes increasingly less feasible when trying to collecta and evaluate evidence in an unbiased manner.

In this lecture we will discuss:

  • How to find data sources/literature; How to find published research based on a specific data source

    • using Google Scholar, Web Of Science, PubMed

  • Basic search strategies

    • Forward/Backward search

  • Literature reviews

    • 1: Elaboration of a Review Question

    • 2: Elaboration of a Review Protocol

      1. Searching for All Eligible Studies

      1. Unbiased Screening of Eligible Studies

  • Literature maps

  • Digital tools for literature search/review

    • ResearchRabbit & more

  • Meta-analysis tools

    • Neurosynth/Neuroquery

  • Evaluate aka. read a scientific paper

  • FInding data

Goals

How to find data sources/literature; How to find published research based on a specific data source

Where to find scientific literature

We’ll start off by briefly discussing the 3 most popular literature search engines, follow the links to get a more in-depth understanding of each engine.

Google Scholar:

Google Scholar has easily become the go-to search engine for scientific literature and is a good place to start to easily find a specific paper. Unfortunately it offers less advanced search options than the two options below.

It is a free search engine providing access to scholarly literature, including articles, books, conference proceedings, and theses. It is a popular tool for researchers, students, and academics, as it offers a simple and user-friendly interface for discovering and accessing a comprehensive library of scholarly literature across most id not all disciplines. It also offers features such as citation tracking and alerting, which allows users to monitor and track the impact of their own research, as well as the work of others in their field. While impact is not something we should focus on too much, it helps identifying the seminal papers on a given topic.

To work with Google Scholar, simply access the platform through the Google Scholar website or through their Google account. They can use the search bar to enter keywords and other search terms, and then refine their results using the various filters available. Users can also set up alerts to be notified when new articles or publications related to their areas of interest become available.

google_scholar.png

Google Scholar generally provides links to full-text articles when available, as well as other related resources, such as author profiles and citation metrics. It add some additionally functionality for the specific search results:

  • Clicking on the “All versions” keyword below a search result link will help you identify other versions of the paper, if e.g. the first source is paywalled.

  • Via the “Cite” keyword you can also easily get a quick citation in a refernce style of your choice.


Clicking on the 3 Bars in the top right, next to the search bar reveals the Scholar menu. Here you can select avanced search, which will open a pop-up looking like this:

scholar_advanced.png

Pretty self-explanatory, but it allows you to match phrases either in title or text, exclude phrases and filter by author, publication and year range.

The advanced menu also allows you to setup E-mail notifications should new literature related to your field of interest be published. To set notifications, expand the menu on the top left, click on Alerts, click the Create Alert button and inout your search criteria, as well as an e-mail address.

To remove an alert, simply move to the alters menu as above and click the cancel link, behind the alert.

created_alert.png

Additionally, if you start publishing it’s a good idea to setip a profile, to track your own publications and citations, making it a useful tool for managing your scholarly online presence and output.

Web of Knowledge/Sciene

The Web of Knowledge/Sciene is a powerful online research database that provides access to a wide range of scholarly literature, including articles, conference proceedings, and patents. Developed by Clarivate Analytics, it offers a suite of tools for researchers, academics, and professionals to discover, analyze, and manage research data.

One of the key benefits of Web of Knowledge is its comprehensive coverage of a vast array of disciplines, including science, technology, social sciences, and humanities. It also provides users with advanced search options, including citation searching and author searching, allowing them to identify influential works and authors in their fields of interest.

One of the problems of the plattform is the need for a personal account and access through your institutional library. But if you’re willing to deal with this you are rewarded with very rich (but somewhat confusing) advanced search options. The platform also offers training and support resources to help users deal with this and get the most out of their research.

Youtube: Web of Science training

Intoduction course Web of Science

Web of Science Platform Training & Support

PubMed

PubMed is a free online database of biomedical literature, maintained by the United States National Library of Medicine. It offers access to millions of articles from thousands of journals, as well as books, conference proceedings, and other scientific resources in the fields of medicine, healthcare, and life sciences.

One of the key benefits of PubMed is its vast coverage of biomedical literature, making it a valuable resource for researchers, healthcare professionals, and students. It offers a range of search options, including keyword searches, author searches, and advanced search options that allow users to narrow down their results by date, article type, and other criteria.

PubMed also provides links to full-text articles when available, as well as other related resources, such as clinical trials and systematic reviews. It also offers tools for citation analysis and tracking the impact of research articles.

To work with PubMed, you can access the platform through the National Library of Medicine website. Simply use the search bar to enter keywords and other search terms and then refine your results using the various filters available. PubMed also offers a variety of tutorials, videos, and other resources to help users get the most out of their research.

Pubmed User Guide

Andvanced search tutorial

local libraries (shocked pikachu face)

Most libraries offer access to their Databases, as well as E-books and Journals that the university subscribed to. Simply google your university library, e.g. the online presence of the library of the Goethe-University Frankfurt.

You’ll have to rely on the information provided by the library in question to find out how to work with their database. E.g.

Goethe-University Frankfurt: Tips for literature search

You can also write or visit your local library/librarian. You’de be suprised how well versed librarians are in finding and managing information! (Well, it’s literally their job, but you know what i mean).


Basic search strategies

There are two basic strategies one can follow when searching for relevant literature, the forward search using keywords/phrasesor the backward search following the refernces from a specific paper. Backwards search can also involve reverse footnote mining/citation search, i.e. searching for other sources that have cited a particular article.


Genereal tips on optimizing your search behavior

It’s recommended to explore multiple combinations of search terms and operators during literature search and document every search query so that you or others can reproduce your liteature search later on. If you’re not taking notes for a literature review this can still be relevant info that you should include in your lab notebook.

Further you should use multiple search engines/databases for your literature search and tailor your search queries and vocabulary for each individual database.

Literature reviews

Systematic reviews & meta-analysis are used evaluate the published literature on a given topic to gain a solid understanding of scientific consenus and degrees of evidence for certain positions. While this is not necessary for every paper you’ll be writing it is a good idea to get comfortable with the basic ideas behind systematic reviews and how they correct search behavior for potetntial biases. This will also help you organize your literature search and make sure that you’ll not miss any important papers or waste time on papers that are irrelevant to your specific question/topic.

We’ll focus on the paper of Pigot & Polanin (2020): Methodological Guidance Paper: High-Quality Meta-Analysis in a Systematic Review. and for a more clinical/neurosience related perspective the work of Bolzan & Oliveira (2021): A compact guide to the systematic review and meta-analysis of the literature in neuroscience.

And explore the following steps:

  • 1: Elaboration of a Review Question

  • 2: Elaboration of a Review Protocol

    1. Searching for All Eligible Studies

    1. Unbiased Screening of Eligible Studies


Bolzan & Oliveira (2021): A compact guide to the systematic review and meta-analysis of the literature in neuroscience.


The first step when searching for literature should be defining your research question, i.e. defining what exact topic or method we’re trying to explore.


1: Elaboration of a Review Question

We’re mostly intrested in research questions, but these need to be necessarily defined into review questions once we start systematically searching for litearture on a specific topic.

This can be done with the help of the mnemonic tools, to turn a complex question into simple, comprehensive and direct terms. In clinical and neurocsciene the following mnemoic tools could be used to formulate a specific review question.

PICO (“has the intervention, I, changed the outcome, O, in the population, P, compared to control treatment, C?”)

SPICE (S – Setting; P –Population; I – Intervention; C – Comparison; E – Evaluation)

SPIDER (S – Sample or population of interest; PI – Phenomenon of Interest; D – Design; E – Evaluation; R – Research type)


Following this example you could also devise your own mnemonic tool suited to the specific needs of your research question.

2: Elaboration of a Review Protocol

After defining a our review question, we’ll construct a review protocol.

A review protocol serves as a comprehensive and structured plan that outlines the methodology and procedures that will be followed during a systematic review. The purpose of a review protocol is to provide a transparent and rigorous framework that ensures the systematic review is conducted in a consistent, reliable, and reproducible manner.

The protocol typically includes a clear research question, explicit inclusion and exclusion criteria for the selection of studies, the search strategy and databases that will be used, data extraction methods, quality assessment criteria, data synthesis and analysis methods, and a plan for reporting/documenting the results of your literature review.

At the heart of the review protocl are the search criteria, as well as the screening procedures. These will be explained in detail in the next section.

Pigot & Polanin (2020): Methodological Guidance Paper: High-Quality Meta-Analysis in a Systematic Review.

**Systematic reviews follow three basic steps: **

1. searching the literature

2. screening abstracts and full-text documents

3. coding included studies

Each of these three basic steps should be documented as much as possible and has to demonstrate that the researcher has included all eligible studies on a specific topic/question. We’ll focus mostly on the first two points in the following chapter.

We’ll explore the basic strategies of systemic review to help ensure that we were thorough and unbiased in the review and selection of potential papers of interest.

3. Searching for All Eligible Studies

A search must necessarily be systematic, comprehensive and well documented to allow for the reproducibility of the search process and to make sure that no relevant results were excluded.


  • systematic - method behind the search:

When conducting a search for relevant studies, it’s important include all:

  • Terms: relevant keywords to identify the research topic.

  • Strings/Phrases: combining terms using logical operators to refine the search

  • Limiters: applying specific filters to limit the search results to the most relevant

  • Tools: utilizing tools such as Boolean operators, proximity searching, and wildcards to fine-tune the search strategy.

  • Databases: selecting the most relevant databases to search for the study.

Terms and Phrases need to be sensitive enough to capture all relevant studies, meaning that e.g. the terms used capture as much as possible of the studies relevant to the topic. In literature search it’s recommended to tailor your search terms/syntax to maximizie sensitivity (proportion of studies identified divided to total number of relevant studies in existence) while being prepared to accept low precision (the proportion of relevant studies to the total number of studies identified) (Higgins et al., 2022). This is necessary as an increase in sensitivity will naturally lead to a reduction in precision,

The search strategy should be informed by the review question and can be based on the mnemonic tool used.

E.g. for the PICO tool, the search terms can be documented based on the number of retrievals (n) for all relevant terms (Bolzan & Oliveira, 2021):

  1. P: Terms related to (population P or synonyms) = (n(p))

  2. I: Terms related to (intervention I or synonyms) = (n(i))

  3. C: Terms related to (control C or synonyms) = (n(c))

  4. O: Terms related to (outcome O or synonyms) = (n(o))

  5. PICO: Combination of terms used in individual searches, i.e., (population P or synonyms) and (intervention I or synonyms) and (control C or synonyms) and (outcome O or synonyms) = (n(pico))

comprehensive - breadth of the search:

require terms unique to several disciplines include both online databases that index published literature as well as sources such as Google Scholar and Web of Science also includes strategies such as retrospective reference harvesting, prospective forward citation searching, and contacting prominent or active authors in the field further attempt to identify unpublished literature such as dissertations and reports from independent research firms

Further we have to attempte to identify unpublished literature, such as dissertations and thesis or reports from independent research firms. This is neccessary to identifiy and ideally reduce publication bias, known as the phneomenon of studies published in research journlas to report larger effect sizes and proprotionally more statistically significant effects as should be expetced (Polanin, Tanner-Smith, & Hennessy, 2016)

There exist in-depth protocols and tools that aim to estimmated the effect of publication bias on a specific topic:


Cochrane: A revised tool to assess risk of bias in randomized trials (RoB 2)

Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach

To evaluate the publication bias you also may use the Replicability Index, which provides a number of tools and information on how to spot problematic trends in published research.

well documented :

The documentation of your review process should involve Include all

  1. The review question

  2. Brief introduction to the research subject

  3. Description of the strategies employes to obtain publications and filter relevant studies

  4. All search queries, including which site was used and the date of the search

You can document your search process using different methods, the easiest is to include your search in a Lab-Notebbook.

4. Unbiased Screening of Eligible Studies

After identifying all relevant studies on a given topic, it’s necessary to screen the abstracts, full-text and citations of the studies in question.

The screening processs involves the creation of a screening tools and following applying them to different apects of the selected literature.

First, is the creation of a screening tool/strategy to filter study abstracts and titles for ineligible articles such as essays or non-empirical studies. This helps you organize and sort the abstracts based on how likely it is that they should be included in your review.

The screening tool consists of a check-list of multiple clear and concise questions that lead to inclusion or exclusion of a study. Following Polaning et al. (2019) The questions/ items should be

* (a) objective, 
* (b) “single-barreled, i.e only related to a single asepct of the abstract: E.g. “Does the abstract indicate that particpants were sampled from the general population?”
* (c) use the same sentence structure,
* (d) include yes/no/unsure answers only

This process can iclude free text-mining software like Abstrackr (Wallace et al., 2012).

After completing title and abstract screening gather all the included full-text PDFs. This step is commonly called retrieval.

The full-text screening process is similar to the abstract screening process and involves the following steps: * Developing a screening tool * Screening each article * Making a decision about whether it should be included

The questions in your screening tool should be informed by your research question and can include topics such as the methods used or the sampled population. Unfortunately there is no validated, dependable text-mining tool to assist with the full-text screening process at the moment.

The resulting collection of studies should then be a solid base for future research. An easy way to keep track of the literature in question is to add them to a citation manager such as Zotero. A so created Zotero library can be used to create a list of references on the fly, shared online or integrated with a number of other digital tools. Find out more about how to work with Zotero in the chapter Project design.

Further Reading

The following studies provide example protocols of screening strategies in neuroscience:

Ramos-Hryb AB, Bahor Z, McCann S, et al. Protocol for a systematic review and meta-analysis of data from preclinical studies employing forced swimming test: an update. BMJ Open Science. 2019;3:e000035 https://openscience.bmj.com/content/3/1/e000043.abstract

Bolzan JA, Lino de Oliveira C. Protocol for systematic review and meta-analysis of the evidence linking hippocampal neurogenesis to the effects of antidepressants on mood and behaviour. BMJ Open Science. 2021;5:e100077 https://doi.org/10.1136/bmjos-2020-100077

Hohls JK, Konig H, Quirke E, Hajek A. Association between anxiety, depression and quality of life - a systematic review of evidence from longitudinal studies. PROSPERO 2018 CRD42018108008 Available from: https://www.crd.york.ac.uk/prospero/display_record.php ?ID=CRD42018108008

Excercise:

Create a Review question
Create a Review Protocol
Write a table for the documentation of your search process containing:
    1. The review question
    2. Brief introduction to the research subject
    3. Description of the strategies employes to obtain publications and filter relevant studies
    4. All search queries, including which site was used and the date of the search

Literature maps

A newer, graph-based approach to literature search is the creation and use of literature maps. If there is one takeaway from this lesson, it should be the use of literature maps for your future research!

https://media1.giphy.com/media/icgArcntfH5C0/giphy.gif?cid=ecf05e477osuhygwt3o5mqowvvtu798736t9f5gh0g6nlddq&rid=giphy.gif&ct=g

A literature map is a visual representation of the relationships between different topics, themes, and concepts in a specific field of study. It can be used to explore the connections and gaps between different areas of research, identify key authors and publications, and gain a broader understanding of the research landscape.

You could do this by hand, but this would be rather time-consuming, so we’ll introduce some tools for the creation of literature maps in the next section.

png depicting the file structure of the course template repository

Literature maps are useful because they help researchers and students to:


>

Identify gaps in the research: By visualizing the relationships between different topics and subfields, literature maps can help to identify areas where there is a lack of research, or where new research could make a significant contribution.

Discover new connections and relationships: Literature maps can help to uncover new connections and relationships between different topics and fields, leading to new insights and avenues for research.

Gain a broader perspective: By providing a visual overview of the research landscape, literature maps can help researchers and students to gain a broader perspective on their field of study, and to see how their own work fits into the larger context.

To make the most of a literature map, it is important to:


Choose the right tools: There are a variety of tools available for creating literature maps, ranging from simple mind-mapping software to more complex data visualization tools. Choose a tool that is appropriate for your needs and skill level.

Start with a clear research question: A literature map is only useful if it is focused on a specific research question or topic. Start by defining your research question or area of interest, and then use the literature map to explore the relationships between different concepts and subfields.

Be critical: While literature maps can be a useful tool for exploring the research landscape, it is important to be critical of the data sources and assumptions that underlie the map. Always evaluate the quality and relevance of the sources you are using, and be aware of any biases or limitations in the data.

Collaborate: Literature maps can be a valuable tool for collaboration, allowing researchers and students to share their knowledge and insights, and to work together to explore new connections and relationships between different topics and fields.

Digital tools for literature search/review

While it’s always sensible to think about how to develop a review question and follow a pre-defined search strategy “by hand”. There are nowadays quite a few tools out there to help you make your search more efficient/organized.

Most of the following tools automatically include literature maps and the ability to search different databases. While they have slightly different functionalities and foci, they are rather similiar, so it’s best to just test what works best for you

Researchrabbit:

To get started we’ll exlore one of these tools more closely.

The free, self proclaimed “most powerful discovery app ever built for researchers!” and “new spotify for academic papers”, is an AI-based app that allows you to search for literature in multiple databases. It mainly works on the idea of structuring your literature into collections for every separate project.

ResearchRabbit builds visual representations of your collections showing the relative importance of paper, as well as the connections between different papers based on e.g. citations (where arrows always point towards the paper cited).


png depicting the file structure of the course template repository

Exploring the literature map

It improves on manually adding papers to your collection, by automatically suggesting related works and authors. That is as soon as a paper is added to your collection, Research Rabbit initiates the process of generating suggested additions. With each additional paper added, the accuracy of these recommendations gradually improves.

The recommendation system works somewhat like Spotify, by providing more information, i.e. by adding more papers, the app will get a better idea on what exactly the keyworded terms you’re interested in are and further help you discover research you may not have been aware was related to your work. This works especially well when you’re fairly unfamiliar with a specific topic, by following the “bread-crumbs” based on a few papers that you’re interested in.

The main way to discover works related to the papers in your specific collection is by using the explore functions:


png depicting the file structure of the course template repository

Similar works

If your looking for papers that are related to your collection you simply click the similar work button and a new literature map will be fit into the original from your collection. Works from your collection will be displayed as green, while works outside your collection will be displayes as blue. By hovering over the elements you can see show their respective connections, the authors (first or last author) name and year of publication.


png depicting the file structure of the course template repository

Exploring citations

Further clicking on one of the similar works will open a second side-bar containing more specific information on the paper in question, e.g. the title, all authors names and the abstract. But we’re not stopping there, as we can now also explore related works to the paper in question, e.g. by looking at all papers that have cited this specific paper using the all citations buttons. This for example reveals that the paper in the screenshot is apparently drawing information from different fields of research as the literature map is distributed into multiple clusters of separate maps.


png depicting the file structure of the course template repository

Down the Rabbit hole

Like the name suggests you can suggest this process infintely (probably not a good idea) and find different rabbit-holes of literature.

By selecting help button on the right hand side, you’ll open the help window. Here you’ll find frequently asked questions (ResearchRabbit FAQ), e.g. where does the data come from or how do i read the maps, as well as a youtube playlist of community created in-depth content and the Feature Overview tour, which will help you get started with ResearchRabbit.



Meta-analysis tools

For the neuroscientifically inclined there is another way to find relevant literature, by using web-based platforms that provides tools for automated meta-analysis of neuroimaging studies.

png depicting the file structure of the course template repository

These tools tend to use natural language processing (NLP) techniques to extract information from published studies and generate statistical summaries of brain activity patterns across different cognitive processes and experimental conditions. These summaries, called “maps,” can be used to identify brain regions that are commonly activated or deactivated across multiple studies, and to generate hypotheses about the functional organization of the brain. Neurosynth can be used to e.g. inform the selection of brain regions for further investigation, but also provides a weighted list on the contribution of separate papers, each with a separate brain map. Following you can use these papers to base further meta-anylsis on the platform or add them to your collection in e.g. ResearchRabbit.

You can further export both graphs and the list of papers and include them in your documentation/reseaerch.

Two platforms for this are:

Neuroquery

Neurosyth


Evaluate aka. read a scientific paper

Of course an essential part on evaluating literature is evaluating the quality of a paper itself.

Some overall tips would be:

  1. First skim the article and identify it’s structure

  2. Distinguish main points/graphs

  3. Take notes/annotate along the way

  4. Read carefully and re-evaulate your annotations

The following is an in-depth guide on how to seriously engage with a scientific paper to gain the maximum of information and benefit from it. This may seem discouraging due to the sheer scope of the task, but it’s a great idea to train and ultimately automatize this process.

The main take-away from the following section are the Six questions you should ask yourself about every paper, to more deeply engage with the content and help kickstart your memory

Ask six questions (Carey et al., 2020)

Regarding the entire work, including all sections ask yourself the following questions:

  1. What do the author(s) want to know (motivation)?

  2. What did they do (approach/methods)?

  3. Why was it done that way (context within the field)?

  4. What do the results show (figures and data tables)?

  5. How did the author(s) interpret the results (interpretation/discussion)?

  6. What should be done next?

Of course, it’s not necessary to engage this deeply with every paper you come across. Don’t waste your time on papers that don’t relate to what you wan’t to find out. In this case concentrate on the first 3 steps presented below to evaluate the worth of a given paper for your work.

Ten simple rules for reading a scientific paper (Carey et al., 2020)

Rule 1: Pick your reading goal

Rule 2: Understand the author’s goal

Rule 3: Ask six questions

Rule 4: Unpack each figure and table

Rule 5: Understand the formatting intentions

Rule 6: Be critical

Rule 7: Be kind

Rule 8: Be ready to go the extra mile

Rule 9: Talk about it

Rule 10: Build on it

Rule 1: Pick your reading goal

The motivation to a read a paper and the desired outcome (can) influence how one reads a paper different priorities for different desired outcomes. Do you want to get a complete overview of the paper or are you only interested in e.g. the methods used?

Evaluate your goal and adapt your reading accordingly (e.g. skim instead of read in detail etc.)

Rule 2: Understand the author’s goal

Ask yourself:

Why did the authors want to share a given study?

  • sientific field, scientific interest, author’s research

  • helps with interpreting data & understand authors’ interpretation

In what form is the information presented?

  • type of article: methods, commentary, resources, research, review, etc.

  • formatting & content

  • intended purpose further shapes understanding of author’s goal

Rule 3: Ask six questions

Regarding the entire work, including all sections ask yourself the following questions:

  1. What do the author(s) want to know (motivation)?

  2. What did they do (approach/methods)?

  3. Why was it done that way (context within the field)?

  4. What do the results show (figures and data tables)?

  5. How did the author(s) interpret the results (interpretation/discussion)?

  6. What should be done next?

Rule 4: Unpack each figure and table

Skim/evaluate figures & tables before actually reading the paper. Evaluate:

  • intelligibility, complexity

  • x- and y-axes, color scheme, statistical approach (if one was used), the particular plotting approach

For each table containing data evaluate:

  • intelligibility, complexity

  • experimental groups & variables

  • presented statistics

Think about six questions of rule 3 & formulate the key outcome/take home message!

Rule 5: Understand the formatting intentions

There are distinct motivations and content for the distinct sections of a paper as discussed above. These may be further influenced by article type, journal policies, etc. but usually reamin the same for every scientific paper.

Depending on what are you looking for, check different sections:

  • Overview of the results? → Abstract, Conclusion, Figures

  • Method? → Methods, Supplementary material

  • Results? → Methods, Results, Tables, Figures

  • Interpretation → Discussion

  • Overview of the literature? → Introduction

Rule 6: Be critical

Test the strength of conclusion by critically evaluating everything: hypotheses, methods, results, interpretation

  • self-fulfilling prophecies & expectations?

    • assumptions about data, results & interpretation

    • alignment of behavior

    • alternative hypothesis

Evaluate the paper in regards to open science practice and how it deals with the replicability/reproducibility crisis, publication bias of prior literature, QRP (questionable research practices, etc.).

  • use the Replicability Index, which provides a number of tools and information on how to spot problematic trends in published research.

  • critically evaluate the reliability of results

Rule 7: Be kind

  • If possible: Give the benefit of the doubt, most folks try to give their best

  • Minor things (typos, reference errors, certain visualizations, etc.) shouldn’t guide/influence evaluation of data, results, interpretation

  • make your critique about facts/data & not beliefs

  • be constructive & objective

Rule 8: Be ready to go the extra mile

Don’t expect everything necessary to fully understand a given paper to be present in it (unfortunately most papers are not written for people new to the field):

  • look up terms, definitions, models, etc.

  • consult cited references & prior work

  • evaluate supplementary materials (also check-out the authors other online presences, as e.g. some papers come with in-depth publications, e.g. https://oreoni.github.io/index.html)

Potentially read a paper more than once, each time with different goals (see rule 1): overview, understanding, evaluation, etc.

Annotate, annotate, annotate:

  • mark questions, unclear paragraphs, connections between figures, etc.

  • share/create annotations with others (e.g. via google docs etc.)

Rule 9: Talk about it

Prepare & engage with articles:

  • attend journal clubs

  • give paper presentations to peers

Discuss work with your colleagues, mentors, friends, families, etc. They will provide:

  • different points of view

  • different levels of discussion

  • different foci

Check open discussions, e.g. on twitter.

Rule 10: Build on it

Think about the bigger picture:

  • how does the paper fit within current research and prior research work

  • situate paper regarding your existing knowledge & new insights

  • use everything to inform your own research (i.e. think outside your own discipline)

Think about the paper and prior research work as building blocks that together create knowledge and the basis for further research.

Think about how aspects the paper (methods, sample, etc.) can be integrated into your research


FInding data

Gregory et al. (2018) provide Eleven quick Tips for Finding Research Data”. Well go through the most important tips in the followin section and list some consideration of our own, but for more in-depth info check out the paper itself.

Tip 1: Think about the data you need and why you need them.

Evaluate whether you seek data as the basis for a new study, for comparison or validation with exiting results/data, or to simply explore or simulate the behavior or characteristics of certain types of data.

Next make a list of the characteristics your data should fulfill for the above identified purpose.

Rewuirements could include:

  • task/process or subject in question

  • data format (e.g. behavioral, questionaire, EEG, fMRI etc.)

  • spatial or temporal coverage (location/region and year or age-range etc.)

  • availability (free, “upon request”)

Tip 2: Select the most appropriate resource.

While you can find data in abvailable online, e.g. in:

Open Science Repositories: Repositories such as OSF, OpenNeuro, the OpenfMRI database, and the IEEE DataPort provide open access to MRI and EEG datasets, as well as other neuroimaging data.

Research Data Repositories: Zenodo, Figshare, and other research data repositories allow scientists to store, share, and publish their data in an open and transparent manner. These repositories are often committed to open access principles and provide a centralized location for data and metadata, as well as version control and preservation features.

or explore databases specific to your field of study: For neuroscience the most prominent wold be:

You can further consult research data search engines:

Research data search engines help researchers discover and access (ideally high-quality) data repositories across different scientific disciplines.

One such service is FAIRsharing:

Fairsharing platform aims to promote the principles of Findability, Accessibility, Interoperability, and Reusability (FAIR) in research data management by providing a curated collection of resources that comply with these standards. With FAIRsharing, researchers can search for datasets, ontologies, and other research resources based on their scientific domain, data type, and other relevant parameters. By facilitating the discovery and reuse of research data, FAIRsharing aims to foster collaboration, accelerate scientific progress, and maximize the impact of research outcomes.

Other research data search engines are for example:

A good starting point to find databases and search engines for your field of study are the constantly updated wikipedia List of academic databases and search engines and List of online databases.

Tip 3: Construct your query strategically.

The same strategies as for literature search apply, make sure to explore multiple combinations of search terms and operators during your search and document every search query. Additionally use multiple search engines/databases and tailor your search queries and vocabulary for each individual database.

Tip 4: Make the repository work for you.

Make yourself familiar with the plattform and investigate further resources on how to best work with the search engine/database. For example, follow the Openneuro User Guide to make the most out of the platform.

Tip 6: Assess data relevance and fitness -for -use.

Crossreference with the requirements provided in Tip 1. Also evaluate the data format (e.g. the BIDS standard] and check the quality parameters that most platforms include. If a dataset looks fit for your purposes, ivestigate further by downloading samples and implementing your own systematic quality assessment or analyzing the descriptive statistics provided with the dataset. This is not a trivial task so make sure to search for relevant tools and papers on the subject, and consult with your colleagues or advisors.

Tip 7: Save your search and data- source details.

Documentation!! Record your search queries, ideally on which date you searched, as well as the name of the dataset in question, it’s persistent identifier (digital object identifier (DOI); Global Unique Identifier (GUID).

Tip 8: Look for data services, not just data.

Data may not necessarily available without using a specific service and can only be accessed via an application programming interface (API).

Examples of such services include for example:

NSW Climate Data Portal

Google Earth Engine

Amazon Web Services (AWS) public datasets

Tip 9: Monitor the latest data.

New publications have at times the requirement to host their data on public plattforms, some platforms provide the ability to set keyword based e-mail alarms to keep you up to date. Otherwise monitor recent publications (e.g. as shown above for Google Scholar) and check back with the databases.

Tip 10: Treat sensitive data responsibly.

This speaks for itself, most data that you’ll find online is annonymized and cleared for the public, but if you ever work with data from colleagues or collect your own data you are required to apply to local data protection laws, especially when dealing with identifiable data of human subjects. For more info see our Lecture on data managment.

Tip 11: Give back (cite and share data).

Attribute: Identify how the authors of a dataset would like to be cited and apply this standard whereever possible

Give Feedback: Offer input to the creators of the data or the data repository regarding any problems related to the accessibility of the data, the quality of the data, or the completeness and interpretability of metadata (i.e. data describing the dataset, such as descriptive population statistics).

Make your own data open & accessible: For a guide on how to best do that see our Lecture on data managment.

TlDR:

References

Bolzan, J. A., & de Oliveira, C. L. A compact guide to the systematic review and meta-analysis of the literature in neuroscience. Journal for Reproducibility in Neuroscience, 2, https://doi.org/10.31885/jrn.2.2021.1669 (2021)

Carey, M. A., Steiner, K. L., & Petri Jr, W. A. (2020). Ten simple rules for reading a scientific paper. PLoS computational biology, 16(7), e1008032.

Gregory, K., Khalsa, S. J., Michener, W. K., Psomopoulos, F. E., De Waard, A., & Wu, M. (2018). Eleven quick tips for finding research data. PLoS Computational Biology, 14(4), e1006038.

Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019). Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses. Research Synthesis Methods, 10(3), 330-342.

Pigott, T. D. & Polanin, J. R. Methodological Guidance Paper: High-Quality Meta-Analysis in a Systematic Review. Review of Educational Research 90, 24–46 https://doi.org/10.3102/0034654319877153 (2020).

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012, January). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT international health informatics symposium (pp. 819-824).

previous

Project Design in the Digital Age

next

Scientific writing & digital tools