Systematic Literature Review about Software for References Extraction

The protocol reports the different and complementary steps required in order to make a systematic literature review. In this case, the topic of the research is focused on the retrieval of software for references extraction from PDF files. The structure of the review is split into two parts, search strategies and quality assessment. The search strategies are based one on the seed papers and the other on the keywords search; to each of both has been devoted a section. They are two complementary ways to maximize the retrieval of papers relevant with respect to the topic of the research question . Then, the quality assessment is positioned in the penultimate section. But before the beginning of the research there is the need to prepare the materials for the research. Indeed, this first step focuses on the creation of the materials that will be used later in the research process to keep track of the materials retrieved.

Create three files:

• “ Papers_to_keep ” will contain the papers compliant with the General IC for papers and with at least one of the points of the IC for screening procedure on papers . These papers will be further analyzed by reading the full text.
• “ Papers_to_discard ”, instead, includes all the files excluded from the research because not compliant with at least one point of the General IC for papers or with neither of the points in the IC for screening procedure on papers . This file is useful to keep track of the discarded papers and to check whether a newly retrieved paper appears in that list, so that we already know it must be discarded.
• “ Papers_to_analyze ” can be considered as an intermediate file. Indeed, this file will be populated with the papers retrieved in each specific step of the research process. After the end of that research step, each of the papers listed in this file will be 1. moved to “ Papers_to_keep ” or to “ Papers_to_discard ” on the basis of some characteristics that will be defined later on; 2. deleted from “ Papers_to_analyze ”. Therefore, since these two steps will be iterated on all the steps of the research process, at the end of the research “ Papers_to_analyze ” will be empty. All these three files will have the same structure, i.e. a 8 columns table containing the information necessary to fully identify a paper: Title, DOI (or link if the DOI is not available), Author(s), Publication Date, Venue, Volume, Issue, Pages. All this information (or only some of them, depending on the kind of publication), are necessary in order to completely identify a production and to keep track of all the information that could be useful also in further steps of the review.

The following table is an example of how the files will look like, the examples are taken from "Construction of the Literature Graph in Semantic Scholar" by Waleed et al. from the ‘ Seed papers list’ (from a book) and "Guidance on Conducting a Systematic Literature Review" by Xiao and Watson (from a journal).

A	B	C	D	E	F	G	H
Construction of the Literature Graph in Semantic Scholar	10.18653/v1/N18-3011	Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey,Ž Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan,Ž Madeleine van Zuylen, and Oren Etzioni	06/2018	Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers)	N/A	N/A	84–91
Guidance on Conducting a Systematic Literature Review	10.1177/0739456X17723971	Yu Xiao, Maria Watson	2019	Journal of Planning Education and Research	39	1	93-112

The second step of the research is based on the seed papers , i.e. papers identified before the beginning of the research. These papers give place to the first typology of research, based on an iterative research performed on these initial papers and on the successively retrieved ones. Once verified whether the seed papers meet some specified requirements, the focus will be shifted to the research and analysis of the papers where the seed papers are cited (forward) and of the papers that the seed papers cite (backward) to retrieve useful papers for the research.

Iteration on the seed papers. For each of the seed papers listed in ‘ Seed papers list’ (in the section "Materials") we make an analysis of the seed paper: we check whether the selected paper meets all the requirements pointed out in General IC for papers. There are two possibilities:

• If the work meets the specified criteria we keep the selected seed paper and we create a new row in “ Papers_to_analyze ” and add there the information about the selected article, as shown in the table at Step 1 .
• Instead, if the paper does not meet the specified criteria, we do not keep the selected paper and we create a new row in “Papers_to_discard” and add there the information of the selected article, as shown in the table at Step 1 . Then, in both cases, we jump to the next paper in ‘ Seed papers list’ or, if we were analysing the last item of the list, we can procede with the iteration on “Papers_to_analyze” Papers_to_analyze ”, in the next step.

Iteration on “Papers_to_analyze” . For each of the papers listed in “Papers_to_analyze” we make:

1. Analysis of the paper: check its title, keywords and abstract. In case the article seems useful for the purposes of the research (i.e. it meets at least one of the requirements listed in IC for screening procedure on papers):

• create a new row in “Papers_to_keep” and add there the information of the selected article, as shown in the table at Step 1 ;

• remove the referring row in “Paper s_to_analyze ”;

• proceed with the backward and forward search (points 2 and 3). Otherwise:

• create a new row in “Papers_to_discard” and add there the information of the selected article, as shown in the table at Step 1 go to step #1.1 ;

• remove the referring row in “ Paper s_to_analyze”;

• jump to the next paper in “ Papers_to_analyze ” (restart from point 1 with the next paper).

2. Backward search . Check the articles cited by the reference papers by looking at the list of references at the end of the article. Then, for each of the articles retrieved with the backward search, if its title is not in “Papers_to_keep”, “Papers_to_discard” or “ Papers_to_analyze ” and if it meets all the requirements listed in General I C for papers , add it to “ Papers_to_analyze ”.

3. Forward search . The forward search is carried out both manually and with the help of some platforms which have that specific functionality (see “ Free open platforms ” in the section "Materials"). Then, for each of the articles retrieved with the forward search, if its title is not in “Papers_to_keep”, “Papers_to_discard” or “ Papers_to_analyze ” and if it meets all the requirements listed in General I C for papers , add it to “ Papers_to_analyze ”.

When no new article is retrieved with this search method or when there are no more rows in the table of “ Papers_to_analyze ”, we can consider finished the research through seed papers and we can skip to the next section, "Search with Keywords".

This step, based on a search with keywords through the most used openly available research platforms, directly follows the previous one, based on the search with seed papers, with a complementary perspective. . Indeed, this approach is able to fill an eventual gap of papers and publications left by the previous step. At the end of this section, all the available documents should be retrieved.

This passage is about the further keywords that will be added to the original list of keywords during the research process. Indeed, there is a possibility that the keywords identified before the beginning of the research won't be enough in order to carry out all the possible information necessary in order to completely review all the software or the subtopics identified throught the retrieved papers. Therefore there could be the necessity to create new keywords, in order to search for a specific topic or to answer to a specific need of the research, and those keywords should be added to the original list of keywords in order to keep track of all the keywords used.

Since in some cases this will be an iterative step, there are two possible cases:

1. It is the first iteration on this step, there is no new keyword to add : just skip the passage and go to the next step, 3.2;
2. It is the second, or more, time in which we go through this step : we need to add the further keywords which we want to use in the keyword search to the keyword list.

Iteration on the keywords . For each of the platforms under the voice “ Free open platforms ” in the section "Materials": Research : Perform on that platform a research for each of the keywords listed in “Keywords list ” (in the section "Materials"). For each of the articles retrieved through each keyword, if its title is not in “Papers_to_keep”, “Papers_to_discard” or “ Papers_to_analyze ”, and it meets all the requirements listed in General IC for papers , add it to “ Papers_to_analyze ”.

Iteration on the retrieved papers. For all the publications listed in “ Papers_to_analyze ”:

1. Analysis of retrieved articles . Check the title, keywords and abstract of the selected article. If the article meets at least one of the requirements listed in IC for screening procedure on papers :* create a new row in “Papers_to_keep” and add there the information of the selected article, as shown in the table at Step 1 ;

• remove the referring row in “ Papers_to_analyze ”;
• proceed with the backward and forward search (points 2 and 3).

Otherwise, if the papers does not meet any of the requirements in IC for screening procedure on papers : * create a new row in “Papers_to_discard” and add there the information of the selected article, as shown in the table at Step 1 .

• remove the referring row in “ Papers_to_analyze ”;
• jump to the next paper in “ Papers_to_analyze ” (restart from point 1 with the next paper).

2. Backward search . Look at the articles cited by the reference paper by looking at the list of references at the end of the article. Then, for each of the articles retrieved with the backward research, if its title is not in “Papers_to_keep”, “Papers_to_discard” or “ Papers_to_analyze ” and if it meets all the requirements listed in General IC for papers , add it to “ Papers_to_analyze ”.

3. Forward search . The forward search is carried out both manually and with the help of some platforms which have that specific functionality (see “ Free open platforms ” in the section "Materials"). Then, for each of the articles retrieved with the forward search, if its title is not in “Papers_to_keep”, “Papers_to_discard” or “Papers_to_analyze” and if it meets all the requirements listed in General IC for papers , add it to “Papers_to_analyze”.

When no new keyword leads to new results or when there are no more rows in the table of “Papers_to_analyze”, we can consider finished the research. We can say that almost all the articles regarding this topic have been found and we can proceed by reading the full text and extracting the information useful to retrieve the software and select the best one with respect to the purposes of the research question.

The step of the quality assessment takes as input the two files produced in the previous step: “Papers_to_keep” and “Papers_to_discard”. The output file will be these two, with some eventual modifications due to the identification or not of parameters defined in section 1.3 ( IC for the papers full text ) in the full text of the articles accepted in the previous passage, and other two files (“papers”, “software”) which allow a further distinction on the basis of the content of the articles.

This step can be considered iterative with respect to the previous one, . Indeed, since only in this section it is fully clear the content of the articles identified during the research process, thanks to the full-text reading, it may result in the necessity of providing more information on a topic or software about which only few or not enough information was retrieved in the previous research steps.

Before the beginning of the research we need to create the materials which will be necessary in order to carry out the task of the annotations of the full-text reading. First of all we have to create two files:* “Papers”: this file will include all the articles which, after the first selection in the research process, will stick also to the full-text parameters, and therefore can be considered useful in order to answer the research question. The format will be again that of the table, in this case of a three columns table, where each column will correspond to the following values: topic, explicit mention of a relevant software, explicit mention of other software. This file will be particularly useful after the end of the research to connect the retrieved papers to their referring resources, in order to investigate the entirety of the software and to collect the previous and relative works.

A	B	C	D
Costruction of the Literature Graph in Semantic Scholar	Use of the software science parse in order to retrieve information from PDF files: title, abstract and citations.	ScienceParse	Apache PDF box

• “Software”: it is a file aimed at collecting the software retrieved through the literature review. It is a 2 columns table which contains the information of the name of the software and of its validity status. In this way it is possible to collect all the software and, together with the information about its validity, analyse and classify them after the end of the research.

A	B	C
ScienceParse	https://github.com/allenai/science-parse	Valid

Also, both the files must be created on a digital file so that clusters of topics can be created. Indeed this action has a different objective for the two files:1. For the first file, "Papers", the reason for creating clusters is that of ordering the articles on the basis of their topic (i.e. the software, in the third column), so that in a following step they can by taken into consideration in groups defined by the topic and, also, it is easier to find the articles describing a specific software by grouping them.

1. Instead, in the case of the file "Software", it is used in order to distinguish the valid software from the invalid ones.

For each of the papers in “Papers_to_keep” read its full text. We outline three possible cases derived from the possible outcomes of the verification whether the articles include the points listed in IC for the papers full text or not. Discard the paper . If the selected paper does not meet any of the requirements listed in IC for the papers full text:

1. remove it from “Papers_to_keep” and add it to “Papers_to_discard”. Indeed, the parameters listed in IC for the papers full text are still necessary in order to distinguish between the papers useful for the purposes of the research and the ones that are not so, even if they could stick to the inclusion criteria of previous steps;
2. do not add anything to the file "Papers" nor to "Software". Since the file does not contain useful information for the research, we must not take it into account in the list of the useful papers and software.

Add the papers to "Papers" and "Software" ( invalid status). If the paper sticks to the second point in IC for the papers full text:

1. create a new row in the file “Papers” and add there its title and information. Indeed, even if it is about an invalid software it is to be considered in order to make a complete review of the software which have been used and eventually investigating the reasons why it cannot be used anymore;
2. if the software name is not already in the file "Software", create a new row, add the software name in the first cell and " invalid " in the second cell, i.e. the one of the status of the software. If the software is already in, skip this step.

Add the papers to "Papers" and "Software" (valid status) . If the paper contents meet at least one of the requirements of the first point in IC for the papers full text:

1. create a new row in the file “Papers” and add there its title and information;
2. if the software name is not already in the file "Software", create a new row, add the software name in the first cell and " valid " in the second cell, i.e. the one of the status of the software. If the software is already in, skip this step.

If at the end of the step 4.2 , there are some information missing about one or more software, we have to go back to step 3 and repeat the search with keywords, using the name or a concept related to the specific software of interest.

Otherwise, if no new information is required we can consider the research and selection of the literature over. At the end of the research and evaluation process we will return three non-empty output files: “Papers_to_keep”, "Papers" and "Software".