How to use Revman and GRADEPRo to Develop Evidence Profile
The purpose of this tutorial is to show you how you can use Revman — the review manager software produced by Cochrane Collaboration, and GRADEPro — a guidelines development web app, to conduct a brief evidence review. The aim of this tutorial is to cover how you can conduct an evidence review by using the following steps: (1) take a research question, (2) reframe this research question to a PICO formatted question, (3) search the literature databases, (4) identify on a first pass a few articles, and then, (5) review the full text of the articles to abstract data from them and process them. You can achieve the first few steps using a regular web browser and any word processing or text processing software (any plain text editor will do). The last few steps starting from right after you search the literature databases till synthesis of information, you can accomplish them in Revman.
In order to work with this exercise you will need to have the following:
- You must have access to a decent Internet connection.
- You should download and install Revman 5.3 software. You can do so by visiting the following website http://tech.cochrane.org/revman/download
- You should visit and create an account at the following webpage, Gradepro http://gdt.guidelinedevelopment.org/
- You should also have a ready list of questions (this could be any health related question. It is best you read the course syllabus and select a particular question to start with).
What are we going to do
In this tutorial we are going to learn how to use Revman software to write a comprehensive review or write answers to a particular research or study question. Specifically, in this tutorial we shall do the following:
- We shall first take a question. For this tutorial, we shall take an example question:
“What is the effectiveness of mindfulness meditation in reducing anxiety symptoms?”
- We shall then convert this question to a PICO formatted question. A PICO formatted question is a question that is formatted in “Person”, “Intervention”, “Comparator”, and “Outcomes” focused question.
- Next we shall search the literature databases to find articles that are relevant for this question. In our example, we shall search the literature database Pubmed to identify articles
- We shall then read the articles into Revman to process these articles further.
- We shall next select five randomised controlled trials. In our case, we shall examine articles that studied whether mindfulness meditation was effective in reducing symptoms of anxiety
- We shall then abstract information from these articles to develop Summary of Findings Tables and combine this with quality appraisal checklist based information to arrive at a reasonable argument as to whether the intervention that we have decided to study is effective for the outcome that we are investigating. In our example, this will be whether mindfulness meditation is effective in controlling anxiety symptoms in adults.
You can use the same approach to write your third assignment where you will be asking similar questions (for a list of questions, see the last section of your course syllabus). You should follow the steps sequentially.
Step I: Frame a PICO formatted question from your main simple worded question
This is your first step. In the first step, we shall convert a simple English language question to a PICO formatted question. A PICO formatted question is set up in the form of Population (P), Intervention (or Exposure, hence I or E), Comparator (or comparison group, hence C), and Outcomes (or O) based question. In our example, we have asked this question:
‘What is the effectiveness of Mindfulness Meditation for adults (when compared with other approaches) for reducing symptoms of Anxiety?’
We shall convert this question to a PICO formatted question. The PICO formatted question will take the form of
‘Compared with Drugs, or Placebo, or Cognitive Behavioural Therapy, or No Intervention, or all other interventions, what is the effectiveness of Mindfulness meditation in Adults with anxiety for reduction in anxiety symptoms?’
Note that the same question is now transformed into a PICO formatted question with the following components:
P = Adults with Anxiety
I = Mindfulness Meditation
C = All other forms of therapy including drugs, CBT, and others
O = Anxiety related scores or anxiety related outcomes
You should start with this approach and convert your own question to a PICO formatted question. We next move to Step II
Step II: Conduct a literature Search
After framing or reframing the question in terms of PICO, you should search literature databases using a combination of various terms. For example, for the question we framed “Mindfulness for Anxiety Symptoms”, we ran our search in Pubmed and we obtained the following table
If you study the search results table, you will see that we started with a set of broad search terms (the first search term). Then we gradually narrowed down the search terms by different types of combinations following the Boolean Logic. You should do the same with your own search terms that you will use for your PICO formatted question. At the end of the search exercise, you should come up with a similar table. You can generate a table like this from the Pubmed website. It will export the search results to a csv file. You can open a csv (comma separated value) file in a spreadsheet software.
How to generate this table in Pubmed
In order to generate a table such as this from Pubmed, do as follows:
- Visit Pubmed
- Run the search as described above
- Then click on “Advanced” link in the search query box and it will bring you to the following table
- Right click on the link “Download History” and save the file as a comma separated value file that you can open in a spreadsheet. Give it a name that you can remember easily
- Then open and format this table in a spreadsheet software
Step III: Select Five Articles that you will need to extract data
At the end of step II, you will get a list of titles and abstracts of primary research articles from the search results. In our search, we got a list of titles and abstracts of 20 articles. We found a list of twenty articles in the first pass as a nice ballpark figure that is easily manageable. The first thing you need to do is to read the for each of the 20 articles, read their titles and abstracts.
In order to that, you should get the list of titles and abstracts directly into Revman and then process it in Revman. Below we have outlined the steps of reading Medline formatted references directly into Revman. When you import them, give them the category “Studies awaiting classification”.
How to use Revman to import articles from Pubmed search to Revman
- Start a new document or review in Revman
- Run a search on Pubmed
- Select the list of final articles, then export it from Pubmed
- When you export from Pubmed, export as “File…” and then format “Medline”
- Open Revman
- Click Import >> References >> Click next till you receive the prompt to select the txt file >> choose the text file >> Select the format to be ‘Pubmed Medline Format’ >>
- Select the default destination as “References Pending Classification”
- Complete the process
How to classify the articles
After you import the full list of references and put them into “References Pending Classification”, you will need to decide whether you want to keep a particular study based on reading its title and abstract (‘Include’) or you want to drop it (‘Exclude’). Develop a plan of excluding an article. Start reading the title and decide whether the title gives you sufficient information about whether to keep or drop the article. Then read the abstract only if you cannot decide on the basis of the title if you should reject it. Your bias will be to keep as many articles as you can. Below, we have provided a plan that we use. You may use the following plan or write your own. This is what we are going to use for our classification.
This is the scheme we followed:
- Wrong Study. — The theme of the study is not meant to answer the research question you have. In our case, this would mean that if the study was not about either mindfulness meditation and its effectiveness in comparison with something else for treatment of anxiety, then we would reject this article. This information should be clear on reading either the title or the abstract.
- Wrong Population. — If the population under study do not match the population you have to study. For example, in our example, we are interested to study adults of all ages and gender. If a study did not include certain adults, or did not include people above the age of 18 years, or did not include men, then this study would be eliminated.
- Wrong Intervention. — If the study did not include the intervention at all. For instance, if we found a study that did not include mindfulness meditation either as a comparison or as a main intervention, then we could not include that study.
- Wrong Comparison Group. — If the study did not include a comparison group at all, or a comparison group we are not interested to study, then the study cannot be included in the review
- Wrong Outcomes. — If the outcomes is wrong, then we cannot take it as well.
- Wrong Study Type. — We cannot take articles that are opinion pieces, etc.
Start from the top (that is, number one clause). You should be able to eliminate a study based on one clause only, whichever is the first one that eliminates it. If the study passes all clauses, then retain the article for further analysis. At that point, it is a good idea to search, retrieve and store the PDFs of the articles.
In Revman, you will need to push these articles by selecting and moving the article to the section titled, “Excluded Studies”. As soon as you do so, under the Excluded Studies section, you will see next to each study is a form that asks why you decided to exclude the study. Enter the exclusion clause (for example using the exclusion clause above).
Develop a Flow Diagram to show how many studies excluded
You will need to show the justifications for rejecting articles visually. Lets say you started with 20 articles, you can choose to remove 15 articles, and indicate that only five of these articles will be used for further data abstraction. The elimination graph will look like as follows:
How to create the flow diagram in Revman
In Revman, do the following:
- Click on Figures, then select Add Figure
- Then select, “Study Flow Diagram (PRISMA Template)”
- Click Next and then “Finish”
- It will bring you a figure that you can edit that will look like the above diagram
Fill in this diagram, by replacing the hashtags with the number of articles and appropriate and relevant information. In our case, at the end of this exercise, we were left with five articles. For each of those five articles, you will need to obtain full text of the articles and then abstract information from them. We shall now explain how you can abstract information from five articles
Step IV: Extract information from the articles using the GRADE Approach
We will need full texts of the articles in the form of PDFs, EPUBs, or printouts. Next, we need to abstract information from the articles using the GRADE approach. The GRADE approach requires that you create what is known as or referred to as Evidence Profile. An Evidence Profile is equal to a composite of two types of information — quality appraisal and Summary of Findings Table. We shall appraise the quality of evidence directly in Revman for each article. Then, we shall use the Gradepro website to create Summary of Findings for either a single article or a bunch of articles.
A short note about GRADE process
The principle of GRADE is roughly as follows. You will need to create an evidence profile for a particular outcome. In GRADE approach, you develop an outcome focused way of judging quality of research articles, not just per article or per piece of research. In quality appraisal, you need to decide if you have sufficient confidence in the findings of the study on the basis of the description of the methods of this study. You will basically look for different ways in which the study results could be biased or based on the methods described in the study, you can think of an alternative explanation for the findings. For example, in case of a Randomised Controlled Trial, you will need to look for information in the methods section of the article whether the randomisation was done correctly, and whether the investigators were blinded. You will also need to study whether the outcomes reported in the study were directly measured or whether they were measured indirectly using some form of proxy measures, or surrogate measures. Further, when you are using more than one study for a particular outcome, you should look for whether the study findings were scattered and ‘all over the place’ or whether they were pointing to some common measures. In other words, are the study results consistent across the range or suite of articles that you have decided to study? Finally, you will need to decide if the results are imprecise. This can be done both for a single article and a group of articles. For a single article, you will report whether the study findings met the rules of statistical significance by judging the 95 percent confidence interval and the point estimate of effect. For a group of articles, you will need to pool together the results and test whether the studies are homogeneous or heterogeneous. The tests of homogeneity and heterogeneity are discussed in Systematic Reviews and Meta Analyses and beyond the scope of this tutorial at the moment. But if you read a bunch of articles pertinent for a particular outcome and find that the articles are very varied, then that is a good signal that the studies are not consistent. In turn, this is expressed in four decision points in Gradepro and Revman (1) whether the study or the studies had major limitations, (2) whether the outcomes were indirectly measures (indirectness), (3) If the findings were imprecise (Imprecision), and (4) Whether the findings were inconsistent (Inconsistence).
There is one more rule in using the GRADEpro framework. You will need to identify no more than seven outcomes. Each outcome is a measured outcome as reported in the article or suite of articles that you are reviewing. That is, you can combine the findings of more than one article to generate your evidence. You can also use only one article to review your evidence, that is also allowed. So, you abstract information from full text of the articles for a maximum of seven outcomes for five articles and write them down. This could be for one article, or per article, or it could be a composite of all five articles. For instance, it could be just one outcome per article, or it could be one outcome for a combination of two articles, or three articles, or all five articles. The exact pattern of reporting will depend on how you want to group your presentation and what outcomes have the articles and researchers/investigators have discussed in these articles. It is difficult to be prescriptive for us. We shall provide you examples.
After you have made a decision what is the quality of evidence presented per outcome based on one or more articles, it is time for you to abstract data from each article or sets of articles and present the data. We shall show that you can do this entirely in Revman or combine GRADEPRo and Revman. See what works best for you.
How to use Revman for entering this information
You will see that we have several articles from where we have to abstract data. Here is an example of filled in Table for the Arch article — the first one in our eligible list of articles.
You should do the same for your article. We recommend that you work with one article after another and in this way you will work your way through all the articles one by one. However, we recommend that for the best results, you also repeat the process in Gradepro as follows.
How to do the same thing in GRADEPro Website
- Visit http://gdt.guidelinedevelopment.org/ and create an account. We already mentioned this above and so we presume that you have already done so.
- Start new Project and give it a name. Remember to create a “Summary of Findings” table, not Evidence Portfolio.
- Then “Add Management Question”
- Fill in the blanks about the intervention and the outcome and the setting, and click the save button.
At the end of this, the form will look like this:
- Then double click on the Question and this will bring up a list of table headers
- Click Add outcomes button. This will bring up a series of buttons and cells which you will need to fill. See the screenshot below
There are different ways to fill in depending on what you want to do, for example, you may pool studies together (select the option “pooled” in that case, or you may want to select a single study). In our case, we shall fill in information from each article, so we shall select “single study” and we shall discuss from that perspective. I have filled in the results from the Arch (2013) article.
In order to do that, I used all information from the methods and the results section of the main article. You will do the same for each of the articles that you will be using for your review. So, start with the top of the methods and the results section and fill in the blanks starting from the leftmost box and working your way through the row. Here is my worked out example for the Arch 2013 article (download a copy of the Arch article and see for yourself.
Several points to note:
- In our study the authors noted that they had arranged for a three month follow up for everyone. However, the web app allows for mean, median range etc. Do what seems most appropriate and we for instance left a blank next to follow up of three months. Now this was because we were appraising only one study. If you appraise more than one study at a time for an outcome, then choose the mean or median or range as appropriate.
- Then fill in the number of studies. If you are appraising only one study at a time for a particular outcome, write “1” and move on.
- Fill each of the other boxes in turn, based on your best estimate after reading the corresponding methods section at this stage
- For one study, inconsistency is not an issue, but the programme assumes that you will be evaluating more than one study at a time, so if you are appraising one study only, just leave it as not significant or not a serious issue
- Next, assess if the outcome is measured indirectly. You will normally consider that an outcome is measured indirectly if you find that the information from the outcome is taken from and reported by people who were not participants in the study (like physician reported or nurse reported or provider reported outcomes). You can also consider that an outcome is reported indirectly if you find that the outcome is measured using surrogate measures. So, read how the outcomes are measured and decide accordingly. In our study, the anxiety score was reported by the patients or participants in this case themselves; according to the paper, it was “Psychiatric diagnoses were assessed with the clinician- administered MINI International Neuropsychiatric Interview for DSM-IV, and … Third and fourth year clinical psychology graduate students conducted the MINI after 10e15 h of training, including co-rating eight gold standard training interviews and demonstrating diag- nostic accuracy on three consecutive interviews.” (verbatim from the paper). So it was directly measured.
- We also noted that the imprecision was not serious as this was a single study and the way the outcomes was measured using an eight point scale, there was not much of a diversity. Look at the results section to understand this (to repeat, download a copy of the article and play with it).
- The last box is “Other considerations”. This is usually applicable only for pooled studies, but if you are using only a single study, just write “undetected’ for publication bias and the software will put “none” in the correct box. Note that when you do that, the study gets a 4 star quality rating under the quality box.
Next, you start with the Summary of Findings Table. This is again, filled in from left to right. Note here, the type of outcome becomes very important. If in your study the outcomes are measured as continuous variables, there will be one set of cells; on the other hand, if the outcomes are measured in dichotomous scale (that is yes or no, or events happened and did not happen), you will have another set of cells to fill in. So, be very careful in choosing your outcome scale. The same rule applies when you pool studies and be particularly attentive to when you pool a number of studies together. In our example, we had only one study with only one type of outcome that was measured in continuous measure. Also, for a continuous variable outcome, we cannot have relative effect (this is standard for dichotomous outcomes). We shall report absolute effect. For some studies, depending on how the authors would report it, this can be tricky. For example, in our case we found that the authors only reported beta coefficient for the slope and and a standard deviation. So we had to report the point estimate and the 95 percent confidence after reconstructing the range. So, to reconstruct the range of 95 percent confidence interval, we recommend you do point estimate and then plus or minus 1.96 times the standard error reported in the study. You will have to do this by hand.
After this, you will need to export the results. Export the results to the “Summary of Findings” format if you want to import them to Revman. Otherwise, you can also export to a regular Word document format, if that works for you. Here, we shall show how you can export to .Sof format so that you can import it back.
Next, open up Revman and import the Summary of Findings file by selecting the dot sof file and importing to Revman as a Summary of Findings Table. You are done.
We recommend that you do this first in GRADEPro website app, and then transfer this table directly to Revman. That way, you will have a standardised presentation of the tables as well. You can also do everything in Revman of course, but Revman assumes that you have or are going to abstract data from multiple articles.
Step V: Now synthesise all the information in the summary of findings table and prepare the report
In this step you will need to review the information contained in the summary of findings table and summarise the information you get to read. Comment on the outcome, and summarise the results you get to see on the summary of findings table.