MRE1x_2020_Module_2_2b_The_literature_review-video

The literature review, understanding  and building on what has been done. So, what does a literature review and why should you do it? So, first of all, it's important in research to build on previous evidence and build on previous researchers work.

And that applies to different questions, on the one hand, what is the state of the art? If you want to do technology-oriented study. Also to understand, what will you do  different and what has not been done before?

So, what is your added value? What evidence is there that you can take into account by considerations for designing your study to have your added value, are there gaps in the research literature? And is there probably an underlying structure you should take into account when doing your experimental variations or how to collect your data?

So, typical questions, and that's at the core of your,  also already literature review, is to understand. What is it what do you want to do the literature review for what is the purpose of your literature review? Do you want to explore the state of the art?

Do you want to see different designs  of applications that have been done? What are alternative approaches to the problem? There might be really different takes on the research problem you want to look at.

What are the underlying models and theories  which you will find in related publications? What kind of theories have been used by previous authors? What are the empirical results so far and is there evidence you have to consider when you develop a hypothesis testing oriented study?

And what kind of evidence is there? Is there qualitative data? Is there quantitative data and so forth?

So, a general statement we use in a lot of our work is the PRISMA statement. I can really recommend this approach, which is  a four-phased model for a literature review, first in identifying your focus, then screening different. .

. sites and databases and screening the articles  by analyzing titles, abstracts and keywords, then defining a set of inclusion and exclusion criteria. And actually, this then by analyzing the full text articles, according to your inclusion and exclusion criteria, leads to a corpus of documents that you can actually then use in your analysis and report the results and analyze.

And so, this is a typical example. This was from a search and is from a publication  about mobile and inquiry-based learning. You see in the columns at the top the different  databases that have been taken into account.

ACM, Springer Web of Science,  Taylor Francis, Science Direct. And then you see the different results. A number of publications found by the different  queries which are considered the key queries.

So, for documenting your literature review, it's very important. Like we heard before, also in the desktop research already, when you do this systematically, it's even more important that you keep exact documentation of your findings  and the queries you use. So, in the second phase, then a screening, a typical procedure that's done in the screening is remove duplicates because you might find different publications that are indexed in different indices from the above.

In this case, five duplicate records were removed. And then in the eligibility phase, you look at inclusion and exclusion criteria and you again reduce the corpus. And this leads to the result  that you analyze later on.

So, in this study, 62 records have been used for deeper analysis. So how to work with search terms and keywords? And this is a kind of repetition, but I cannot  keep repeating this enough in the sense of use different types of combinations play with these combinations, look at the results, look at their distribution over time.

Do a quick scan of what articles pop up in a certain combination of and or not combinations work with synonyms, work with combinations of different application areas. So, you really have to learn and get a feel for the domain you're working in. And to combine these in the queries, then you  should really keep a record of the outcomes of these different publications  and these different searches.

So, you have to adopt the focus of your search to get a realistic body. It will be very tough to analyze  a corpus of thousand documents, that will take you really a long time if you have them to read them all full text. So, try to find different combinations, probably reduce the number of years you are considering.

So, reflect on the number of publications in the query and look at the results in the fast scan, adopt the keywords, use synonyms, record reflections and put comments to them. That's essential for remembering later on how your search process went forward. Here you see an example, what we typically do, you see a bit more complex query  and the first search strategy.

So, you're searching for title,  abstract and keys in that sense, in that case, for higher education or university or college or undergraduate. So different educational sections you use for terms. Also, you use synonyms programming,  computational thinking, computer science or even shortcuts as CS or CT, which might occur in the title and then third one in this search.

. . The author is looking for intelligent tutoring  or ITS or assessment or feedback or indicators.

So, they're different variants. And you see it gives you about nineteen thousand results. So that makes clear you should focus  in one or the other way to reduce this.

So, I think a logbook like this is really helpful later on to understand your processes and continuously work  on the refinement of your searches. Another example here, then you go down,  depending on the number of components you use and the AND combinations or the OR combinations you use, so you come to a realistic result of, let's say, 160 publications, probably in a bit a different kind of focus. But that also helps you later on to understand the different areas and foci of the research field you are in.

So now about the inclusion and exclusion criteria as a fourth step. So, you need to define them. And it is very helpful while you screen the articles in the phase before you already try to understand better  what you're searching for.

Is it, for example, only empirical studies you're searching for? Is it a certain methodology you're  looking for, a certain application domain? So, you can use this already to define inclusion and exclusion criteria, also to reduce and focus your corpus of documents.

And these are, this is one example. So, exclusion criteria might be you only  want journal publications, for example. So, exclude all theses, all books,  all workshops, all work in progress.

You want published solid research on your topic. You exclude, for example, also conceptual  papers not published and so on and so forth. But on the other hand, you can also explicitly  put things in your inclusion criteria.

So, in this case, an eligibility  and inclusion criteria was really on: are there game elements described? But these are some things you can just decide then also in a later stage, actually an eligibility stage. So, what is very nice about this scheme here?

You see already that there are different  inclusion and exclusion criteria you can explicitly define in the different phases of your literature study from the identification to the final inclusion in your corpus. So, when you then have your documents and you already tag them while understanding what your corpus develops around, it's important to develop a kind of classification system. And this can happen either bottom up that while reading your part through your articles, you have your key research questions  and on these key research questions, you identify the different variations  that are in your publications.

So, in this case, the author tries  to build the classification system on the different types of collaborative virtual reality applications that are out there. And you can see, for example, in the hardware section, he identifies different types of technologies like 3D on a surface, a cave system or head mounted displays. So, and while reading through your articles,  then you, of course, can extend this taxonomy if you come across a new kind of category or value for a category.

But in some cases, it also makes a lot of sense in before to identify the main categories that you want to use in your classification. So, it's both a top down and a bottom-up system. The more top down you work, the more rigorous is the method, the more systematic you are in your approach.

So basically, then taking this categorization system or these CAT taxonomies, you have to classify all your documents  according to these categories. Sometimes you have to work with multiple  classifications, but that's a different issue. An important question is,  is your taxonomy you develop actually would several authors agree to classifying the documents in the way you do?

And for that in the literature review, we classically use the inter-rater reliability. There are different ways to calculate inter-rater reliability. In principle, the measure gives  the agreement of the classification of authors on a certain part of or one document in your corpus.

So, would all authors agree on classifying the given document in that way, according your taxonomy? I think it's a valuable measure if you want  to make higher level conclusions about your taxonomy and that can be important for your publication and for your results. So, for the data analysis, you then later  and in the explanation of your results, you have to explain the categories best, give examples for specific values and categories, consider the distribution over your categories  so it can be that you find that a certain, coming back to our example, hat a certain type of hardware is preferrable used and there are  big gaps with the other ones.

Identify patterns, probably of combinations  of different parts of your taxonomy. So do the HMD. Do the head-mounted displays.

Examples always include a specific design pattern in the software, for example. Yeah, other results you can look at is, are there gaps in the distribution? Is something that you would think of not done in the literature before?

And cross links between the values and categories. So, for the reporting, I think you should  use something like Prisma or comparable to have a scheme in the steps where you clearly  document how you start your search, in which databases you've searched, how you reduced  your corpus, how you've analyzed your corpus. You should, of course, use citation  standards to collect all your publications.

I highly recommend a reference management system. It can be something like EndNote,  Mendelay, Reference Manager. There is Zotero.

There are varieties of them, but use something  like that, to collect your references. This makes it much easier when writing up your reporting. You should rigorously report on the method and the analysis, you basically in the results section, you focus on the different research questions  and your concluding taxonomy you found, but also on the cross relations  and the higher-level conclusions.

So, I think it's very, very important that you really go back also to your purpose and to your research questions  when reporting the results. And look at higher level results  and conclusions you can find. You will find a set of examples on the website and just to sum up, so focus on your research question, identify databases, key terms, work with queries, vary the queries in the screening, you can have different  inclusion and exclusion criteria than later on.

In the eligibility, you need to have them well formulated. Classify, use categories and values. And when you do the reporting, start with simple, one-dimensional results and then go to higher level results and reporting.