Methodology is the greatest flaw in this study but it’s not the only flaw. Let’s go over simple ways this study can be improved (cont) https://docs.google.com/document/d/1MLfxIyKSODsTPVw3sRIYZ9sfCWH_G14qUboO6pPmNRc/edit

1) first of all, the overall structure and presentation of the study is inappropriate for an empirical study. typically research publications are published in sections in the order of abstract, introduction, methods, results, and discussion. +

Discussion throughout the study and lack of headings makes it difficult for readers to follow along the researcher’s process. it also makes it difficult for a reader to look for exact methodology should they wish to replicate the study with a different sample. It’s hard to read +

and hard to digest despite the language and methods not being inherently complex. 2) Now, let’s discuss the issues of the methodology. Most notably is the sample size being 19 people (n=19). A sample is meant to be representative of a population which in this case is ARMY. +

Not only is the sample size too small to generalize, the publication mentions using snowball sampling. This means the first interviewees recruited their peers to be the other interviewees, which creates an even smaller net of diverse responses. +

If they wanted a true diverse sample, random sampling from many countries or continents would have been a major improvement. 3) The methodology for collecting data was a one hour interview. The data for this research being qualitative is its biggest flaw. Quality is subjective+

so the question is how do you scientifically measure quality? First with having good variables. Construct validity, meaning how you construct your measured variables, is absolutely vital. The questions should have been asked via a poll using a likert scale like this. This way+

the qualitative responses match a set variable that can be more easily compared to other survey results. If not a likert scale, a semantic differential scale such as the one above is also a way to collect qualitative data. The problem with open-ended answers is that they+

are difficult to code into data. For example, words like “tacky” and “ugly” that appeared in the study responses could be interpreted differently depending on whose coding. Is tacky “worse” than ugly? Or vice versa? It’s impossible to tell. 4) Let’s talk about the findings.

5) It’s difficult to tell because there’s no results or discussion section, but it seems the overall conclusion is that “ARMYS are dissatisfied with the quality and timing of Weverse merch drops”. Here’s why a frequency claim (or any claim but especially this) CANT be concluded:+

A frequency claim MUST have a representative sample which is clearly not achieved via snowballing. A frequency claim should have good construct validity which we have concluded is absent. External validity, meaning the extent to which the findings generalize to the population,+

is also absent. If the findings are not generalizable, there is no external validity. Lastly, statistical validity. Do the numbers make sense? Does it truly resemble the population? What is the margin of error? Since they didn’t use random sampling we don’t have that data.

Margin of error tells us how reliable the results from a survey or poll is. The lesser the degree of error is, the more confidence a researcher can have in their results. That is not present due to methodology used.

The study was never going to be well received because it was poorly constructed prior to the first interview. There should have been more peer review throughout the process to better construct the study. A survey and poll not only would be better for data collection +

purposes, but it would also allow a MUCH larger sample size to accurately reflect ARMY. The study being shared with a huge journal is a complete oversight on the researcher’s part, especially because journals care more about clicks than accurate data. The study mentions the+

impact of COVID-19 briefly but seems to not take it seriously as a confounding factor that impacts the entire validity of the study. Perhaps a survey on weverse customer satisfaction is better conducted not during a pandemic?

I’m a bit disappointed that some big ARMY research accounts didn’t catch these glaring errors throughout the process, but hopefully this is a lesson for everyone involved. Let’s work together to be more diligent when conducting research in the future