Research Methods: Academic Research
This is a continuation in our series on Research in an Age of Distrust. See our last blog for an introduction to this series which outlines our mission for transparency and trust-based research.
Unsurprisingly, the cornerstone of our research process is academic and scientific research. We first look to high-quality, peer-reviewed research papers from reliable sources when inspecting a product, method, or modality. For topics like air quality, water quality, and sauna, there is significant literature published and reviewed by world-class institutions that make questions of efficacy an open-and-shut case. Overwhelming evidence in the form of high-quality research sets a terrific precedent, meaning that the following steps merely add greater context and understanding.
Here are some principles we follow to ensure that we put trustworthy findings into practice:
Cite Reliable Studies: Strong, reliable databases like JSTOR, PubMed, and Academic Search Premier are good places to start, as part of their value is that they have a strong filtering mechanism to weed out unreliable sources, pulling mostly from well-reputed journals. Using points of reference like this Google Scholar journal prestige ranking, SJR Journal Ranking tool, and the Scopus site score are helpful to further vet a specific journal’s reliability. Also look at the authors: is this their 3rd or 300th piece of published research? What’s their background and what are their motives? If a study is being funded in a biased way (like a sleep study funded by a company selling sleep medicine) or the author has commercial ties to the study’s outcome, the study’s methods may merit greater scrutiny than those funded by universities or government agencies.
Cross-Verify Findings: Are there numerous studies pointing to the same results, or are results inconsistent? The reappearance of similar findings across studies is a strong indicator of a reliable conclusion.
Check Citations: do the secondary sources meet the above criteria as well? A reliable source is likely using other reliable sources to fuel their findings.
Consider Bias and Perspective: Make note of any ethical considerations surrounding a given topic, whether or not they are mentioned in the paper.. A good paper will outline any biases that may exist, such as sources of fundraising or, potentially, the personal motives of authors. Be aware that authors may be motivated by one conclusion more than another. Don’t let this erode your trust, but critically examine the study, its methods, and its conclusions.
Research Methodology: Read into the actual experimental structure, not just the abstract and the conclusions. Understanding methodology is key to appropriately assessing the nuances that affect the value of a specific conclusion.
Stay Updated: Current information is key, as fields are constantly evolving with new findings, considerations, and challenges. This is especially true in studies evaluating or relying on technologies. With the constant evolution of hardware and software comes a constant evolution of best-in-class tools and solutions.
Consult Experts: If at all possible, don’t go about research alone. Forming relationships with experts in the field can not only help ensure sound conclusions, but also improve the quality of research going forward. Learning from others’ mistakes is invaluable.
Of course, while we prioritize the scientific method, it is not the end-all be-all to vetting new products and ideas. Some of the things we investigate, like feng shui, positive psychology, and sacred principles of design, do not easily meet the criteria to qualify for significant, repeatable scientific inquiry, and it’s not because they aren’t quality solutions. Rather, some things are simply more difficult to test, not popular enough to gain widespread attention, or are too new to be backed by a substantial body of research.
To avoid throwing out the baby with the bathwater, we consider these (and all) cases in a few different lights. After looking into how a solution fares under the scrutiny of the scientific method, we examine its merit and history in the broader context of our human story, a process we will outline in our next blog.