fbpx
Connect with us

The Conversation

Early COVID-19 research is riddled with poor methods and low-quality results − a problem for science the pandemic worsened but didn’t create

Published

on

Early COVID-19 research is riddled with poor methods and low-quality results − a problem for science the pandemic worsened but didn't create

The pandemic spurred an increase in research, much of it with methodological holes.
Andriy Onufriyenko/Moment via Getty Images

Dennis M. Gorman, Texas A&M University

Early in the COVID-19 pandemic, researchers flooded journals with studies about the then-novel coronavirus. Many publications streamlined the peer-review for COVID-19 papers while keeping acceptance rates relatively high. The assumption was that policymakers and the public would be able to identify valid and useful research among a very large volume of rapidly disseminated information.

However, in my review of 74 COVID-19 papers published in 2020 in the top 15 generalist public journals listed in Google Scholar, I found that many of these studies used poor quality methods. Several other reviews of studies published in medical journals have also shown that much early COVID-19 research used poor research methods.

Some of these papers have been cited many times. For example, the most highly cited public health publication listed on Google Scholar used data from a sample of 1,120 people, primarily well-educated young women, mostly recruited from social over three days. Findings based on a small, self-selected convenience sample cannot be generalized to a broader population. And since the researchers ran more than 500 analyses of the data, many of the statistically significant results are likely occurrences. However, this study has been cited over 11,000 times.

A highly cited paper means a lot of people have mentioned it in their own work. But a high number of citations is not strongly linked to research quality, since researchers and journals can and manipulate these metrics. High citation of low-quality research increases the chance that poor evidence is being used to inform policies, further eroding public confidence in science.

Advertisement

Methodology matters

I am a public health researcher with a long-standing interest in research quality and integrity. This interest lies in a belief that science has helped solve important social and public health problems. Unlike the anti-science movement spreading misinformation about such successful public health measures as vaccines, I believe rational criticism is fundamental to science.

The quality and integrity of research depends to a considerable extent on its methods. Each type of study design needs to have certain features in order for it to provide valid and useful information.

For example, researchers have known for decades that for studies evaluating the effectiveness of an intervention, a control group is needed to know whether any observed effects can be attributed to the intervention.

Systematic reviews pulling together data from existing studies should describe how the researchers identified which studies to include, assessed their quality, extracted the data and preregistered their protocols. These features are necessary to ensure the review will cover all the available evidence and tell a reader which is worth attending to and which is not.

Advertisement

Certain types of studies, such as one-time surveys of convenience samples that aren't representative of the target population, collect and analyze data in a way that does not allow researchers to determine whether one variable caused a particular outcome.

Systematic reviews involve thoroughly identifying and extracting information from existing research.

All study designs have standards that researchers can consult. But adhering to standards slows research down. a control group doubles the amount of data that needs to be collected, and identifying and thoroughly reviewing every study on a topic takes more time than superficially reviewing some. Representative samples are harder to generate than convenience samples, and collecting data at two points in time is more work than collecting them all at the same time.

Studies comparing COVID-19 papers with non-COVID-19 papers published in the same journals found that COVID-19 papers tended to have lower quality methods and were less likely to adhere to standards than non-COVID-19 papers. COVID-19 papers rarely had predetermined hypotheses and plans for how they would report their findings or analyze their data. This meant there were no safeguards against dredging the data to find “statistically significant” results that could be selectively reported.

Such methodological problems were likely overlooked in the considerably shortened peer-review process for COVID-19 papers. One study estimated the average time from submission to acceptance of 686 papers on COVID-19 to be 13 days, compared with 110 days in 539 pre-pandemic papers from the same journals. In my study, I found that two online journals that published a very high volume of methodologically weak COVID-19 papers had a peer-review process of about three weeks.

Advertisement

Publish-or-perish culture

These quality control issues were present before the COVID-19 pandemic. The pandemic simply pushed them into overdrive.

Journals tend to favor positive, “novel” findings: that is, results that show a statistical association between variables and supposedly identify something previously unknown. Since the pandemic was in many ways novel, it provided an for some researchers to make bold claims about how COVID-19 would spread, what its effects on mental health would be, how it could be prevented and how it might be treated.

Person with head in hands, elbows planted on stacks of paperwork and books littering a desk, glasses and laptop on the side
Many researchers feel pressure to publish papers in order to advance their careers.
South_agency/E+ via Getty Images

Academics have worked in a publish-or-perish incentive system for decades, where the number of papers they publish is part of the metrics used to evaluate employment, promotion and tenure. The flood of mixed-quality COVID-19 information afforded an opportunity to increase their publication counts and boost citation metrics as journals sought and rapidly reviewed COVID-19 papers, which were more likely to be cited than non-COVID papers.

Online publishing has also contributed to the deterioration in research quality. Traditional academic publishing was limited in the quantity of articles it could generate because journals were packaged in a printed, physical document usually produced only once a month. In contrast, some of today's online mega-journals publish thousands of papers a month. Low-quality studies rejected by reputable journals can still find an outlet happy to publish it for a fee.

Healthy criticism

Criticizing the quality of published research is fraught with risk. It can be misinterpreted as throwing fuel on the raging fire of anti-science. My response is that a critical and rational approach to the production of knowledge is, in fact, fundamental to the very practice of science and to the functioning of an open society capable of solving complex problems such as a worldwide pandemic.

Advertisement

Publishing a large volume of misinformation disguised as science during a pandemic obscures true and useful knowledge. At worst, this can lead to bad public health practice and policy.

Science done properly produces information that allows researchers and policymakers to better understand the world and test ideas about how to improve it. This involves critically examining the quality of a study's designs, statistical methods, reproducibility and transparency, not the number of times it has been cited or tweeted about.

Science depends on a slow, thoughtful and meticulous approach to data collection, analysis and presentation, especially if it intends to provide information to enact effective public health policies. Likewise, thoughtful and meticulous peer review is unlikely with papers that appear in print only three weeks after they were first submitted for review. Disciplines that reward quantity of research over quality are also less likely to protect scientific integrity during crises.

Two scientists pipetting liquids under a fume hood, with another scientist in the background examining a sample
Rigorous science requires careful deliberation and attention, not haste.
Assembly/Stone via Getty Images

Public health heavily draws upon disciplines that are experiencing replication crises, such as psychology, biomedical science and biology. It is similar to these disciplines in terms of its incentive structure, study designs and analytic methods, and its inattention to transparent methods and replication. Much public health research on COVID-19 shows that it suffers from similar poor-quality methods.

Reexamining how the discipline rewards its scholars and assesses their scholarship can it better prepare for the next public health crisis.The Conversation

Dennis M. Gorman, Professor of Epidemiology and Biostatistics, Texas A&M University

Advertisement

This article is republished from The Conversation under a Creative Commons license. Read the original article.

The Conversation

Vaccines tell a success story that Robert F. Kennedy Jr. and Trump forget – here are some key reminders

Published

on

theconversation.com – Mark R. O'Brian, Professor and Chair of Biochemistry, Jacobs School of Medicine and Biomedical Sciences, University at Buffalo – 2024-07-26 07:11:29
Many fatal childhood illnesses can be prevented with vaccination.
Halfpoint Images/Moment via Getty Images

Mark R. O'Brian, University at Buffalo

Vaccinations have provided significant protection for the public against infectious diseases. However, there was a modest decrease in support in 2023 nationwide for vaccine requirements for to attend .

In addition, the presidential candidacy of Robert F. Kennedy Jr., a leading critic of childhood vaccination, has given him a prominent platform in which to amplify his views. This includes an extensive interview on the “Joe Rogan Experience,” a with over 14 million subscribers. Notably, former has said he is opposed to mandatory school COVID-19 vaccinations, and in a phone call Trump apparently wasn't aware was being recorded, he appeared to endorse Kennedy's views toward vaccines.

I am a biochemist and molecular biologist studying the roles microbes play in health and disease. I also teach medical and am interested in how the public understands science.

Advertisement

Here are some facts about vaccines that skeptics like Kennedy get wrong:

Vaccines are effective and safe

Public health data from 1974 to the present conclude that vaccines have saved at least 154 million lives worldwide over the past 50 years. Vaccines are also constantly monitored for safety in the U.S.

Nevertheless, the false claim that vaccines cause autism persists despite study after study of large populations throughout the world showing no causal link between them.

Claims about the dangers of vaccines often come from misrepresenting scientific research papers. Kennedy cites a 2005 report allegedly showing massive brain inflammation in monkeys in response to vaccination, when in fact the authors of that study that there were no serious medical complications. A separate 2003 study that Kennedy claimed showed a 1,135% increase in autism in vaccinated versus unvaccinated children actually found no consistent significant association between vaccines and neurodevelopmental outcomes.

Advertisement

Kennedy also claims that a 2002 vaccine study included a control group of children 6 months of age and younger who were fed mercury-contaminated tuna sandwiches. This claim is false.

Gloved hands of clinician placing bandaid on child's arm, a syringe and vaccine vial beside them
Vaccines are continuously monitored for safety before and long after they're available to the general public.
Elena Zaretskaya/Moment via Getty Images

Aluminum adjuvants help boost immunity

Kennedy is co-counsel with a law firm that is suing the pharmaceutical company Merck based in part on the unfounded assertion that the aluminum in one of its vaccines causes neurological disease. Aluminum is added to many vaccines as an adjuvant to strengthen the body's immune response to the vaccine, thereby enhancing the body's defense against the targeted microbe.

The law firm's claim is based on a 2020 report showing that brain tissue from some with Alzheimer's disease, autism and multiple sclerosis have elevated levels of aluminum. The authors of that study do not assert that vaccines are the source of the aluminum, and vaccines are unlikely to be the culprit.

Notably, the brain samples analyzed in that study were from 47- to 105-year-old patients. Most people are exposed to aluminum primarily through their diets, and aluminum is eliminated from the body within days. Therefore, aluminum exposure from childhood vaccines is not expected to persist in those patients.

Vaccines undergo the same approval process as other drugs

Clinical trials for vaccines and other drugs are blinded, randomized and placebo-controlled studies. For a vaccine trial, this means that participants are randomly divided into one group that receives the vaccine and a second group that receives a placebo saline solution. The researchers carrying out the study, and sometimes the participants, do not know who has received the vaccine or the placebo until the study has finished. This eliminates bias.

Advertisement

Results are published in the public domain. For example, vaccine trial data for COVID-19, human papilloma virus and rotavirus is available for anyone to access.

Vaccine manufacturers are liable for injury or death

Kennedy's lawsuit against Merck contradicts his insistence that vaccine manufacturers are fully immune from litigation.

His claim is based on an incorrect interpretation of the National Vaccine Injury Compensation Program, or VICP. VICP is a no-fault federal program created to reduce frivolous lawsuits against vaccine manufacturers, which threaten to cause vaccine shortages and a resurgence of vaccine-preventable disease.

A person injury from a vaccine can petition the U.S. Court of Federal Claims through the VICP for monetary compensation. If the VICP petition is denied, the claimant can then sue the vaccine manufacturer.

Advertisement
Gloved hand picking up vaccine vial among a tray of vaccine vials
Drug manufacturers are liable for any vaccine-related death or injury.
Andreas Ren Photography Germany/Image Source via Getty Images

The majority of cases resolved under the VICP end in a negotiated settlement between parties without establishing that a vaccine was the cause of the claimed injury. Kennedy and his law firm have incorrectly used the payouts under the VICP to assert that vaccines are unsafe.

The VICP gets the vaccine manufacturer off the hook only if it has complied with all requirements of the Federal Food, Drug and Cosmetic Act and exercised due care. It does not protect the vaccine maker from claims of fraud or withholding information regarding the safety or efficacy of the vaccine during its development or after approval.

Good nutrition and sanitation are not substitutes for vaccination

Kennedy asserts that populations with adequate nutrition do not need vaccines to avoid infectious diseases. While it is clear that improvements in nutrition, sanitation, water treatment, food safety and public health measures have played important roles in reducing deaths and severe complications from infectious diseases, these factors do not eliminate the need for vaccines.

After World War II, the U.S. was a wealthy nation with substantial health-related . Yet, Americans reported an average of 1 million cases per year of now-preventable infectious diseases.

Vaccines introduced or expanded in the 1950s and 1960s against diseases like diphtheria, pertussis, tetanus, measles, polio, mumps, rubella and Haemophilus influenza type B have resulted in the near or complete eradication of those diseases.

Advertisement

It's easy to forget why many infectious diseases are rarely encountered today. The success of vaccines does not always tell its own story. It must be retold again and again to counter misinformation.The Conversation

Mark R. O'Brian, Professor and Chair of Biochemistry, Jacobs School of Medicine and Biomedical Sciences, University at Buffalo

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read More

The post Vaccines tell a success story that Robert F. Kennedy Jr. and Trump forget – here are some key reminders appeared first on .com

Advertisement
Continue Reading

The Conversation

Tagging seals with sensors helps scientists track ocean currents and a changing climate

Published

on

theconversation.com – Lilian (Lily) Dove, Postdoctoral Fellow of Oceanography, Brown – 2024-07-25 07:08:14

Tagging seals with sensors helps scientists track ocean currents and a changing climate

Lilian Dove, Brown University

A surprising technique has helped scientists observe how Earth's oceans are changing, and it's not using specialized robots or artificial intelligence. It's tagging seals.

Several species of seals around and on Antarctica and regularly dive more than 100 meters in search of their next meal. These seals are experts at swimming through the vigorous ocean currents that make up the Southern Ocean. Their tolerance for deep waters and ability to navigate rough currents make these adventurous creatures the perfect research assistants to oceanographers like my colleagues and me study the Southern Ocean.

Advertisement

Seal sensors

Researchers have been attaching tags to the foreheads of seals for the past two decades to collect data in remote and inaccessible regions. A researcher tags the seal during mating season, when the marine mammal to shore to rest, and the tag remains attached to the seal for a year.

A researcher glues the tag to the seal's head – tagging seals does not affect their behavior. The tag detaches after the seal molts and sheds its fur for a new coat each year.

The tag collects data while the seal dives and transmits its location and the scientific data back to researchers via satellite when the seal surfaces for .

First proposed in 2003, seal tagging has grown into an international collaboration with rigorous sensor accuracy standards and broad data sharing. Advances in satellite technology now allow scientists to have near-instant access to the data collected by a seal.

Advertisement

New scientific discoveries aided by seals

The tags attached to seals typically carry pressure, temperature and salinity sensors, all properties used to assess the ocean's rising temperatures and changing currents. The sensors also often contain chlorophyll fluorometers, which can data about the 's phytoplankton concentration.

Phytoplankton are tiny organisms that form the base of the oceanic food web. Their presence often means that animals such as fish and seals are around.

The seal sensors can also tell researchers about the effects of climate change around Antarctica. Approximately 150 tons of ice melts from Antarctica every year, contributing to global sea-level rise. This melting is driven by warm water carried to the ice shelves by oceanic currents.

With the data collected by seals, oceanographers have described some of the physical pathways this warm water travels to reach ice shelves and how currents transport the resulting melted ice away from glaciers.

Advertisement

Seals regularly dive under sea ice and near glacier ice shelves. These regions are challenging, and can even be dangerous, to sample with traditional oceanographic methods.

Across the open Southern Ocean, away from the Antarctic coast, seal data has also shed light on another pathway causing ocean warming. Excess heat from the atmosphere moves from the ocean surface, which is in contact with the atmosphere, down to the interior ocean in highly localized regions. In these areas, heat moves into the deep ocean, where it can't be dissipated out through the atmosphere.

The ocean stores most of the heat energy put into the atmosphere from human activity. So, understanding how this heat moves around helps researchers monitor oceans around the globe.

Seal behavior shaped by ocean physics

The seal data also provides marine biologists with information about the seals themselves. Scientists can determine where seals look for food. Some regions, called fronts, are hot spots for elephant seals to hunt for food.

Advertisement

In fronts, the ocean's circulation creates turbulence and mixes water in a way that brings nutrients up to the ocean's surface, where phytoplankton can use them. As a result, fronts can have phytoplankton blooms, which attract fish and seals.

Scientists use the tag data to see how seals are adapting to a changing climate and warming ocean. In the short term, seals may benefit from more ice melt around the Antarctic continent, as they tend to find more food in coastal areas with holes in the ice. Rising subsurface ocean temperatures, however, may change where their prey is and ultimately threaten seals' ability to thrive.

Seals have helped scientists understand and observe some of the most remote regions on Earth. On a changing planet, seal tag data will continue to provide observations of their ocean , which has vital implications for the rest of Earth's climate system.The Conversation

Lilian Dove, Postdoctoral Fellow of Oceanography, Brown University

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Advertisement

Read More

The post Tagging seals with sensors helps scientists track ocean currents and a changing climate appeared first on .com

Continue Reading

The Conversation

Cheesemaking is a complex science – a food chemist explains the process from milk to mozzarella

Published

on

theconversation.com – John A. Lucey, Professor of Food Science, of Wisconsin- – 2024-07-24 07:18:57
Storing cheese wheels to let them age intensifies the flavor.
AP Photo/Antonio Calanni

John A. Lucey, University of Wisconsin-Madison

Cheese is a relatively simple food. It's made with milk, enzymes – these are proteins that can chop up other proteins – bacterial cultures and salt. Lots of complex chemistry goes into the cheesemaking process, which can determine whether the cheese turns out soft and gooey like mozzarella or hard and fragrant like Parmesan.

In fact, humans have been making cheese for about 10,000 years. Roman soldiers were given cheese as part of their rations. It is a nutritious food that provides protein, calcium and other minerals. Its long shelf allows it to be transported, traded and shipped long distances.

I am a food scientist at the University of Wisconsin who has studied cheese chemistry for the past 35 years.

Advertisement

In the U.S., cheese is predominantly made with cow's milk. But you can also find cheese made with milk from other animals like sheep, goats and even buffalo and yak.

Unlike with yogurt, another fermented dairy product, cheesemakers whey – which is water – to make cheese. Milk is about 90% water, whereas a cheese like cheddar is less than about 38% water.

Removing water from milk to make cheese results in a harder, firmer product with a longer shelf life, since milk is very perishable and spoils quickly. Before the invention of refrigeration, milk would quickly sour. Making cheese was a way to preserve the nutrients in milk so you could eat it weeks or months in the future.

How is cheese made?

All cheesemakers first pump milk into a cheese vat and add a special enzyme called rennet. This enzyme destabilizes the proteins in the milk – the proteins then aggregate together and make a gel. The cheesemaker is essentially turning milk from a liquid into a gel.

Advertisement

After anywhere from 10 minutes to an hour, depending on the type of cheese, the cheesemaker cuts this gel, typically into cubes. Cutting the gel helps some of the whey, or water, separate from the cheese curd, which is made of aggregated milk and looks like a yogurt gel. Cutting the gel into cubes lets some water escape from the newly cut surfaces through small pores, or openings, in the gel.

The cheesemaker's goal is to remove as much whey and moisture from the curd as they need to for their specific recipe. To do so, the cheesemaker might stir or heat up the curd, which helps release whey and moisture. Depending on the type of cheese made, the cheesemaker will drain the whey and water from the vat, leaving behind the cheese curds.

A man in a white lab coat, hairnet and gloves pulls a device through a large tub of white liquid.
Wisconsin Master Cheesemaker Gary Grossen cuts a vat of cheese with a cheese harp during a cheesemaking short course at the Center for Dairy Research in Madison, Wis. Cutting helps release whey during the cheesemaking .
UW Center for Dairy Research

For a harder cheese like cheddar, the cheesemaker adds salt directly to the curds while they're still in the vat. Salting the curds expels more whey and moisture. The cheesemaker then packs the curds together in forms or hoops – these are containers that shape the curds into a block or wheel and hold them there – and places them under pressure. The pressure squeezes the curds in these hoops, and they knit together to form a solid block of cheese.

Cheesemakers salt other cheeses, like mozzarella, by placing them in a salt solution called a brine. The cheese block or wheel floats in a brine tank for hours, days or even weeks. During that time, the cheese absorbs some of the salt, which adds flavor and protects against unwanted bacterial or pathogen growth.

A graphic showing the many steps between a farmer harvesting milk from cows and the cheese reaching the consumer.
The cheese production process.
UW Center for Dairy Research

Cheese is a living, fermented food

While the cheesemaker is completing all these steps, several important bacterial processes are occurring. The cheesemaker adds cheese cultures, which are bacteria they choose that produce specific flavors, at the beginning of the process. Adding them to the milk while it is still liquid gives the bacteria time to ferment the lactose in the milk.

Historically, cheesemakers used raw milk, and the bacteria in the raw milk soured the cheese. Now, cheesemakers use pasteurization, a mild heat treatment that destroys any pathogens present in the raw milk. But using this treatment means the cheesemakers need to add back in some bacteria called starters – these “start” the fermentation process.

Advertisement

Pasteurization provides a more controlled process for the cheesemaker, as they can select specific bacteria to add, rather than whatever is present in the raw milk.
Essentially, these bacteria eat (ferment) the sugar – the lactose – and in doing so produce lactic acid, as well as other desirable flavor compounds in the cheese like diacetyl, which smells like hot buttered popcorn.

In some types of cheese, these cultures stay active in the cheese long after it leaves the cheese vat. Many cheesemakers age their cheeses for weeks, months or even years to give the fermentation process more time to develop the desired flavors. Aged cheeses include Parmesan, aged cheddars and Gouda.

A person in a white coat holds a wheel of cheese.
A Wisconsin cheesemaker inspects a wheel of Parmesan in the aging room. Aging is an important step in the production of many cheeses, as it allows for flavor .
The Dairy Farmers of Wisconsin

In essence, cheesemaking is a milk concentration process. Cheesemakers want their final product to have the milk proteins, fat and nutrients, without as much of the water. For example, the main milk protein that is captured in the cheesemaking process is casein. Milk might contain about 2.5% casein content, but a finished cheese like cheddar may contain about 25% casein (protein). So cheese contains lots of nutrients including protein, calcium and fat.

Infinite possibilities with cheese

There are hundreds of different varieties of cow's milk cheese made across the globe, and they all start with milk. All of these different varieties are produced by adjusting the cheesemaking process.

For some cheeses, like Limburger, the cheesemaker rubs a smear – a solution containing various types of bacteria – on the cheese's surface during the aging process. For others, like Camembert, the cheesemaker places the cheese in an (e.g., a cave) that encourages mold growth.

Advertisement

Others like bandaged cheddar are wrapped with bandages or covered with ash. Adding a bandage or ash onto the cheese's surface helps protect it from excessive mold growth, and it reduces the amount of moisture lost to evaporation. This creates a harder cheese with stronger flavors.

A man in a white apron and hat stands in a room full of shelves stacked with cheese.
Wisconsin Master Cheesemaker Joe Widmer in his brick cheese aging room. Brick cheese is a smear-ripened cheese – it is produced by applying a salt solution to the exterior of the cheese as it ages.
Dairy Farmers of Wisconsin

Over the past 60 years, cheesemakers have figured out how to select the right bacterial cultures to make cheese with specific flavors and textures. The possibilities are endless, and there's no limit to the cheesemaker's imagination.The Conversation

John A. Lucey, Professor of Food Science, University of Wisconsin-Madison

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read More

Advertisement

The post Cheesemaking is a complex science – a food chemist explains the process from milk to mozzarella appeared first on .com

Continue Reading

Trending