JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
Original Paper
Investigating COVID-19’s Impact on Mental Health: Trend and
Thematic Analysis of Reddit Users’ Discourse
Jianfeng Zhu1, PhD; Neha Yalamanchi1, MD; Ruoming Jin1, Prof Dr; Deric R Kenne2,3, Prof Dr; NhatHai Phan4, Prof
Dr
1
Department of Computer Science, Kent State University, Kent, OH, United States
2
Center for Public Policy and Health, Kent State University, Kent, OH, United States
3
College of Public Health, Kent State University, Kent, OH, United States
4
Data Science Department, New Jersey Institute of Technology, Newark, NJ, United States
Corresponding Author:
Jianfeng Zhu, PhD
Department of Computer Science
Kent State University
800 E Summit St
Kent, OH, 44240
United States
Phone: 1 2348639445
Email: jzhu10@kent.edu
Abstract
Background: The COVID-19 pandemic has resulted in heightened levels of depression, anxiety, and other mental health issues
due to sudden changes in daily life, such as economic stress, social isolation, and educational irregularity. Accurately assessing
emotional and behavioral changes in response to the pandemic can be challenging, but it is essential to understand the evolving
emotions, themes, and discussions surrounding the impact of COVID-19 on mental health.
Objective: This study aims to understand the evolving emotions and themes associated with the impact of COVID-19 on mental
health support groups (eg, r/Depression and r/Anxiety) on Reddit (Reddit Inc) during the initial phase and after the peak of the
pandemic using natural language processing techniques and statistical methods.
Methods: This study used data from the r/Depression and r/Anxiety Reddit communities, which consisted of posts contributed
by 351,409 distinct users over a period spanning from 2019 to 2022. Topic modeling and Word2Vec embedding models were
used to identify key terms associated with the targeted themes within the data set. A range of trend and thematic analysis techniques,
including time-to-event analysis, heat map analysis, factor analysis, regression analysis, and k-means clustering analysis, were
used to analyze the data.
Results: The time-to-event analysis revealed that the first 28 days following a major event could be considered a critical window
for mental health concerns to become more prominent. The theme trend analysis revealed key themes such as economic stress,
social stress, suicide, and substance use, with varying trends and impacts in each community. The factor analysis highlighted
pandemic-related stress, economic concerns, and social factors as primary themes during the analyzed period. Regression analysis
showed that economic stress consistently demonstrated the strongest association with the suicide theme, whereas the substance
theme had a notable association in both data sets. Finally, the k-means clustering analysis showed that in r/Depression, the number
of posts related to the “depression, anxiety, and medication” cluster decreased after 2020, whereas the “social relationships and
friendship” cluster showed a steady decrease. In r/Anxiety, the “general anxiety and feelings of unease” cluster peaked in April
2020 and remained high, whereas the “physical symptoms of anxiety” cluster showed a slight increase.
Conclusions: This study sheds light on the impact of COVID-19 on mental health and the related themes discussed in 2 web-based
communities during the pandemic. The results offer valuable insights for developing targeted interventions and policies to support
individuals and communities in similar crises.
(J Med Internet Res 2023;25:e46867) doi: 10.2196/46867
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
J Med Internet Res 2023 | vol. 25 | e46867 | p. 1
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
KEYWORDS
COVID-19; Reddit; r/Depression; r/Anxiety; pandemic; mental health; trend analysis; thematic analysis; natural language
processing (NLP); Word2Vec
Introduction
Background
The COVID-19 pandemic has had a profound impact on mental
health, as individuals worldwide have been subjected to feelings
of depression, anxiety, fear, guilt, and anger [1]. In the United
States, a considerable increase in the symptoms of anxiety
disorder and depressive disorder has been observed during the
period from April to June 2020, compared with the same period
in 2019 [2,3]. There were >310 million confirmed cases and
5.4 million deaths worldwide [4], underscoring the need to
investigate the precise emotional and behavioral changes that
have arisen in response to the pandemic. Multidisciplinary
research is called for to address the psychological impact of
quarantine, social isolation, and economic stress, among many
other factors, on mental health [5].
Social media sites have become increasingly popular avenues
for seeking and sharing health information, making them
important tools for understanding the mental health impact of
the pandemic. Especially, during the pandemic, social media
use surged as billions of people stayed at home and practiced
social distancing [6]. Platforms such as TikTok (ByteDance
Ltd), Pinterest (Pinterest, Inc), and Reddit (Reddit Inc) reported
growth in monthly active users in 2021 compared with 2019,
with increases of 38%, 32%, and 30%, respectively [7]. Despite
the increased use of social media during the pandemic, the
relationship between social media posts and mental health during
times of crisis is not yet fully comprehended.
Prior Work
In recent years, natural language processing (NLP) and statistical
techniques have been increasingly used to analyze social media
posts, providing valuable insights into mental health issues. For
example, Park et al [8] examined the thematic similarities and
differences between and membership in 3 web-based mental
health communities from Reddit using a text mining and
visualization approach. The study used topic modeling to
identify the most frequently discussed themes across the
communities and explored the differences and similarities in
the language used in each community [8]. Tadesse et al [9]
developed a machine learning model using support vector
machines to detect depression-related posts on Reddit. They
extracted linguistic features such as negation, positive and
negative sentiments, and medical terms from the text to classify
the posts [9]. In a different approach, Kolliakou et al [10] used
time-series regression analysis to investigate the relationship
between mental health–related conversations on Twitter
(Twitter, Inc) and the incidence of crisis episodes [10]. More
recently, Liu et al [11] used time-to-event modeling to examine
the transition patterns of other subreddits to r/SuicideWatch.
They used Bayesian Poisson regression to analyze the temporal
factors associated with the subreddit transitions. In addition,
they used latent Dirichlet allocation (LDA) to identify key topics
and examined the association between topic trends and subreddit
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
transitions [11]. These studies demonstrate the potential of using
social media data to identify individuals in need of support and
gain a deeper understanding of the impact of public health crises
on mental health.
Recent studies have tried to understand the impact of the
COVID-19 pandemic on mental health, particularly on
depression and anxiety, which have been identified as major
consequences through social media content analysis. Thukral
et al [12] used statistical and NLP methods, such as LDA topic
modeling, to identify pandemic-related stress factors from
Reddit posts, with young adults and students being particularly
affected by stress related to academic and financial issues. Low
et al [13] and Biester et al [14] highlighted the importance of
social media platforms, for example, web-based communities
such as r/Depression and r/Anxiety, as a source of data for
studying the impact of the pandemic on mental health using
NLP techniques. Marshall et al [15] implemented an NLP
platform to analyze tweet frequency and identify prevalent
discussion topics among United Kingdom residents, identifying
consistent concerns such as the pandemic’s influence on mental
health, lockdown-induced fear and anxiety, and anger and
distrust directed at the government. Brewer et al [16] conducted
a thematic analysis of discussion forum posts related to anxiety,
depression, and obsessive-compulsive disorder (OCD) during
the pandemic. The study demonstrated the potential of social
media forums as a source of data for understanding the impact
of public health crises on mental health [16]. These studies
demonstrate the potential of NLP and social media data to
provide valuable insights into mental health trends during the
COVID-19 pandemic.
Despite previous research, measuring the emotional and
behavioral changes in response to the pandemic remains
challenging. A comprehensive understanding of the impact of
the pandemic on population mental health through social media
discourse is still lacking. Prior studies [12-16] mainly focused
on relatively short periods during the early stages of the
pandemic and lacked prepandemic baseline data and
postpandemic mental health analysis. Moreover, few of these
studies explored both the r/Depression and r/Anxiety subreddits,
and none investigated the differences and relationships between
them over time in terms of themes and factors. This study aimed
to address these limitations by exploring the impact of the
COVID-19 pandemic on mental health through a trend and
thematic analysis of Reddit users’ discourse, specifically
focusing on the r/Depression and r/Anxiety subreddits from
2019 to 2022. The analysis was conducted by examining users’
emotional expressions at 3 time points: before the pandemic,
during key events, and after the peak of the pandemic. This
study sought to investigate the evolution of mental health themes
over time and answer the following research questions:
1.
How did the discourse in mental health–related subreddit
communities evolve between 2019 and 2021?
J Med Internet Res 2023 | vol. 25 | e46867 | p. 2
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
2.
3.
4.
5.
When did Reddit users start discussing COVID-19 within
the r/Depression and r/Anxiety subreddits after the
nationwide emergency declaration on March 13, 2020?
How do engagement patterns and the prevalence of the
studied themes differ between the r/Depression and
r/Anxiety communities over the course of the study period?
How did the COVID-19 pandemic impact the discussion
of mental health topics on the r/Depression and r/Anxiety
subreddits, as evidenced by the k-means clustering analysis?
How do the identified themes, such as economic stress and
substance abuse, correlate with mental health outcomes
such as suicidal ideation within the r/Depression and
r/Anxiety subreddits?
In the Results and Discussion sections, we examine these 5 key
questions that offer valuable insights into the pandemic’s effects
on mental health, as revealed through the personal perspectives
of the Reddit users engaging in conversations within the
r/Depression and r/Anxiety communities. These 2 subreddits
are among the largest web-based communities where people
discuss and seek support for issues related to depression and
anxiety, respectively, with >995,000 members in r/Depression
and >609,000 members in r/Anxiety. These communities offer
a vast amount of text data that can provide a unique perspective
into the struggles experienced by users during the pandemic,
covering the entire timeline from the prepandemic to
postpandemic periods.
Methods
Data Collection
The Reddit platform allows users to post longer-form content
and encourages discussion through comments. The
demographics of the specific users of the subreddits examined
in this study are not available; however, data on the general user
base of the platforms have been provided. Approximately half
of the Reddit user base is from the United States, with 22% of
them being young adults aged 18 to 29 years and 14% of them
being aged 30 to 49 years [17]. The platform witnessed an
increase of 44% in daily active users in 2020, reaching 52
million.
The Pushshift multithread application programming interface
(API) wrapper was used to download data from mental
health–related subreddits, such as r/Addiction, r/ADHD
(attention-deficit/hyperactivity
disorder),
r/Alcoholism,
r/Anxiety, r/Autism, r/Bipolar, r/BPD (bipolar disorder),
r/Bulimia,
r/Depression,
r/Drugs,
r/HealthAnxiety,
r/MentalHealth, r/OCD, r/PTSD (posttraumatic stress disorder),
r/Schizophrenia,
r/Selfharm,
r/SocialAnxiety,
and
r/SuicideWatch. Details of the subreddits are provided in Tables
S1 and S2 in Multimedia Appendix 1. We downloaded 3 million
posts from 2019 to 2021. In addition, we acquired additional
data sets for the r/Depression and r/Anxiety subreddits that
extended through 2021 and 2022. The Pushshift multithread
API wrapper is a tool for efficiently retrieving submissions from
the Pushshift API using multithreading [18]. Multithreading is
a programming technique where a single program or process
can have multiple threads of execution running concurrently,
each performing a different task.
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
Zhu et al
Data Preprocessing
In this study, we applied a series of cleaning steps to preprocess
the posts. These techniques include expanding contractions,
replacing nonalphanumeric characters with whitespace,
converting text to lower case, replacing empty strings with
not-a-number values, removing stopwords, and lemmatizing
the text. Not-a-number is a special floating-point value that
represents undefined or unrepresentable values. Stopwords are
common words such as “a,” “an,” “the,” “and,” and “of” that
are often removed from text data because they do not add much
value to the analysis. Lemmatization is the process of reducing
words to their base or dictionary form, which can improve the
accuracy of text analysis by reducing the number of unique
words. Proper handling of these techniques is crucial to avoid
errors in data analysis. A detailed sample of the cleaned posts
is provided in Table S3 in Multimedia Appendix 1.
Extracting Terms Within the Target Theme
Overview
Understanding the prevalent themes in mental health–related
subreddit posts during the pandemic can provide valuable
insights into users’ attitudes, beliefs, and experiences related to
mental health, as well as patterns and trends. Low et al [13]
conducted a feature extraction from 15 subreddit posts and
manually built lexicons about suicidality, economic stress,
isolation, substance use, domestic stress, and guns. The terms
associated with the themes are provided in Figure S1 in
Multimedia Appendix 1. We used this as our baseline to
construct the target themes for our study.
To verify the presence of these themes in our subreddit posts,
we applied topic modeling to identify the 10 most discussed
topics. In addition, we used the Word2Vec embedding model
trained on the corpus of Reddit posts for the semantic refinement
of theme terms.
Identifying Baseline Themes Using LDA Topic Modeling
Topic modeling with LDA is a powerful unsupervised machine
learning technique used to discover hidden thematic structures
within a large corpus of textual data. LDA assumes that
documents are composed of a mixture of topics, and each topic
is represented by a distribution of words. The genism library
was used to perform LDA model estimation. LDA generates a
predefined number of topics in posts across mental health
subreddits, each characterized by a set of terms ranked by their
probability of occurrence within that topic [19]. To analyze
pandemic-related topics, a sample of 464,264 posts from 2020
to 2022 was taken from the r/Depression and r/Anxiety
subreddits to feed into the LDA model for topic extraction.
Semantic Refinement of Theme Terms Using the
Word2Vec Embedding Model
Word2Vec is a neural network–based algorithm used for NLP
that efficiently processes large amounts of text data. One of the
most intriguing and powerful features of Word2Vec is its ability
to identify and manipulate semantic relationships between
words. For example, a classic example for Word2Vec is
“king−man+woman=queen.” This demonstrates the ability of
the model to identify and manipulate semantic relationships
J Med Internet Res 2023 | vol. 25 | e46867 | p. 3
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
between words, in this case, the gender of royalty. The model
is able to learn that the vector distance between “king” and
“man” is similar to that between “queen” and “woman,”
allowing it to accurately predict the relationship between the 2
pairs of words. This type of analogical reasoning has been
applied in various NLP tasks such as language translation,
sentiment analysis, and text classification [20].
We used Word2Vec and the Embedding Projector Platform,
both of which are available through TensorFlow (Google Brain
Team) [21,22], to train and visualize semantic word relationships
within 2 subreddit posts. The Embedding Projector Platform is
a robust tool developed by Google’s TensorFlow team for
exploring and visualizing high-dimensional data, such as word
embeddings produced by the Word2Vec algorithm. Its
user-friendly interface enables easy identification of related
word clusters and semantic relationships and even the creation
of custom embeddings by combining or modifying existing
ones. Owing to its powerful visualization capabilities, the
Embedding Projector Platform is now an indispensable tool for
NLP and other high-dimensional data applications.
Trend Analysis
Trend analysis is a widely used technique for identifying patterns
and changes over time. It involves the analysis of data over a
period of time to detect trends, such as increasing or decreasing
values, changes in direction, or fluctuations in patterns. In our
study, we applied several analytical methods, including
time-to-event analysis, theme trend analysis, and k-means
clustering analysis, to 2 subreddits from 2019 to 2022. These
methods provided valuable insights into various aspects of the
Reddit posts’ themes, such as COVID-19, economic, social,
domestic, educational, substance, and suicide. We labeled each
post with corresponding themes, such as labeling a post 1 for
the economic feature if it contained any terms related to the
economic theme and 0 otherwise. These labels were used as
features in subsequent analyses. Examples of labeled samples
can be seen in Figure S2 in Multimedia Appendix 1.
1.
2.
Time-to-event analysis, also known as survival analysis, is
a statistical method used to analyze the time until an event
of interest occurs. It is commonly used in medical research
to analyze the time until a patient experiences a certain
event, such as death or disease recurrence [23].
Time-to-event analysis was used to investigate the timing
of the first post by unique authors containing
COVID-19–related keywords after March 13, 2020, when
the Trump Administration declared a nationwide emergency
and issued an additional travel ban on non-US citizens
traveling from 26 European countries owing to COVID-19
[24]. The 2020 data set was the primary focus,
encompassing 10,852 unique authors in r/Depression and
6291 unique authors in r/Anxiety. Table S4 in Multimedia
Appendix 1 provides samples of cleaned tokens of
COVID-19 posts in both subreddits.
Theme trend analysis is a valuable tool that enables
researchers to examine patterns and changes in data over
time. It has wide-ranging applications, including social
media research for mental health [25]. In our study, we used
trend analysis to gain insights into public attitudes and
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
Zhu et al
3.
identify emerging concerns during the pandemic by
conducting a long-term analysis of the target themes that
we previously defined, covering the period between 2019
and 2022.
K-means clustering is an unsupervised learning technique
that groups similar documents together based on their
content without the use of labeled data using the term
frequency–inverse document frequency scheme to create
vectors representing documents [26]. The elbow method
was used to determine the optimal number of clusters. In
this case, the technique was applied to the r/Depression and
r/Anxiety subreddits to uncover latent topics in posts before,
during, and after the pandemic.
Thematic Analysis
Thematic analysis is a method of identifying and analyzing
patterns, themes, and trends in qualitative data. In this study,
we used various analytical methods, including heat map analysis,
factor analysis, and ordinary least squares (OLS) regression, to
gain insights into the relationships between the COVID-19,
economic, social, domestic, educational, substance, and suicide
themes in the r/Anxiety and r/Depression subreddits from 2019
to 2022.
1.
2.
3.
The heat map analysis is a powerful visualization method
used to display the distribution of selected themes in a large
corpus of textual data. We used the heat map to represent
the relationships between the themes in the 2 subreddits.
Factor analysis is a statistical method that identifies
underlying patterns or structures within a data set by
reducing the dimensionality. We performed factor analysis
on 3 separate data sets: r/Depression, r/Anxiety, and a
combined data set of both subreddits.
We applied the OLS regression method [27] to analyze the
relationship between the “suicide” variable and the
independent variables (COVID-19, economic, social,
domestic, educational, and substance). This analysis helped
us determine the influence of these factors on the occurrence
of “suicide” and estimate the change in the dependent
variable associated with a 1-unit change in each independent
variable.
Ethics Approval
To protect the privacy and confidentiality of the individuals
whose data were analyzed, all study data were deidentified
before analysis. The data analyzed in this study were obtained
from publicly available sources and contain no identifiable
information. The sample posts in this study were preprocessed
by removing stopwords and lemmatizing, and the resulting
tokens make it impossible to identify users’ information. No
personal information, including author names or any other
private information, was included in the data set. By addressing
these ethical considerations, we conducted a valuable and trend
study on the impacts of COVID-19 on the content posted by
the r/Anxiety and r/Depression users. In addition, this research
was partially supported by the National Science Foundation
project (IIS-2041065), which was approved by the institutional
review board at Kent State University under the reference
number KSU IRB20-182. After this paper was accepted for
publication, the Institutional Review Board at Kent State
J Med Internet Res 2023 | vol. 25 | e46867 | p. 4
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
University was consulted regarding privacy ethics concerns and
the study was deemed exempt (KSU 819).
Results
Zhu et al
insights gained. These methods enabled us to identify and
analyze trends in the data, allowing us to gain a deeper
understanding of the themes and patterns within the Reddit posts
related to mental health during the pandemic.
Overview
Question 1: How Did the Discourse in Mental
Health–Related Subreddit Communities Evolve Between
2019 and 2021?
Trend analysis often involves the visualization of data on a
graph or chart to identify patterns and trends. In this study, we
used a variety of methods, including statistical analysis,
time-series analysis, and k-means clustering analysis, to better
understand the data and make informed decisions based on the
In the initial stage of our analysis, we examined 18 mental
health–related subreddits. Figure 1 illustrates a decline in the
number of posts in the r/Depression subreddit and a rise in the
number of posts in the r/Anxiety subreddit between 2019 and
2021.
Trend Analysis
Figure 1. Mental health subreddit post distribution from 2019 to 2021. ADHD: attention-deficit/hyperactivity disorder; BPD: bipolar disorder; OCD:
obsessive-compulsive disorder; PTSD: posttraumatic stress disorder.
The results indicate a decrease in the number of posts in the
r/Depression subreddit during the midpandemic period and an
increase in the number of posts in the r/Anxiety subreddit during
the same period. A similar trend in tweets related to depression
and anxiety can be found in Figure S3 in Multimedia Appendix
1.
Depression and anxiety are psychiatric disorders, often reflected
in text written by undiagnosed individuals on social media.
Medical experts can use linguistic markers to improve guidelines
and treatments [28-30]. Focusing on the r/Depression and
r/Anxiety subreddits allows us to study these specific disorders
more closely, providing valuable insights into how individuals
express their experiences and emotions during a global crisis
such as the COVID-19 pandemic. In the r/Depression and
r/Anxiety data sets, we extended the time frame to 2022, which
was used in the following trend analysis and thematic analysis.
The data set used in this study contains posts (652,452) collected
from January 2019 to December 2022. It includes data obtained
from the r/Depression and r/Anxiety subreddits. For the
r/Depression subreddit, a total of 138,517 posts were collected
in 2019, followed by 119,543 posts in 2020, 95,242 posts in
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
2021, and 85,283 posts in 2022, resulting in a cumulative total
of 438,585 posts. Similarly, for the r/Anxiety subreddit, the data
consisted of 49,295 posts in 2019, 54,053 posts in 2020, 53,992
posts in 2021, and 56,527 posts in 2022, accounting for a total
of 213,867 posts. These statistics highlight the volume of posts
obtained from each subreddit throughout the specified time
period. The post distribution of the data set can be found in
Figure S4 in Multimedia Appendix 1.
To validate the trend analysis on the Reddit data sets, we
extracted tweets associated with anxiety and depression from
our Twitter data set using hashtags and keywords. The trends
observed in the number of posts on r/Depression and r/Anxiety
were similar to those found in the number of tweets related to
depression and anxiety hashtags and keywords. The hashtags
and keywords used for filtering can be found in Table S5 in
Multimedia Appendix 1.
Terms of Target Themes
Through LDA topic modeling, we extracted 10 topics based on
the top 10 frequently occurring words, and Figure S5 in
Multimedia Appendix 1 presents the keywords associated with
J Med Internet Res 2023 | vol. 25 | e46867 | p. 5
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
each topic. These 10 topics were then used to verify the baseline
themes, as shown in Figure S1 in Multimedia Appendix 1. Topic
9 focused on the economic theme, which included job search
and work-related challenges. Social themes were found in topics
6 and 0, which covered personal relationships and social
interactions and family and household matters, respectively.
Domestic themes were mainly found in topic 0. Educational
themes were highlighted in topic 8, including academic
experiences and challenges. We excluded the gun theme because
of a lack of relevant data and included the educational theme
because of its high frequency. Although the COVID-19,
substance, and suicide themes were not explicitly identified,
they may be implicitly present within the broader categories of
health and well-being, personal struggles, and life challenges
during the pandemic. Our analysis identified 7 themes, namely
economic, social, domestic, educational, substance, suicide, and
COVID-19.
To preprocess the 652,452 corpora, we tokenized the posts from
r/Depression and r/Anxiety between 2019 and 2022 and
constructed a vocabulary. We generated training examples by
creating pairs of target and context words within a defined
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
Zhu et al
window size of 2. We set the parameters of the model, such as
dimension=128, batch size=1024, and buffer size=10,000, and
trained the model using categorical cross-entropy loss and
adaptive moment estimation as the optimization algorithm.
Upon the completion of the training process with 20 epochs,
the accuracy of the model was 0.633. The vectors and metadata
files are available on GitHub [31], which can be loaded onto
the Embedding Projector Platform [22] for projecting
embeddings in 3D space and for the interactive exploration and
analysis of semantic relationships among the corpora. Figure 2
displays the projecting visualization of the keyword “school”
in the Embedding Projector Platform, illustrating the semantic
relationship between “school” and the other words in the corpora
in 3D space.
After considering the effects of the pandemic and reviewing the
previous results, we manually constructed the final terms for
the target themes. The resulting terms are presented in Table 1,
with italicized words indicating extensions made using the
top-ranked words from the Word2Vec pretraining model for
the key terms.
J Med Internet Res 2023 | vol. 25 | e46867 | p. 6
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
Figure 2. Visualization of the word “school” using the Embedding Projector Platform.
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
J Med Internet Res 2023 | vol. 25 | e46867 | p. 7
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
Table 1. Terms of target themes.
a
Themes
Termsa
Economic
unemployed, economy, mortgage, layoff, recession, stimulus, evict, enough money, more money, pay the bill, owe, afford, wage,
job, eviction, income, rent, credit, salary, bills, laid, jobless, savings, lost job, fulltime, debt, financial, paycheck
Social
bullying, loneliness, emptiness, quarantine, alone, lonely, lockdown, distancing, insecurities, no one cares, trapped, feel ignored,
single, can’t see my, ignoring me
Domestic
domestic violence, abuse, yelling, fighting, single mom, single dad, single parent, hit me, slapped me, divorced, abusive,cheating,separation,toxic,abused,custody battles
Educational
exam, assignment, online classes, school closures, distance learning grade, homework, courses, school, presentation, classroom,
test, virtual learning, hybrid learning, remote learning, online meeting, Zoom, Microsoft Teams, Google classroom, virtual classrooms
Substance
smoke, smoked, drink, snort, drugs, smoking, alcohol, nicotine, caffeine, beer, substance, ketamine, tablets,valium, opioid, vodka,
whiskey, whisky, meth, addiction,rehab,relapse,overdose
Suicide
commit suicide, jump off a bridge, I want to overdose, will overdose, thinking about overdose, kill myself, killing myself, hang myself,
hanging myself, cut myself, cutting myself, hurt myself, hurting myself, want to die, wanna die, do not wake up, do not want to be
alive, wish it would all end, done with living, want it to end, all ends tonight, live anymore, living anymore, life anymore, be dead,
end my life, death, hopeless, shoot me, kill me, suicide, no point, intrusive
COVID-19
corona, coronavirus, covid, covid-19, epidemic, infect, lockdown, pandemic, quarantine, viral, virus, mask, ventilator, symptomatic,
incubation, transmission, immune, vaccine, national emergency, flatten
Italicized words indicate extensions made using the top-ranked words from the Word2Vec pretraining model for the key terms.
Question 2: When Did Reddit Users Start Discussing
COVID-19 Within the r/Depression and r/Anxiety
Subreddits After the Nationwide Emergency Declaration
on March 13, 2020?
We used time-to-event analysis to examine the initiation of
COVID-19 discussions in the r/Depression and r/Anxiety
subreddits following the nationwide emergency declaration on
March 13, 2020. The Kaplan-Meier estimator [32], which is a
nonparametric method for estimating the survival function, was
applied to visualize the results. The Kaplan-Meier survival curve
shows the estimated probability of not posting over time since
March 13, 2020. Figure 3 illustrates the curve starting at 1.0
(ie, 100% probability of not posting) and decreasing over time
as more authors post their first post with COVID-19 keywords
in 2020. The vertical lines on the graph indicate the days on
which a certain percentage of authors have posted, providing
insights into the timing and probability of authors posting their
first post after the national emergency declaration.
Figure 3. Kaplan-Meier survival curve in r/Depression and r/Anxiety. KM: Kaplan Meier.
Question 3: How Do Engagement Patterns and the
Prevalence of the Studied Themes Differ Between the
r/Depression and r/Anxiety Communities Over the
Course of the Study Period?
We conducted a trend analysis to examine the differences in
themes between the 2 subreddits over the specified period.
Figures 4-10 depict the posting activity within the r/Depression
and r/Anxiety communities on Reddit from 2019 to 2022, with
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
a focus on the target themes COVID-19, economic, social,
domestic, educational, substance, and suicide. The figure
consists of 2 sections: the left side displays the distribution of
the number of posts, whereas the right side shows the
distribution of the proportion of posts for each community over
time. The 2 lines in each section represent the r/Depression and
r/Anxiety communities, providing insight into the varying
dynamics of these web-based forums in the context of the
specified themes.
J Med Internet Res 2023 | vol. 25 | e46867 | p. 8
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
Figure 4. Number and proportion of posts that contain any keywords related to COVID-19.
Figure 5. Number and proportion of posts that include any keywords related to the economic theme.
Figure 6. Number and proportion of posts that include any keywords related to the social theme.
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
J Med Internet Res 2023 | vol. 25 | e46867 | p. 9
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
Figure 7. Number and proportion of posts that include any keywords related to the domestic theme.
Figure 8. Number and proportion of posts that include any keywords related to the educational theme.
Figure 9. Number and proportion of posts that include any keywords related to the substance theme.
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
J Med Internet Res 2023 | vol. 25 | e46867 | p. 10
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
Figure 10. Number and proportion of posts that include any keywords related to the suicide theme.
Question 4: How Did the COVID-19 Pandemic Impact
the Discussion of Mental Health Topics on the
r/Depression and r/Anxiety Subreddits, as Evidenced by
the K-Means Clustering Analysis?
K-means clustering analysis was conducted on Reddit posts
from the r/Depression and r/Anxiety subreddits with the aim of
identifying distinct clusters of posts based on their textual
content. The optimal elbow value for r/Depression was found
to be 15, whereas the optimal elbow value for r/Anxiety was
found to be 16. The results of k-means clustering are provided
in Figures S6 and S7 in Multimedia Appendix 1. The clusters
were then manually collated into 8 different latent clusters,
which are provided in Table S6 in Multimedia Appendix 1, and
the trends of these clusters from 2019 to 2022 were analyzed
and displayed in Figures 11 and 12.
Figure 11. K-mean clustering analysis of r/Depression posts.
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
J Med Internet Res 2023 | vol. 25 | e46867 | p. 11
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
Figure 12. K-mean clustering analysis of r/Anxiety posts.
Thematic Analysis
We used a combination of heat map analysis, factor analysis,
and regression analysis to explore the correlation between the
identified themes and mental health outcomes. Specifically, we
aimed to address the following question.
Question 5: How Do the Identified Themes, Such as
Economic Stress and Substance Abuse, Correlate With
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
Mental Health Outcomes Such as Suicidal Ideation
Within the r/Depression and r/Anxiety Subreddits?
Heat Map Analysis
We created a heat map to visually display the associations
between the themes (COVID-19, economic, social, domestic,
educational, substance, and suicide) and the 2 subreddits. A
color gradient was used, where the deeper cell colors represent
stronger associations. The x-axis of the heat map shows the 2
subreddits, with r/Anxiety on the left and r/Depression on the
right, whereas the y-axis displays the selected themes. Each cell
of the heat map represents the strength of the relationship
between a given theme and subreddit. Figure 13 shows the
resulting heat map.
J Med Internet Res 2023 | vol. 25 | e46867 | p. 12
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Zhu et al
Figure 13. Heat map of the relationship between themes and subreddits.
Factor Analysis
We used the Kaiser criterion (eigenvalues>1) to determine the
optimal number of factors and applied the varimax rotation
method to increase the interpretability of the results using the
Factor Analyzer package in Python [33]. We calculated the
eigenvalues of the correlation matrix for the 7 themes and
plotted a scree plot to identify the “elbow” point, where the
decrease in eigenvalues becomes less substantial. A scree plot
is provided in Figure S7 in Multimedia Appendix 1. This helped
us select the most meaningful structure in the data, with 2 factors
identified as optimal for each data set. The factor loading matrix,
shown in Figure 14, illustrates the strength and direction of the
relationships between the themes and the 2 extracted factors,
with higher absolute factor loadings indicating stronger
relationships. Positive and negative loadings indicate direct and
inverse relationships, respectively [34].
Figure 14. Factor loading heat map.
Regression Analysis
The OLS regression analysis on the r/Depression and r/Anxiety
data sets from 2020 to 2022 showed that the suicide-dependent
feature had a significant relationship with other independent
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
features such as economic, social, domestic, educational, and
substance. Figure S8 in Multimedia Appendix 1 presents the
results, including the R-squared value, which indicates the
proportion of variance in the dependent feature explained by
J Med Internet Res 2023 | vol. 25 | e46867 | p. 13
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
the independent features (r/Depression: 0.055 and r/Anxiety:
0.031); the coefficients representing the strength and direction
of the relationships; and the P values (P>|t|, both at .00) that
determine statistical significance. In both subreddits, a P value
<.05 indicates a statistically significant relationship.
Discussion
We used a range of NLP techniques and statistical methods to
perform a trend and thematic analysis, aimed at gauging the
impact of COVID-19 on the mental well-being of individuals
who are part of the r/Depression and r/Anxiety support groups.
In the upcoming sections, we delve into each analysis in detail.
Trend Analysis
Time-to-Event Analysis for Question 2
The results indicate that 20% of the authors posted by day 28
in both r/Depression and r/Anxiety. In addition, 40% of the
authors posted by day 62 in r/Depression and day 67 in
r/Anxiety, whereas 60% of the authors posted by day 133 in
r/Depression and day 132 in r/Anxiety. The 28-day period after
the announcement could be considered a critical window when
mental health concerns started becoming more prominent for
many individuals on Reddit.
This time-to-event analysis contributes to our understanding of
how individuals affected by the pandemic increasingly turned
to web platforms to discuss their mental health struggles. This
emphasizes the need for mental health professionals and support
organizations to recognize critical periods, such as the first 28
days following a major event, and prioritize resource allocation,
interventions, and support measures accordingly. These findings
can help inform future crisis management strategies to address
the mental health impact of large-scale events on the general
population.
Mental Health Theme Trend Analysis for Question 3
In early 2020, there was a noticeable rise in posts mentioning
COVID-19 keywords (eg, corona, COVID, ventilator, vaccine,
and mask) in both subreddits. A peak in April coincided with
high COVID-19 cases and deaths in the United States, reflecting
individuals sharing their fear and uncertainty.
Regarding the proportion of posts, the r/Anxiety community
peaked at 0.25, whereas the r/Depression community reached
0.20 around April 2020. Subsequently, there was a sharp
decrease until July 2020, followed by a stable decline.
Throughout the pandemic, the r/Anxiety community maintained
a slightly higher ratio than the r/Depression community,
suggesting different levels of concern and coping mechanisms
among web-based forums. However, this difference can be
accounted for by many different factors, such as the variations
in moderation principles and community guidelines. Therefore,
it is not possible to draw any precise conclusions. This represents
the limitation of the study that we attempt to address.
We observed prevalent themes such as economic stress, social
isolation, domestic issues, education, substance use, and suicide
ideation in the r/Depression and r/Anxiety subreddits. These
themes presented varying trends and degrees of impact across
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
Zhu et al
the 2 communities, revealing the unique experiences and
struggles faced by individuals coping with depression and
anxiety during the pandemic.
In the r/Anxiety subreddit, the suicide theme emerged as the
most dominant, accounting for around 63% of the posts,
followed by economic stress and substance use. This indicates
that the individuals in r/Anxiety may have experienced increased
susceptibility to suicidal thoughts and harmful coping
mechanisms during the pandemic. Conversely, in the
r/Depression subreddit, economic stress was the most prevalent
theme, emphasizing the significant impact of financial instability
on individuals already grappling with depression. Social
isolation was a shared concern across both communities,
highlighting the negative impact of COVID-19 on interpersonal
relationships.
The importance of addressing educational challenges during
the pandemic was evident in both subreddits. The transition to
remote learning and associated uncertainties exacerbated
feelings of anxiety and depression among adolescents, with
potential long-lasting consequences on their mental health.
By identifying and analyzing these important terms and themes,
researchers, advocates, and practitioners can better understand
the needs and experiences of individuals discussing mental
health on social media during the pandemic. This can inform
the development of more effective interventions and policies
that are tailored to the unique challenges faced by individuals
experiencing mental health concerns during the pandemic and
beyond.
K-Means Clustering Analysis for Question 4
For the r/Depression subreddit, the number of posts related to
the “depression, anxiety, and medication” cluster peaked in
August 2019 and has been decreasing since 2020. The
“self-improvement and personal growth” cluster showed a trend
similar to that shown by the “depression, anxiety, and
medication” cluster. The “social relationships and friendship”
cluster showed a steady decrease since the beginning of the
pandemic in 2020. The “fatigue and tiredness” cluster remained
consistent throughout the period. Entertainment and hobbies
did not show a significant increase or decrease after 2020. The
“job-related stress” and “academic and school-related stress”
clusters showed a stable trend throughout the period. The “life
events and changes” cluster had a peak in April 2020 and has
been decreasing since then. These findings in the depression
group may suggest that the lockdown made people stay at home,
where it is difficult to get any resources such as medication
service, personal activities, and entertainment. This can be an
explanation for the decrease of these clusters since 2020:
because people did not have much chance to do these things,
they did not discuss these things as much as they did in 2019.
Meanwhile, the r/Anxiety subreddit showed that general anxiety
and feelings of unease peaked in April 2020 and remained at
higher-than-average levels throughout the year. The “physical
symptoms of anxiety” cluster remained stable but showed a
slight increase since early 2020. The “work-related anxiety”
cluster also remained relatively stable throughout the period.
The “medication and side effects,” “heart palpitations and chest
J Med Internet Res 2023 | vol. 25 | e46867 | p. 14
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
pain,” and “driving anxiety” clusters remained at consistent
levels throughout the period. Overall, the pandemic seems to
have had a significant impact on the clusters being discussed
in the r/Anxiety subreddit, with many individuals seeking
support for their general anxiety and unease during this
challenging time.
Zhu et al
Thematic Analysis for Question 5
In conclusion, the factor analysis of the r/Depression, r/Anxiety,
and combined data sets highlighted the importance of
considering pandemic-related stress, economic concerns, and
social factors when examining and addressing mental health
issues during the period from 2020 to 2022. These findings
contribute to our understanding of the challenges faced by these
web-based communities and inform the development of targeted
interventions and support.
Heat Map Analysis
Regression Analysis
Through the heat map analysis, we were able to visually identify
the patterns and relationships between the themes and subreddits.
The heat map of the r/Anxiety subreddit revealed that the themes
with the strongest relationship, in descending order, were
suicide, economic, and substance. This suggests that individuals
in this community were particularly affected by suicidal ideation,
economic stress, and substance use during the analyzed period.
The heat map of the r/Depression subreddit displayed the
strongest relationships with the themes of economic, suicide,
and social, suggesting that economic stress and social isolation
were significant concerns for individuals in this community,
along with ongoing struggles with suicidal thoughts.
Two OLS regression analyses were conducted on the
r/Depression and r/Anxiety data sets to examine the relationship
between suicide theme rates (dependent variable) and various
independent variables, such as economic, social, domestic,
educational, and substance themes. On the one hand, the
r/Depression data set yielded an R-squared value of 0.055,
indicating that the independent variables accounted for 5.5%
of the variation in the suicide theme. On the other hand, the
r/Anxiety data set showed an R-squared value of 0.031,
explaining 3.1% of the variance in the suicide theme.
Factor Analysis
The factor analysis of the r/Depression, r/Anxiety, and combined
data sets from 2020 to 2022 offered crucial insights into the
concerns and challenges faced by these web-based communities
during this period. A discussion of the results highlights the
following key findings:
1.
2.
3.
4.
COVID-19 as a prevalent concern: the strong association
between the first factor and the “covid” theme (loading of
0.99) in both the r/Depression and r/Anxiety data sets
suggests that the pandemic has had a significant impact on
the mental health and well-being of the individuals in both
communities. The results emphasize the need for mental
health support and resources tailored to address
pandemic-related stressors and anxieties.
Economic stress as a major issue: the second factor’s strong
association with the “economic” theme (loadings of 0.53,
0.46, and 0.52) in all data sets indicates that economic stress
has been a significant concern across both subreddits. This
finding underscores the importance of addressing financial
stress and providing support to those affected by job loss,
reduced income, and other economic challenges.
Social concerns in the combined data set: the combined
data set’s analysis revealed that social concerns (loading
of 0.99) were also a primary theme, suggesting that
interpersonal relationships and social interactions may be
essential factors affecting mental health during this period.
This observation highlights the need to consider social
support and connectedness when addressing mental health
issues within these communities.
Other themes with weaker associations: the remaining
themes (“domestic,” “educational,” “substance,” and
“suicide”) exhibited relatively weaker relationships with
the extracted factors. However, their presence in the data
sets suggests that they still hold relevance within these
communities and should not be overlooked.
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
In the r/Depression data set, the coefficients revealed the
following order of strength of association with the suicide theme:
economic (0.1273), substance (0.0912), domestic (0.0815),
educational (0.0745), and social (0.0773). All independent
variables had statistically significant relationships with the
suicide theme, as evidenced by their P values being <.05.
Similarly, in the r/Anxiety data set, the coefficients showed that
the suicide theme had the strongest association with the
economic theme (0.1093), followed by the substance (0.0890),
social (0.0722), educational (0.0633), and domestic (0.0634)
themes. All these relationships were statistically significant,
with P values <.05.
In both data sets, the economic theme consistently demonstrated
the strongest association with the suicide theme. This finding
could indicate that financial stress and economic hardships may
have a considerable impact on mental health, leading individuals
to experience suicidal thoughts. In addition, the substance theme
had a notable association in both data sets, suggesting a possible
link between substance abuse and suicidal ideation.
Limitations
We acknowledge certain limitations in our study. First, the data
used for analysis were self-reported by users on subreddits,
which may introduce social desirability biases. Second, the
focus on r/Depression and r/Anxiety subreddits may not
represent the mental health struggles of individuals across all
web platforms or in real life, limiting the generalizability of our
findings. Third, users may not necessarily discuss their own
experiences when using the identified terms, and the identified
theme terms may not encompass all terms related to the themes,
which may lead to an overestimation of the relevance of certain
themes. We also acknowledge that differences in post volumes
could be attributed to various factors such as community
guidelines and moderation principles. Finally, ethical concerns
related to the use of social media data for mental health
assessment, such as privacy and informed consent, need to be
considered. Further research is needed to better understand the
J Med Internet Res 2023 | vol. 25 | e46867 | p. 15
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
advantages and disadvantages of using social media for mental
health assessment during the pandemic.
Future Research
The study’s findings suggest that there are associations between
certain themes, such as economic and substance use, and a
higher proportion of suicide ideation. To gain a more
comprehensive understanding of these associations, future
research should explore them in greater detail and identify
explanatory themes or factors that contribute to suicidal
tendencies. Such research could help us develop more effective
prevention and intervention strategies for individuals at risk of
suicide. In addition, future studies should investigate other social
media platforms and regions to enhance the generalizability of
the findings. Future research could also examine how
demographic factors, such as age, gender, and socioeconomic
status, influence mental health discussions and emotions during
the pandemic. By addressing these gaps, we can gain a more
nuanced understanding of mental health impacts during global
crises and develop targeted interventions and support systems
for affected individuals.
Conclusions
In conclusion, our study used a variety of NLP techniques to
gain insights into the mental health struggles of the individuals
participating in the r/Depression and r/Anxiety subreddits during
the COVID-19 pandemic. Our time-to-event analysis revealed
that the first 28 days following a major event could be
considered a critical window for mental health concerns to
become more prominent. The COVID-19 keyword trend analysis
showed a peak in April 2020, reflecting individuals sharing
their uncertainty during the pandemic. Our thematic analysis
Zhu et al
identified prevalent themes such as economic stress, social
isolation, domestic issues, education, substance use, and suicide
ideation, with varying trends and degrees of impact across the
2 communities. The factor analysis highlighted pandemic-related
stress, economic concerns, and social factors as the primary
themes affecting mental health during the analyzed period. The
regression analysis showed that economic stress consistently
demonstrated the strongest association with the suicide theme,
whereas the substance theme had a notable association in both
data sets. Finally, the k-means clustering analysis showed that
the number of posts related to the “depression, anxiety, and
medication” cluster decreased after 2020 in r/Depression,
whereas the number of posts related to the “social relationships
and friendship” cluster showed a steady decrease. In r/Anxiety,
the “general anxiety and feelings of unease” cluster peaked in
April 2020 and remained high, whereas the “physical symptoms
of anxiety” cluster showed a slight increase.
However, we acknowledge certain limitations of our study, such
as potential biases in the data collection and analysis methods,
limited generalizability of the findings, and ethical concerns
related to using social media data for mental health assessment.
Further research is required to address these limitations and
gain a more comprehensive understanding of mental health
impacts during global crises. Future studies should explore the
associations between themes in greater detail, investigate other
social media platforms and regions, and examine the influence
of demographic factors on mental health discussions and
emotions during the pandemic. By addressing these gaps, we
can develop more effective interventions and support systems
tailored to the unique challenges faced by individuals
experiencing mental health concerns during the pandemic and
beyond.
Acknowledgments
This research was partially supported by the National Science Foundation (grants IIS-2041065 and IIS-2041096) and the Substance
Abuse and Mental Health Services Administration Strategic Prevention Framework (grant 1H79SP081502).
Data Availability
The data and code used in this study are publicly available on GitHub [31].
Authors' Contributions
JZ as responsible for the experimental design, data collection, and analysis. RJ proposed the initial idea, oversaw the study, and
provided feedback on the analyses. NY conducted the initial analysis of 18 subreddits. All the authors reviewed the manuscript
and provided feedback on the analyses.
Conflicts of Interest
None declared.
Multimedia Appendix 1
Additional data sets details, sample Reddit posts, and analysis descriptions to support the main paper’s investigation into the
impact of COVID-19 on mental health.
[DOCX File , 1717 KB-Multimedia Appendix 1]
References
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
J Med Internet Res 2023 | vol. 25 | e46867 | p. 16
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
Stanton R, To QG, Khalesi S, Williams SL, Alley SJ, Thwaite TL, et al. Depression, anxiety and stress during COVID-19:
associations with changes in physical activity, sleep, tobacco and alcohol use in Australian adults. Int J Environ Res Public
Health 2020 Jun 07;17(11):4065 [FREE Full text] [doi: 10.3390/ijerph17114065] [Medline: 32517294]
Indicators of anxiety or depression based on reported frequency of symptoms during last 7 days. Centers for Disease Control
and Prevention. URL: https://data.cdc.gov/NCHS/Indicators-of-Anxiety-or-Depression-Based-on-… [accessed
2021-03-15]
Early release of selected mental health estimates based on data from the January–June 2019 national health interview survey.
Centers for Disease Control and Prevention. 2020 May. URL: https://www.cdc.gov/nchs/data/nhis/earlyrelease/
ERmentalhealth-508.pdf [accessed 2021-03-08]
WHO Coronavirus (COVID-19) dashboard. World Health Organization. URL: https://COVID19.who.int/ [accessed
2022-12-16]
Holmes EA, O'Connor RC, Perry VH, Tracey I, Wessely S, Arseneault L, et al. Multidisciplinary research priorities for
the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiat 2020 Jun;7(6):547-560 [FREE Full
text] [doi: 10.1016/S2215-0366(20)30168-1] [Medline: 32304649]
Zhang T, Schoene AM, Ji S, Ananiadou S. Natural language processing applied to mental illness detection: a narrative
review. NPJ Digit Med 2022 Apr 08;5(1):46 [FREE Full text] [doi: 10.1038/s41746-022-00589-7] [Medline: 35396451]
Statista homepage. Statista Research Department. URL: https://www.statista.com/topics/7863/social-media-use-duringcoronavirus… [accessed 2022-02-08]
Park A, Conway M, Chen AT. Examining thematic similarity, difference, and membership in three online mental health
communities from reddit: a text mining and visualization approach. Comput Human Behav 2018 Jan;78:98-112 [FREE
Full text] [doi: 10.1016/j.chb.2017.09.001] [Medline: 29456286]
Tadesse MM, Lin H, Xu B, Yang L. Detection of depression-related posts in reddit social media forum. IEEE Access
2019;7:44883-44893 [FREE Full text] [doi: 10.1109/access.2019.2909180]
Kolliakou A, Bakolis I, Chandran D, Derczynski L, Werbeloff N, Osborn DP, et al. Mental health-related conversations
on social media and crisis episodes: a time-series regression analysis. Sci Rep 2020 Feb 06;10(1):1342 [FREE Full text]
[doi: 10.1038/s41598-020-57835-9] [Medline: 32029754]
Liu X, Fang S, Mohler G, Carlson J, Xiao Y. Time-to-event modeling of subreddits transitions to r/SuicideWatch. In:
Proceedings of the 21st IEEE International Conference on Machine Learning and Applications (ICMLA). 2022 Presented
at: ICMLA '22; December 12-14, 2022; Nassau, Bahamas p. 974-979 URL: https://ieeexplore.ieee.org/document/10069420
[doi: 10.1109/ICMLA55696.2022.00163]
Thukral S, Sangwan S, Chatterjee A, Dey L. Identifying pandemic-related stress factors from social-media posts – effects
on students and young-adults. In: Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020. 2020
Presented at: EMNLP '20; November 20, 2020; Virtual Event URL: https://aclanthology.org/2020.nlpcovid19-2.23.pdf
Low DM, Rumker L, Talkar T, Torous J, Cecchi G, Ghosh SS. Natural language processing reveals vulnerable mental
health support groups and heightened health anxiety on reddit during COVID-19: observational study. J Med Internet Res
2020 Oct 12;22(10):e22635 [FREE Full text] [doi: 10.2196/22635] [Medline: 32936777]
Biester L, Matton K, Rajendran J, Provost EM, Mihalcea R. Understanding the impact of COVID-19 on online mental
health forums. ACM Trans Manage Inf Syst 2021 Sep 08;12(4):1-28 [FREE Full text] [doi: 10.1145/3458770]
Marshall C, Lanyi K, Green R, Wilkins GC, Pearson F, Craig D. Using natural language processing to explore mental
health insights from UK Tweets during the COVID-19 pandemic: infodemiology study. JMIR Infodemiology 2022 Mar
31;2(1):e32449 [FREE Full text] [doi: 10.2196/32449] [Medline: 36406146]
Brewer G, Centifanti L, Caicedo JC, Huxley G, Peddie C, Stratton K, et al. Experiences of mental distress during COVID-19:
thematic analysis of discussion forum posts for anxiety, depression, and obsessive-compulsive disorder. Illn Crises Loss
2022 Oct;30(4):795-811 [FREE Full text] [doi: 10.1177/10541373211023951] [Medline: 36199441]
Perrin A, Anderson M. Share of U.S. adults using social media, including Facebook, is mostly unchanged since 2018. Pew
Research Center. 2019 Apr 10. URL: https://www.pewresearch.org/short-reads/2019/04/10/share-of-u-s-adults-… [accessed 2022-02-16]
pushshift / api. GitHub. URL: https://github.com/pushshift/api [accessed 2022-01-16]
Topic modeling. Gensim. URL: https://radimrehurek.com/gensim/ [accessed 2022-02-15]
Goldberg Y, Levy O. word2vec explained: deriving Mikolov et al.'s negative-sampling word-embedding method. arXiv
2014 Feb 15 [FREE Full text] [doi: 10.48550/arXiv.1402.3722]
word2vec. TensorFlow. URL: https://www.tensorflow.org/tutorials/text/word2vec [accessed 2023-03-16]
Embedding projector. Tenserflow. URL: https://projector.tensorflow.org/ [accessed 2023-03-16]
Cox DR. Regression models and life-tables. J R Stat Soc Series B Stat Methodol 1972 Jan;34(2):187-202 [FREE Full text]
[doi: 10.1111/j.2517-6161.1972.tb00899.x]
CDC museum COVID-19 timeline. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/museum/
timeline/covid19.html [accessed 2022-03-16]
Tian X, Batterham P, Song S, Yao X, Yu G. Characterizing depression issues on Sina Weibo. Int J Environ Res Public
Health 2018 Apr 16;15(4):764 [FREE Full text] [doi: 10.3390/ijerph15040764] [Medline: 29659489]
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
Zhu et al
J Med Internet Res 2023 | vol. 25 | e46867 | p. 17
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
26.
27.
28.
29.
30.
31.
32.
33.
34.
Zhu et al
Gurung P, Wagh R. A study on topic identification using K means clustering algorithm: big vs. small documents. Adv
Comput Sci Technol 2017;10(2):221-233 [doi: 10.13140/RG.2.2.16409.98405]
statsmodels.regression.linear_model.OLS. statsmodels. URL: https://www.statsmodels.org/dev/generated/statsmodels.
regression.linear_model.OLS.html [accessed 2023-03-01]
Cavazos-Rehg PA, Krauss MJ, Sowles S, Connolly S, Rosas C, Bharadwaj M, et al. A content analysis of depression-related
Tweets. Comput Human Behav 2016 Jan 01;54:351-357 [FREE Full text] [doi: 10.1016/j.chb.2015.08.023] [Medline:
26392678]
Wang Y, Zhao Y, Zhang J, Bian J, Zhang R. Detecting associations between dietary supplement intake and sentiments
within mental disorder tweets. Health Informatics J 2020 Jun;26(2):803-815 [FREE Full text] [doi:
10.1177/1460458219867231] [Medline: 31566452]
Hswen Y, Naslund JA, Brownstein JS, Hawkins JB. Online communication about depression and anxiety among twitter
users with schizophrenia: preliminary findings to inform a digital phenotype using social media. Psychiatr Q 2018
Sep;89(3):569-580 [FREE Full text] [doi: 10.1007/s11126-017-9559-y] [Medline: 29327218]
Sallyzhu / RedditImpact. GitHub. URL: https://github.com/Sallyzhu/RedditImpact [accessed 2023-04-28]
Kaplan EL, Meier P. Nonparametric estimation from incomplete observations. J Am Stat Assoc 1958 Jun;53(282):457-481
[doi: 10.2307/2281868]
Welcome to the FactorAnalyzer documentation!. FactorAnalyzer. URL: https://factor-analyzer.readthedocs.io/en/latest/
[accessed 2023-03-03]
Gorsuch RL. Factor Analysis. 2nd edition. New York, NY, USA: Psychology Press; Oct 31, 1983.
Abbreviations
ADHD: attention-deficit/hyperactivity disorder
API: application programming interface
BPD: bipolar disorder
LDA: latent Dirichlet allocation
NLP: natural language processing
OCD: obsessive-compulsive disorder
OLS: ordinary least squares
PTSD: posttraumatic stress disorder
Edited by A Mavragani; submitted 28.02.23; peer-reviewed by N Boettcher, M Navarro; comments to author 04.04.23; revised version
received 03.05.23; accepted 09.05.23; published 12.07.23
Please cite as:
Zhu J, Yalamanchi N, Jin R, Kenne DR, Phan N
Investigating COVID-19’s Impact on Mental Health: Trend and Thematic Analysis of Reddit Users’ Discourse
J Med Internet Res 2023;25:e46867
URL: https://www.jmir.org/2023/1/e46867
doi: 10.2196/46867
PMID:
©Jianfeng Zhu, Neha Yalamanchi, Ruoming Jin, Deric R Kenne, NhatHai Phan. Originally published in the Journal of Medical
Internet Research (https://www.jmir.org), 12.07.2023. This is an open-access article distributed under the terms of the Creative
Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly
cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright
and license information must be included.
https://www.jmir.org/2023/1/e46867
XSL• FO
RenderX
J Med Internet Res 2023 | vol. 25 | e46867 | p. 18
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
Step et al
Original Paper
Using the Positive Peers Mobile App to Improve Clinical Outcomes
for Young People With HIV: Prospective Observational Cohort
Comparison
Mary M Step1, MA, PhD; Jennifer McMillen Smith2*, MSSA; Steven A Lewis1,3*, MBA, MPH; Ann K Avery4*, MD
1
College of Public Health, Kent State University, Kent, OH, United States
2
Division of Social Work, Metrohealth System, Cleveland, OH, United States
3
Center for Health Care Research and Policy, Population Health Research Institute, Case Western Reserve University School of Medicine at The
MetroHealth System, Cleveland, OH, United States
4
Division of Infectious Diseases, Case Western Reserve University School of Medicine at The MetroHealth System, Cleveland, OH, United States
*
these authors contributed equally
Corresponding Author:
Mary M Step, MA, PhD
College of Public Health
Kent State University
Lowry Hall, 305b
750 Hilltop Drive
Kent, OH, 44242
United States
Phone: 1 330 672 2630
Fax: 1 330 672 6505
Email: mstep@kent.edu
Abstract
Background: Disparities in HIV outcomes persist among racial, gender, and sexual minorities in the United States. Younger
people face a greater risk of contracting HIV, often living without knowledge of their HIV status for long periods. The Positive
Peers App (PPA) is a multifunctional HIV support tool designed to improve HIV-related clinical outcomes for young people with
HIV. The app was designed according to the specifications of an in-care young adult HIV community in Northeast Ohio. Data
provided in this study provide preliminary evidence of the usefulness of PPA as a relevant tool for engaging this clinical patient
population in care and facilitating viral suppression.
Objective: In this study, we aimed to describe variations in PPA use and examine the associations between use and HIV clinical
outcomes between self-selected user and nonuser cohorts in the same clinical population.
Methods: The PPA was offered free of charge to persons with HIV, aged 13 to 34 years of age, diagnosed with HIV within the
last 12 months, out of care for 6 months during the last 24 months, or not virally suppressed (HIV viral load >200 copies/mL) in
the prior 24 months. Baseline and 6- and 12-month surveys were administered via an audio computer-assisted self-interviewing
system to all participants. The app’s user activity was tracked natively by the app and stored in a secure server. Participant
demographic and HIV care data were extracted from clinical records within 12 months before the start of the study and across
the duration of the study period. HIV care outcomes of PPA users (n=114) were compared with those of nonusers (n=145) at the
end of the study period (n=259).
Results: The analysis showed that younger PPA users (aged 13-24 years) were more likely to obtain HIV laboratories (adjusted
odds ratio 2.85, 95% CI 1.03-7.90) and achieve sustained viral suppression than nonusers (adjusted odds ratio 4.2, 95% CI
1.2-13.9).
Conclusions: The PPA appears to help younger users sustain HIV suppression. The app offers an important tool for addressing
this critical population. The PPA remains in the field and is currently being adopted by other localities to facilitate their efforts
to end the HIV epidemic. Although our reported observational results require additional validation and stringent ongoing
surveillance, the results represent our best efforts in a pilot study to provide a measure of efficacy for the PPA. Next steps include
a large-scale evaluation of the PPA acceptability and effectiveness. Given the building evidence of user reports and outcomes,
the freely available PPA could be a helpful tool for achieving Ending the HIV Epidemic goals.
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 1
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
Step et al
(JMIR Mhealth Uhealth 2022;10(9):e37868) doi: 10.2196/37868
KEYWORDS
mobile health; mHealth; HIV; Positive Peers; retention in care; re-engagement in care; viral suppression
Introduction
Background
Although clinical outcomes have improved in adults diagnosed
with HIV [1], significant disparities remain for young people
[2-4]. Most new HIV diagnoses in the United States are among
adolescents and young adults [5]. Although ≥60% of younger
(aged 13-24 years) people living with HIV are virally
suppressed, significantly fewer Black and Latinx demographic
groups are not [4]. Furthermore, among all young people, close
to 80% are transgender or cisgender males, who most often
(69%) reported HIV transmission as occurring via male-to-male
sexual contact [4,5]. Given these trends, young people with HIV
can experience the intersection of multiple disenfranchised
communities, resulting in compounded stigma, social and family
isolation, and socially determined barriers to HIV care [6-8].
The downstream effects of this burden can determine decreased
lifetime health and overall longevity. Importantly, tailored
mobile health interventions have been shown to effectively
reduce HIV disparities for younger people and those who
identify with a gender or sexual minority identity [9-12].
Mobile health apps can harness the dissemination dynamics of
social media either by linking to existing platforms or by
creating networks of people with similar health challenges [13].
As social media networks allow for a more user-centric,
collaborative communication process, they offer greater
opportunities for engagement with both health information and
similar others [13,14]. However, although several studies have
shown that social media platforms can serve as an effective
channel for disseminating information [15,16], fewer studies
link to health outcomes or identify mechanisms for change [17].
Therefore, mobile platforms that offer an effective interface for
receiving tailored HIV-related information, track use, and afford
users an opportunity to engage in their own recovery may have
a meaningful impact on the HIV care cascade.
Prior Work
The Positive Peers App (PPA) was created as a suite of app
functions that can address the range of possible needs a young
person living with HIV might have [18]. Following the
formation of a community advisory board, we developed specific
technical features that promoted user agency to best address
users’ needs and provide continuous, vetted, and tailored content
directly to demographically defined user groups. The resulting
PPA is the center of a social media–supported network that
consists of a website, Instagram, TikTok, and Twitter feed that
reaches out to the HIV community with a stream of
evidence-based HIV-relevant content, targeted at adolescent
and young adult user groups. PPA functions range from passive
to highly, including the provision of local resources (eg, housing
and counseling), topical blogs, narrative accounts, medication
reminders, a community forum and private chat. Evidence to
date supports the PPA as being acceptable and received as
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
intended by users [18,19]. User feedback suggests that the shared
experience found among PPA users is perceived as both
restorative and transitional [19]. PPA use has been linked to
decreased perceived stigma, and users report that the privacy,
opportunities for private instant chat, and simple
self-management tools provide a useful, safe, and supportive
space protected from discrimination and judgment [19].
Generally, we expect that the more a user engages with
personally relevant aspects of the app, the more likely the person
is to learn from the app content and form internet-based
supportive relationships with other users. This prediction rests
on a user-centric model of mobile app use that suggests user’s
needs and characteristics of the technology interact to determine
user engagement [20-22]. We expect greater engagement with
the mobile app to influence the acceptance of promoted HIV
messaging and increase the likelihood of desirable HIV clinical
outcomes [23].
Goal of This Study
Although users report liking the PPA and community [19], it
is important to evaluate whether the app provides a clinical
benefit to users. Our aim for this study was to determine whether
PPA use provides a clinical benefit to young people living with
HIV. We expect that (1) PPA users will be more engaged in
care than a nonparticipating cohort from the same clinic and (2)
PPA users will demonstrate greater viral suppression than those
who do not use the app. In addition, we aim to learn whether
relevant user demographics or personal characteristics are
associated with app use or whether defined user engagement
groups experience greater or lesser benefits.
Methods
Research Design
The parent study for this work was designed to develop, build,
and analyze the feasibility and acceptability of the PPA by the
targeted user group [18]. This study used a prospective
observational single-cohort design, with measures assessed at
baseline and at 6, 12, and 18 months. This study was designed
after the app was introduced in the field to extend our evaluation
of PPA use to HIV clinical outcomes.
App use was logged in real time and tracked directly by the
operating system of the app. Clinical outcome data before and
after PPA use were obtained from the electronic health record.
We recognize that although a randomized controlled trial is
ideal for isolating predicted effects, this pilot demonstration
project was preceded by a lengthy preliminary design stage that
precluded an additional clinical trial evaluation. Consequently,
a cohort comparison design of PPA users and eligibility-matched
nonusers within the same clinical population during the same
time frame provided a reasonable option for evaluating clinical
outcomes retrospectively [24] during the study period (October
2016 to May 2019).
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 2
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
Ethics Approval
This study was approved by the Institutional Review Board for
Human Research Protections of the MetroHealth System
(IRB:15-00741) on July 20, 2016, and is reviewed annually.
Participants
The study sample was derived from the HIV clinic population
at a public hospital in Cleveland, Ohio, which serves as the
primary source of medical care for the surrounding underserved
neighborhoods in and around Cuyahoga County. Eligibility
requirements for participation included (1) age between 13 and
34 years; (2) receiving HIV care within the health system; and
(3) an HIV diagnosis within the last 12 months, out of care for
6 months during the last 24 months, or lack of viral suppression
(HIV viral load >200 copies/mL) in the previous 24 months.
Essentially, the participants were either newly diagnosed or not
fully engaged in HIV care.
Step et al
variables. Consequently, we categorized the number of user
acts variable to better compare the user’s app activity. Using
the median value as a cutoff point, 3 ordinal categories of PPA
use were created based on the number of actions a user took
during the first 3 months of having the app on their phone: 0,
none; 1, low or moderate (at or below median use); and 2, high
(above median use).
HIV Outcomes
Consistent with the Health Resources Service Administration
Ryan White program standards, HIV outcomes included
engagement in care and HIV viral suppression [26]. Engagement
in care was coded as yes if that had an office visit or laboratory
tests completed at both 6 and 12 months for prestudy or
poststudy entry. Viral suppression was coded yes if the viral
load was less than 200 copies per ml at both 6 and 12 months
following diagnosis.
Study Recruitment and Comparison Cohort
Statistical Analysis
Potential participants were first identified via an electronic
health record query and referrals from clinic staff. At the end
of recruitment, the study sample included 114 young people
with HIV who remained enrolled for the duration of the study
period.
Baseline characteristics were examined across and within
samples of PPA participants and the nonparticipating cohort
where Pearson chi-square test was used to formally test for
differences between the app user and nonuser groups. Full data
were present for all characteristics except race, for which ≤3%
of the values were missing.
A local cohort of young people with HIV not enrolled in the
PPA pilot was identified for comparison with the pilot sample
of PPA users. This comparison cohort (n=259) comprised
patients who met the same eligibility criteria for enrollment in
the parent PPA study and had a visit to the HIV clinic during
the enrollment period but did not enroll to use the app or
participate in the study. A manual chart review confirmed the
eligibility criteria for the entire sample and provided a record
of all clinic visits and laboratory results completed during the
study period. It is unknown whether non-PPA user patients were
invited and declined study participation or were simply not
made aware of the app.
App user activity, including the number of log-ins, features
used, and number of user acts, was examined for PPA
participants during the 6-month period following enrollment.
Inspection of these data showed that app activity tended to
diminish rapidly after 3 months. Consequently, user engagement
with the app was derived from app activity in the first 3 months.
Medians and IQRs were reported for the number of log-ins and
the number of user acts owing to the noted skewness in
distributions. Formal tests of significance were not performed
because of concerns regarding the sample size.
Measurement
Demographic Characteristics
Demographic information was collected from the PPA
participants and comparison cohort. Variables included age,
race, ethnicity, education, employment status, sexual orientation,
and incarceration history, all known social determinants that
influence disparities in HIV outcomes [25-29]. Age was
categorized as 13 to 24, 25 to 29, and 30 to 34 years to facilitate
comparisons among commonly defined age classes in HIV
research [26]. Race and ethnicity were reported using the US
Census Bureau categories. Finally, respondents were asked to
report the number of times they were incarcerated in a jail or
prison. Incarceration history was categorized for analysis as
none, 1 or 2 times, or ≥3 times.
PPA Engagement
PPA engagement among app users was assessed directly from
native app performance data associated with each user and stored
on a secure server. Variables included the number of times the
user logged in and the number of acts the user completed while
logged in. Wide variability was observed across these app
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
To test the contribution of app use to outcomes, 3 separate
logistic regression models were developed by regressing each
HIV outcome (ie, office visits, completion of HIV laboratory
tests and HIV viral suppression) on PPA participation (yes or
no), while controlling for baseline characteristics and measures.
Interaction effects were tested in the models to assess the
potential for effect modification relative to PPA participation
and individual characteristics. For each outcome modeled, odds
ratios and corresponding 95% CIs were reported for PPA
participation versus nonparticipation in either the overall or
stratified models (in cases of significant interaction or effect
modification).
In addition, outcomes were evaluated from the medical records
before participation (before using the PPA) and following
participation (after using the PPA) to determine if those
measures had significantly changed for PPA participants. A
McNemar test of agreement was used as a formal test of
differences. We also examined the differences for each outcome
with respect to app engagement using the categorized version
of the number of user acts described earlier. Each outcome was
evaluated separately for pre-PPA use and post-PPA use
outcomes. Fisher exact tests were performed to test for
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 3
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
Step et al
differences across the 3 categories of user acts. Statistical
significance was determined with a P value cutoff of .05.
Results
Demographic Differences Between PPA User and
Nonuser Comparison Groups
Demographic characteristics between enrolled PPA users and
the comparison cohort at the same clinic were compared. The
unenrolled group were registered patients at the host clinic who
either chose not to enroll or did not learn about the study. Table
1 provides the baseline characteristics and outcomes of the study
sample groups. The young people with HIV studied across the
PPA user and comparison groups were predominately male
(310/373, 83.1%) and Black (257/373, 70.8%). At the start of
the PPA study period, 69.2% (258/373) of patients had been
out of HIV care. PPA participants, relative to the comparison
group, were more likely to be younger, multiracial or other race,
and newly diagnosed.
Table 1. Baseline characteristics of Positive Peers App (PPA) participants and nonparticipant comparison groups.
Characteristic
Total (n=373), n (%)
PPA (n=114), n (%)
Non-PPA (n=259), n (%)
Age group (years)
<.001
13-24
88 (23.6)
40 (35.1)
48 (18.5)
25-29
158 (42.4)
52 (45.6)
106 (40.9)
30-34
127 (34.1)
22 (19.3)
105 (40.5)
Sex at birth
.12
Male
310 (83.1)
100 (87.7)
210 (81.1)
Female
63 (16.9)
14 (12.3)
49 (18.9)
Race
.006
African American
257 (70.8)
78 (68.4)
179 (71.9)
White
83 (22.9)
22 (19.3)
61 (24.5)
Multiracial or other
23 (6.3)
14 (12.3)
9 (3.6)
Newly diagnosed
<.001
Yes
107 (28.7)
45 (39.5)
62 (23.9)
No, noncongenital
252 (67.6)
59 (51.8)
193 (74.5)
No, congenital
14 (3.8)
10 (8.8)
4 (1.5)
Out of care
.008
Yes
258 (69.2)
68 (59.7)
190 (73.4)
No
115 (30.8)
46 (40.4)
69 (26.6)
Office visits 6-12 months prior
.91
Yes
100 (27.2)
31 (27.2)
69 (26.6)
No
273 (72.8)
83 (72.8)
190 (73.4)
HIV laboratory test 6-12 months prior
.88
Yes
80 (21.5)
25 (21.9)
55 (21.2)
No
293 (78.6)
89 (78.1)
204 (78.8)
HIV viral suppression 6-12 months priorb
a
P valuea
.27
Yes
61 (16.4)
15 (13.2)
46 (17.8)
No
312 (83.7)
99 (86.8)
213 (82.2)
P values generated from Pearson χ2 tests.
b
HIV viral suppression defined as not detectable: <200 copies/mL.
PPA Use Across Demographic Groups
Table 2 summarizes the types of PPA used across demographic
groups. A total of 81.6% (93/373) of participants logged on to
the PPA during the first 3 months following enrollment. The
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
median number of log-ins by users was 9 (IQR 4.0-18.0), and
the median number of user acts was 101 (IQR 46-183). Both
median number of log-ins and user acts were lowest for the
oldest 30 to 34 age group (vs other age groups), and the median
number of user acts was higher for White users (vs African
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 4
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
American), single (vs in a relationship), and “nonstraight” (vs
“straight”) users. Although people employed full time logged
into the app more than the other groups, unemployed participants
showed the highest number of user acts. Similarly, the median
number of log-ins was the highest for females, while the median
number of user acts was the highest for males. Both median
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
Step et al
number of log-ins and user acts were highest for Latinx ethnicity
(vs not Latinx), those newly diagnosed with HIV (vs not newly
diagnosed), those carrying private or commercial insurance (vs
other forms of insurance or no insurance), and those without a
prior incarceration history (vs with an incarceration history).
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 5
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
Step et al
Table 2. Positive Peers App use by patient characteristics (months 1-3).
Characteristic
Logged in? (n=114), n (%)
Number of times user logged in (n=93),
median (IQR)
Number of user acts (n=93),
median (IQR)
Yes
No
93 (81.6)
21 (18.4)
9.0 (4.0-18.0)
101.0 (46.0-183.0)
13-24
34 (85)
6 (15)
11.0 (4.0-18.0)
93.5 (48.0-161.0)
25-29
42 (80.8)
10 (19.2)
9.0 (5.0-18.0)
115.5 (65.0-183.0)
30-34
17 (77.3)
5 (22.7)
6.0 (3.0-21.0)
58.0 (30.0-214.0)
Male
82 (82)
18 (18)
8.5 (4.0-18.0)
104.0 (48.0-184.0)
Female
11 (78.6)
3 (21.4)
11.0 (2.0-19.0)
85.0 (30.0-151.0)
African American
61 (78.2)
17 (21.8)
8.0 (3.0-16.0)
94.0 (38.0-171.0)
White
19 (86.4)
3 (13.6)
9.0 (5.0-18.0)
140.0 (63.0-193.0)
Multiracial or other
13 (92.9)
1 (7.1)
9.0 (4.0-19.0)
88.0 (65.0-183.0)
Yes
12 (85.7)
2 (14.3)
12.5 (9.0-23.5)
145.5 (91.5-237.0)
No
81 (81)
19 (19)
8.0 (4.0-16.0)
93.0 (43.0-180.0)
Yes
38 (84.4)
7 (15.6)
12.0 (6.0-18.0)
118.5 (65.0-192.0)
No, noncongenital
46 (78)
13 (22)
6.5 (3.0-18.0)
97.5 (41.0-193.0)
No, congenital
9 (90)
1 (10)
9.0 (2.0-12.0)
65.0 (38.0-94.0)
Straight
21 (80.8)
5 (19.2)
8.0 (2.0-15.0)
70.0 (31.0-150.0)
Not straight
72 (81.8)
16 (18.2)
9.0 (5.0-18.0)
113.5 (57.0-188.0)
HSa graduate
75 (83.3)
15 (16.7)
9.0 (4.0-18.0)
102.0 (45.0-184.0)
Not an HS graduate
18 (75)
6 (25)
8.0 (3.0-18.0)
93.5 (46.0-169.0)
Full time
26 (92.9)
2 (7.1)
11.5 (5.0-18.0)
106.0 (70.0-193.0)
Part time
21 (84)
4 (16)
5.0 (4.0-16.0)
88.0 (42.0-182.0)
Unemployed
37 (75.5)
12 (24.5)
9.0 (5.0-18.0)
113.0 (45.0-183.0)
Otherb
9 (75)
3 (25)
4.0 (2.0-12.0)
84.0 (38.0-150.0)
No insurance
10 (90.9)
1 (9.1)
8.5 (4.0-16.0)
114.5 (48.0-161.0)
Medicaid or Medicare
60 (76)
19 (24)
8.0 (4.0-15.5)
84.5 (36.0-183.5)
Private
13 (100)
0 (0)
12.0 (8.0-22.0)
171.0 (106.0-248.0)
Other insurance
9 (100)
0 (0)
9.0 (5.0-21.0)
94.0 (88.0-137.0)
Single and not dating anyone
64 (83.1)
13 (16.9)
9.0 (4.0-18.5)
104.0 (44.5-199.5)
In a relationship
29 (78.4)
8 (21.6)
8.0 (3.0-16.0)
94.0 (45.0-171.0)
44 (84.6)
8 (15.4)
10.5 (4.5-18.0)
104.0 (42.5-214.5)
Overall
Age group (years)
Sex at birth
Race and ethnicity
Latinx
Newly diagnosed
Sexual preference
School completed
Employment status
Health insurers
Relationship status
Incarceration
0 times
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 6
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
Characteristic
a
Step et al
Logged in? (n=114), n (%)
Number of times user logged in (n=93),
median (IQR)
Number of user acts (n=93),
median (IQR)
Yes
No
1-2 times
26 (83.9)
5 (16.1)
8.5 (5.0-14.0)
97.5 (58.0-169.0)
≥3 times
23 (74.2)
8 (25.8)
6.0 (3.0-18.0)
89.0 (31.0-310.0)
HS: high school.
b
In school, disability.
HIV Outcomes Between PPA Users and Nonusers
We were interested in determining whether there were
significant differences across HIV outcomes within our cohort
of PPA users. Table 3 presents the results of regressing 6- to
12-month postbaseline office visit attendance on PPA
participation while adjusting for baseline characteristics and
prebaseline medical records. The data suggest that no significant
differences in clinic attendance exist based on the PPA
participation status. In other words, the clinical comparison
group was no different than the group of app users in prestudy
period clinic attendance. After the app use period, there were
no significant differences in PPA participation in HIV clinical
outcomes based on race or new diagnosis or out-of-care status.
However, among demographic variables, an interaction effect
was detected for age. Age-stratified results suggested
significantly improved outcomes for PPA users in the youngest
(13-24 years) age group. Across the youngest age groups, PPA
participants were more likely to obtain their HIV laboratory
tests (adjusted odds ratio 2.85, 95% CI 1.03-7.90) than same-age
nonparticipants. Importantly, younger PPA participants were
also more likely to be virally suppressed (adjusted odds ratio
4.22, 95% CI 1.28-13.89) compared with nonparticipants. No
significant differences in PPA participation in either HIV
laboratory tests or viral suppression were observed among the
older age groups.
Table 3. HIV outcomes between Positive Peers App (PPA) users and non-PPA comparison cohort.
Outcomesa, aORb (95% CI)
Office visits
HIV laboratory tests
HIV viral suppression
1.66 (0.99-2.80)
—d
—
Reference
—
—
PPA
—
2.85 (1.03-7.90)
4.22 (1.28-13.89)
Non-PPA
—
Reference
Reference
PPA
—
1.86 (0.84-4.12)
1.07 (0.46-2.50)
Non-PPA
—
Reference
Reference
PPA
—
0.54 (0.18-1.64)
0.45 (0.11-1.75)
Non-PPA
—
Reference
Reference
All patientsc
PPA
Non-PPA
Age 13-24 years
e
Age 25-29 years
Age 30-34 years
a
Outcomes are measured 6-12 months after baseline measure.
b
aOR: adjusted odds ratio.
c
Overall model adjusted for age, sex at birth, race, newly diagnosed, out of care, and 6-12 months prebaseline measures.
d
When modeling outcomes, all-patient models are not relevant when age acts as an effect modifier, and age-stratified models are not relevant when age
does not act as an effect modifier.
e
Age-stratified models adjusted for sex at birth, race, newly diagnosed, out of care, and 6-12 months prebaseline measures.
Effects of PPA Use on HIV Outcomes
Tables 4 and 5 illustrate that engagement in care and HIV viral
suppression significantly improved following participation in
the PPA (27.2% vs 52.6%, 21.9% vs 45.6%, and 13.2% vs
29.8%). Interestingly, before PPA enrollment, eventual high
users of the app were less likely to have had office visits (7/43,
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
15%), to have a HIV laboratory test drawn (4/43, 9%), or to
have been virally suppressed (2/43, 4%) than eventual nonusers
or low to moderate users. As shown in the prestudy columns,
before downloading the PPA, only 15% (7/43) of high users
had office visits compared with 33.3% (7/114) and 36% (17/43)
for eventual nonusers and low to moderate users, respectively.
However, following PPA participation, these groups converged
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 7
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
Step et al
(11/21, 52%; 26/47, 55.3; vs 23/46, 50%), suggesting that app
use facilitated engagement in care and viral suppression for
patients most out of compliance.
Table 4. HIV outcomes (Office visits and HIV labs) before and after Positive Peers App (PPA) participation and stratified by app use (N=114).
Office visits?
Pre-PPA use
Post-PPA use
Overall, n (%)
P valuea
HIV laboratory tests?
Pre-PPA use
Post-PPA use
<.001
P value
<.001
Yes
31 (27.2)
60 (52.6)
Yes
25 (21.9)
52 (45.6)
No
83 (72.8)
54 (47.4)
No
89 (78.1)
62 (54.4)
Yes
7 (33.3)
11 (52.4)
—b
Yes
3 (14.3)
9 (42.9)
—
No
14 (66.7)
10 (47.6)
—
No
18 (85.7)
12 (57.1)
—
Yes
17 (36.2)
26 (55.3)
—
Yes
18 (38.3)
21 (44.7)
—
No
30 (63.8)
21 (44.7)
—
No
29 (61.7)
26 (55.3)
—
Yes
7 (15.2)
23 (50.0)
—
Yes
4 (8.7)
22 (47.8)
—
No
39 (84.8)
23 (50.0)
—
No
42 (91.3)
24 (52.2)
—
.05
.91
—
—
.002
.91
—
Number of user acts, n (%)
None (n=21)
Low-moderate (n=47)
High (n=46)
P valuec
a
P values compare HIV outcomes before and after PPA participation using the McNemar test of agreement.
b
Not applicable.
c
P values compare the categories of user activity against Pre-PPA Post-PPA HIV outcomes using the Fisher exact test.
Table 5. HIV Viral suppression before and after Positive Peers App (PPA) participation and stratified by app use (N=114).
Viral suppression?
Pre-PPA use
P valuea
Post-PPA use
Overall, n (%)
<.001
Yes
15 (13.2)
34 (29.8)
—b
No
99 (86.8)
80 (70.2)
—
Yes
2 (9.5)
5 (23.8)
—
No
19 (90.5)
16 (76.2)
—
Yes
11 (23.4)
16 (34.0)
—
No
36 (76.6)
31 (66.0)
—
Yes
2 (4.4)
13 (28.3)
—
No
44 (95.6)
13 (71.7)
—
—
.02
.72
—
Number of user acts, n (%)
None (n=21)
Low to moderate (n=47)
High (n=46)
P valc
ue
a
P values compare HIV outcomes before and after PPA participation using the McNemar test of agreement.
b
Not applicable.
c
P values compare the categories of user activity against Pre-PPA Post-PPA HIV outcomes using the Fisher exact test.
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 8
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
Step et al
Discussion
for managing the changing health and support needs as young
people living with HIV adjust to a postdiagnosis life.
Principal Findings
Limitations
This study aimed to assess the effects of PPA use on HIV
clinical outcomes within a sample of out-of-care or newly
diagnosed young people in one clinic community. Following
at least 6 months of intervention participation, the youngest
patient group (age 13-24 years) was more likely than the
same-age nonuser cohort to see an improvement in completing
laboratory tests and achieving or sustaining viral suppression.
This is a significant finding because younger people are not
only more likely to be unsuppressed and out of care but also
burdened by psychosocial and behavioral risks that define this
group as a high-impact target population for ending the
HIV/AIDS epidemic [30,31].
There are limitations associated with this work. Research
volunteers may have been more motivated to maintain
adherence, regardless of their enrollment in the mobile app. The
app was assertively promoted in the clinic to all potential
volunteers for 2 years. There are many reasons a patient may
choose not to download a health app, including privacy
concerns, data download costs, and past experience with apps.
We had no quantitative data on people’s reasons for participating
or declining participation. However, selection bias is a known
risk in observational studies and may be evident if PPA users
and nonusers were different in notable ways. Data from Table
1 suggest that nonusers skewed older and were more likely to
be returning after a lapse in care rather than as a newly
diagnosed patient. People returning to HIV care are experiencing
a different affective or psychosocial experience than newly
diagnosed younger adults. We need to determine whether these
differences are sufficient to preclude these patients from
considering using the PPA. In addition, it may be that young
adults aged >25 years are more engaged in employment,
established relationships, or other life responsibilities, reducing
free time for app use. Finally, there were significant and
unexpected differences in app use across races. Additional
qualitative or mixed methods studies may reveal the nature of
these differences and allow us to craft messaging and in-app
tools tailored to their needs. Future analysis beyond this single
clinic population will allow us to better determine additional
interaction effects that may be contributing to this pattern.
It is notable that most PPA use occurred within the first 3
months of downloading the app. Although novelty may draw
users to download and try an app, engagement quickly peaks
as a function of continued exposure [32,33]. It may be that the
knowledge gained from app engagement bolsters greater user
health self-management. Specialized health apps such as PPA
may facilitate positive habits such as tracking medication or
marshaling support during difficult times. However, mobile app
designers may also need to address novelty effects on study
retention and design their recruitment and prospective data
collection accordingly.
Demographic Differences in Use
The most frequent PPA users were aged <30 years, newly
diagnosed, and White or Latinx. Females logged into the app
more often, but males engaged in more acts overall while using
the app, possibly suggesting more surfing within the app or less
purposeful use [34]. For example, females may prefer to use
the app primarily to meet relational needs by using private chat
or community forum functions. We caution that these
demographic differences may be unique to a single-site location
and should not be used for generalization to larger groups.
However, these findings provide a reason to further explore
these patterns across sites with a larger sample size. They could
be markers of more complex behavioral patterns associated with
mobile app uses and effects.
Among all PPA users, engagement in care and viral suppression
improved 6 months from baseline, particularly for those in the
highest app use category. Although we supported our prediction
that greater involvement with the app would lead to greater
effects, the mechanism underlying these effects remains
obscured. Mere exposure to information is insufficient for
explaining complex behavioral processes and outcomes [35].
Emerging models of user engagement with interactive digital
media suggest that physical interaction with, and positive
perceptions of, the technology interface predict greater cognitive
involvement with provided content, which in turn motivates a
user’s intent to manage, apply, and share that content [20,23,36].
Using this theoretical framework, we plan to conduct future
research that will allow us to identify distinct use patterns over
time to better understand how the PPA can be enhanced to
support HIV self-management. We believe that freely available
mobile apps such as the PPA could serve as a significant tool
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
Finally, a limitation is also found in the small pilot study sample
size, limiting our ability to fully adjust data outcomes to
additional relevant influences. The results are representative of
our local public hospital community only. However, we are
currently in the field collecting a larger and more diverse sample
that will enable greater precision in our estimates. Along these
lines, a randomized controlled trial design will present a more
robust test of app impact. A key concern for future research is
the determination of a potential threshold for app engagement
to facilitate positive outcomes. Nevertheless, the results reported
here confirm the usefulness of the PPA as a supportive tool for
young people living with HIV. PPA engagement may be
occurring at a particularly formative time following a new
diagnosis or return to responsible care. In this way, the PPA is
a feasible and effective patient-centered tool for facilitating
engagement in care at a crucial turning point in HIV treatment.
Implications
This study is predicated on the idea that greater engagement
with a mobile app will facilitate the likelihood of achieving
desirable HIV clinical outcomes. This idea of engagement or
involvement in content is fundamental to modeling
communication processes and effects [37]. Our findings support
this hypothesis. Although the relationship between targeted
digital content engagement and positive outcomes was
supported, how that process occurs remains unseen. The
cognitive or affective processes inherent in digital message
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 9
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
processing are at the core of this process but not the sole
determinant of a given outcome.
Current explanations of media use and effects are inclusive of
the characteristics and features of various technologies to explain
and ultimately predict outcomes [17,21,23]. Different
technologies afford different experiences to users. Unlike
general social media apps, the PPA affords users opportunities
to tailor their anonymity as their confidence in the community
grows, provides frequent social and medical information vetted
by credible clinicians, and offers a supportive and monitored
community that shares similar life experiences. Although there
is variability across app users in terms of support needs and
adjustment to living with HIV, these user differences can be
intentionally targeted in app messaging and content [38,39].
This model frames our ongoing work and holds implications
for future theorizing of mobile app uses and effects.
This study also has implications for HIV care. Table 1 shows
the nonuser comparison cohort to include significantly more
people returning to care after a lapse in the medical management
of HIV. This returning-to-care population may face different
psychosocial challenges as they resume an adherent lifestyle.
Continued targeting of this group may address a noted literature
gap regarding best practices for re-engagement in care [40-42].
Importantly, the youngest set of PPA users was most likely to
realize positive outcomes 3 months after enrollment. This
Step et al
population of people with HIV is of greatest concern to ending
the epidemic efforts. Data presented here suggest that the app
had a positive impact on these groups. Counselors and clinics
who care for people living with HIV need socially relevant tools
for younger patients. This is particularly relevant for rural
communities or young people with HIV who desire remote,
around-the-clock community support. Within these communities,
the PPA offers acceptance, tangible support, self-management
tools, and credible HIV-relevant information in one place.
Conclusions
HIV self-management is a significant challenge for young
people that can be alleviated with the use of mobile apps that
bring health information, tools, and supports directly to wherever
they may be. Given that new HIV cases are predominantly
among younger people, this approach is crucial for achieving
an undetectable HIV status for young people living with HIV.
Furthermore, acceptance into a knowing and supportive
community may be a key resource for increasing HIV literacy
and lessening internalized stigma [42]. The PPA is currently
available via Google and iOS app stores, although users are
required to verify their age and diagnosis via an electronic
onboarding system. These data, taken with previously published
results, point to the PPA as a useful tool for helping young
people living with HIV achieve clinical outcomes that will both
preserve their health and contribute to ending the epidemic.
Acknowledgments
The authors would like to thank the Positive Peers App community for their willingness, enthusiasm, and relentless optimism.
The authors would also like to acknowledge the support and collaboration of many professionals of the MetroHealth System who
helped make this work possible. This research was funded by the cooperative agreement number H97HA28892 from the US
Department of Health and Human Services, Health Resources and Services Administration, and HIV/AIDS Special Projects of
National Significance program. Its contents are solely the responsibility of the authors and do not necessarily represent the official
views of the government.
Conflicts of Interest
None declared.
References
1.
2.
3.
4.
5.
6.
7.
8.
Moore RD, Keruly JC, Bartlett JG. Improvement in the health of HIV-infected persons in care: reducing disparities. Clin
Infect Dis 2012 Nov;55(9):1242-1251 [FREE Full text] [doi: 10.1093/cid/cis654] [Medline: 23019271]
Allan-Blitz L, Mena LA, Mayer KH. The ongoing HIV epidemic in American youth: challenges and opportunities. Mhealth
2021;7:33 [FREE Full text] [doi: 10.21037/mhealth-20-42] [Medline: 33898602]
Crepaz N, Hess KL, Purcell DW, Hall HI. Estimating national rates of HIV infection among MSM, persons who inject
drugs, and heterosexuals in the United States. AIDS 2019 Mar 15;33(4):701-708. [doi: 10.1097/QAD.0000000000002111]
[Medline: 30585840]
HIV surveillance supplemental report. Centers for Disease Control and Prevention. 2020. URL: http://www.cdc.gov/hiv/
library/reports/hiv-surveillance.html [accessed 2021-01-05]
HIV in the United States by Age. Centers for Disease Control and Prevention. 2021 May. URL: https://www.cdc.gov/hiv/
group/age/youth/index.html [accessed 2021-09-22]
America's HIV Epidemic Dashboard (AHEAD). AHEAD. URL: https://ahead.hiv.gov/data/viral-suppression [accessed
2021-09-08]
Reisner SL, Jadwin-Cakmak L, White Hughto JM, Martinez M, Salomon L, Harper GW. Characterizing the HIV prevention
and care continua in a sample of transgender youth in the U.S. AIDS Behav 2017 Dec;21(12):3312-3327 [FREE Full text]
[doi: 10.1007/s10461-017-1938-8] [Medline: 29138982]
Shacham E, Estlund AL, Tanner AE, Presti R. Challenges to HIV management among youth engaged in HIV care. AIDS
Care 2017 Feb;29(2):189-196. [doi: 10.1080/09540121.2016.1204422] [Medline: 27397139]
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 10
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
Senn TE, Braksmajer A, Coury-Doniger P, Urban MA, Rossi A, Carey MP. Development and preliminary pilot testing of
a peer support text messaging intervention for HIV-infected Black men who have sex with men. J Acquir Immune Defic
Syndr 2017 Feb 01;74 Suppl 2:S121-S127 [FREE Full text] [doi: 10.1097/QAI.0000000000001241] [Medline: 28079722]
Babel RA, Wang P, Alessi EJ, Raymond HF, Wei C. Stigma, HIV risk, and access to HIV prevention and treatment services
among men who have sex with men (MSM) in the united states: a scoping review. AIDS Behav 2021 Nov;25(11):3574-3604
[FREE Full text] [doi: 10.1007/s10461-021-03262-4] [Medline: 33866444]
Arayasirikul S, Turner C, Trujillo D, Le V, Wilson EC. Efficacy and impact of digital HIV care navigation in young people
living with HIV in San Francisco, California: prospective study. JMIR Mhealth Uhealth 2020 May 08;8(5):e18597 [FREE
Full text] [doi: 10.2196/18597] [Medline: 32383680]
Tanner AE, Song EY, Mann-Jackson L, Alonzo J, Schafer K, Ware S, et al. Preliminary impact of the weCare social media
intervention to support health for young men who have sex with men and transgender women with HIV. AIDS Patient Care
STDS 2018 Nov;32(11):450-458 [FREE Full text] [doi: 10.1089/apc.2018.0060] [Medline: 30398955]
Marcolino MS, Oliveira JA, D'Agostino M, Ribeiro AL, Alkmim MB, Novillo-Ortiz D. The impact of mHealth interventions:
systematic review of systematic reviews. JMIR Mhealth Uhealth 2018 Jan 17;6(1):e23 [FREE Full text] [doi:
10.2196/mhealth.8873] [Medline: 29343463]
Craig E, Wright KB. Computer-mediated relational development and maintenance on Facebook®. Commun Res Report
2012 Apr;29(2):119-129. [doi: 10.1080/08824096.2012.667777]
Walther JB, Jang J. Communication processes in participatory websites. J Comput Mediat Comm 2012 Oct 10;18(1):2-15.
[doi: 10.1111/j.1083-6101.2012.01592.x]
Susarla A, Oh J, Tan Y. Social networks and the diffusion of user-generated content: evidence from YouTube. Inform Syst
Res 2012 Mar;23(1):23-41. [doi: 10.1287/isre.1100.0339]
Merolli M, Gray K, Martin-Sanchez F. Health outcomes and related effects of using social media in chronic disease
management: a literature review and analysis of affordances. J Biomed Inform 2013 Dec;46(6):957-969 [FREE Full text]
[doi: 10.1016/j.jbi.2013.04.010] [Medline: 23702104]
Step MM, McMillen Smith J, Kratz J, Briggs J, Avery A. "Positive Peers": function and content development of a mobile
app for engaging and retaining young adults in HIV care. JMIR Form Res 2020 Jan 30;4(1):e13495 [FREE Full text] [doi:
10.2196/13495] [Medline: 32012035]
Levy MR, Windahl S. Audience activity and gratifications. Commun Res 2016 Jun 29;11(1):51-78. [doi:
10.1177/009365084011001003]
Sundar SS, Limperos AM. Uses and grats 2.0: new gratifications for new media. J Broadcasting Electron Media 2013
Oct;57(4):504-525. [doi: 10.1080/08838151.2013.845827]
Phua J, Jin SV, Kim J. Uses and gratifications of social networking sites for bridging and bonding social capital: a comparison
of Facebook, Twitter, Instagram, and Snapchat. Comput Human Behav 2017 Jul;72:115-122. [doi: 10.1016/j.chb.2017.02.041]
Oh J, Bellur S, Sundar SS. Clicking, assessing, immersing, and sharing: an empirical model of user engagement with
interactive media. Commun Res 2015 Sep 21;45(5):737-763. [doi: 10.1177/0093650215600493]
Step MM, Knight K, McMillen Smith J, Lewis SA, Russell TJ, Avery AK. Positive Peers mobile application reduces stigma
perception among young people living with HIV. Health Promot Pract 2020 Sep;21(5):744-754. [doi:
10.1177/1524839920936244] [Medline: 32757838]
Monitoring and Evaluating Digital Health Interventions A Practical Guide to Conducting Research and Assessment. Geneva:
World Health Organization; 2016.
Johnson Lyons S, Gant Z, Jin C, Dailey A, Nwangwu-Ike N, Satcher Johnson A. A census tract-level examination of
differences in social determinants of health among people with HIV, by race/ethnicity and geography, United States and
Puerto Rico, 2017. Public Health Rep 2022;137(2):278-290. [doi: 10.1177/0033354921990373] [Medline: 33629905]
HIV/AIDS bureau performance measures. HRSA. 2019. URL: https://ryanwhite.hrsa.gov/sites/default/files/ryanwhite/
grants/core-measures.pdf [accessed 2022-09-12]
Rodrigues A, Struchiner CJ, Coelho LE, Veloso VG, Grinsztejn B, Luz PM. Late initiation of antiretroviral therapy:
inequalities by educational level despite universal access to care and treatment. BMC Public Health 2021 Feb 19;21(1):389
[FREE Full text] [doi: 10.1186/s12889-021-10421-8] [Medline: 33607975]
Maulsby CH, Ratnayake A, Hesson D, Mugavero MJ, Latkin CA. A scoping review of employment and HIV. AIDS Behav
2020 Oct;24(10):2942-2955 [FREE Full text] [doi: 10.1007/s10461-020-02845-x] [Medline: 32246357]
Khan MR, McGinnis KA, Grov C, Scheidell JD, Hawks L, Edelman EJ, et al. Past year and prior incarceration and HIV
transmission risk among HIV-positive men who have sex with men in the US. AIDS Care 2019 Mar;31(3):349-356 [FREE
Full text] [doi: 10.1080/09540121.2018.1499861] [Medline: 30064277]
Laurenzi CA, Toit du S, Ameyan W, Melendez-Torres GJ, Kara T, Brand A, et al. Psychosocial interventions for improving
engagement in care and health and behavioural outcomes for adolescents and young people living with HIV: a systematic
review and meta-analysis. J Int AIDS Soc 2021 Aug 02;24(8):e25741 [FREE Full text] [doi: 10.1002/jia2.25741] [Medline:
34338417]
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
Step et al
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 11
(page number not for citation purposes)
JMIR MHEALTH AND UHEALTH
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
Step et al
Wood SM, Dowshen N, Lowenthal E. Time to improve the global human immunodeficiency virus/AIDS care continuum
for adolescents: a generation at stake. JAMA Pediatr 2015 Jul 01;169(7):619-620 [FREE Full text] [doi:
10.1001/jamapediatrics.2015.58] [Medline: 25985061]
Berlyne D. Novelty and curiosity as determinants of exploratory behaviour. Br J Psychol 1950;41(1):68-80.
Dubey R, Griffiths TL. Reconciling novelty and complexity through a rational analysis of curiosity. Psychol Rev 2020
Apr;127(3):455-476. [doi: 10.1037/rev0000175] [Medline: 31868394]
Maddux WW, Brewer MB. Gender differences in the relational and collective bases for trust. Group Processes Intergroup
Relation 2005 Apr 12;8(2):159-171. [doi: 10.1177/1368430205051065]
Montoya RM, Horton RS, Vevea JL, Citkowicz M, Lauber EA. A re-examination of the mere exposure effect: the influence
of repeated exposure on recognition, familiarity, and liking. Psychol Bull 2017 May;143(5):459-498. [doi:
10.1037/bul0000085] [Medline: 28263645]
Coyne SM, Padilla-Walker LM, Howard E. Emerging in a digital world. Emerg Adulthood 2013 Mar 26;1(2):125-137.
[doi: 10.1177/2167696813479782]
Valkenburg, PM. Theoretical foundations of social media uses and effects. In: Nesi J, Teller EH, Prinstein MJ, editors.
Handbook of Adolescent Digital Media Use and Mental Health. Cambridge, United Kingdom: Cambridge University Press;
Jun 2022:39-60.
Volkom M, Stapley J, Amaturo V. Revisiting the digital divide: generational differences in technology use in everyday
life. N Am J Psychol 2014;16(3):557-574 [FREE Full text]
Higa DH, Crepaz N, Mullins MM, Prevention Research Synthesis Project. Identifying best practices for increasing linkage
to, retention, and re-engagement in HIV medical care: findings from a systematic review, 1996-2014. AIDS Behav 2016
May;20(5):951-966 [FREE Full text] [doi: 10.1007/s10461-015-1204-x] [Medline: 26404014]
Henkhaus ME, Hussen SA, Brown DN, Del Rio C, Fletcher MR, Jones MD, et al. Barriers and facilitators to use of a mobile
HIV care model to re-engage and retain out-of-care people living with HIV in Atlanta, Georgia. PLoS One
2021;16(3):e0247328 [FREE Full text] [doi: 10.1371/journal.pone.0247328] [Medline: 33705421]
Risher KA, Kapoor S, Daramola AM, Paz-Bailey G, Skarbinski J, Doyle K, et al. Challenges in the evaluation of interventions
to improve engagement along the HIV care continuum in the united states: a systematic review. AIDS Behav 2017
Jul;21(7):2101-2123 [FREE Full text] [doi: 10.1007/s10461-017-1687-8] [Medline: 28120257]
Paasche-Orlow MK, Wolf MS. The causal pathways linking health literacy to health outcomes. Am J Health Behav 2007;31
Suppl 1:S19-S26. [doi: 10.5555/ajhb.2007.31.supp.S19] [Medline: 17931132]
Abbreviations
PPA: Positive Peers App
Edited by L Buis; submitted 09.03.22; peer-reviewed by J Jones, K Schafer; comments to author 02.05.22; revised version received
01.08.22; accepted 03.08.22; published 28.09.22
Please cite as:
Step MM, McMillen Smith J, Lewis SA, Avery AK
Using the Positive Peers Mobile App to Improve Clinical Outcomes for Young People With HIV: Prospective Observational Cohort
Comparison
JMIR Mhealth Uhealth 2022;10(9):e37868
URL: https://mhealth.jmir.org/2022/9/e37868
doi: 10.2196/37868
PMID:
©Mary M Step, Jennifer McMillen Smith, Steven A Lewis, Ann K Avery. Originally published in JMIR mHealth and uHealth
(https://mhealth.jmir.org), 28.09.2022. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete
bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license
information must be included.
https://mhealth.jmir.org/2022/9/e37868
XSL• FO
RenderX
JMIR Mhealth Uhealth 2022 | vol. 10 | iss. 9 | e37868 | p. 12
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
Original Paper
Fast and Accurate Detection of COVID-19 Along With 14 Other
Chest Pathologies Using a Multi-Level Classification: Algorithm
Development and Validation Study
Saleh Albahli1,2*, BSc, PhD; Ghulam Nabi Ahmad Hassan Yar3*, BS, MS
1
Department of Information Technology, College of Computer, Qassim University, Buraydah, Saudi Arabia
2
Department of Computer Science, Kent State University, Kent, OH, United States
3
Depratment of Electrical and Computer Engineering, Air University, Islamabad, Pakistan
*
all authors contributed equally
Corresponding Author:
Saleh Albahli, BSc, PhD
Department of Information Technology
College of Computer
Qassim University
Buraydah, 51452
Saudi Arabia
Phone: 966 163012604
Email: salbahli@qu.edu.sa
Abstract
Background: COVID-19 has spread very rapidly, and it is important to build a system that can detect it in order to help an
overwhelmed health care system. Many research studies on chest diseases rely on the strengths of deep learning techniques.
Although some of these studies used state-of-the-art techniques and were able to deliver promising results, these techniques are
not very useful if they can detect only one type of disease without detecting the others.
Objective: The main objective of this study was to achieve a fast and more accurate diagnosis of COVID-19. This study proposes
a diagnostic technique that classifies COVID-19 x-ray images from normal x-ray images and those specific to 14 other chest
diseases.
Methods: In this paper, we propose a novel, multilevel pipeline, based on deep learning models, to detect COVID-19 along
with other chest diseases based on x-ray images. This pipeline reduces the burden of a single network to classify a large number
of classes. The deep learning models used in this study were pretrained on the ImageNet dataset, and transfer learning was used
for fast training. The lungs and heart were segmented from the whole x-ray images and passed onto the first classifier that checks
whether the x-ray is normal, COVID-19 affected, or characteristic of another chest disease. If it is neither a COVID-19 x-ray
image nor a normal one, then the second classifier comes into action and classifies the image as one of the other 14 diseases.
Results: We show how our model uses state-of-the-art deep neural networks to achieve classification accuracy for COVID-19
along with 14 other chest diseases and normal cases based on x-ray images, which is competitive with currently used state-of-the-art
models. Due to the lack of data in some classes such as COVID-19, we applied 10-fold cross-validation through the ResNet50
model. Our classification technique thus achieved an average training accuracy of 96.04% and test accuracy of 92.52% for the
first level of classification (ie, 3 classes). For the second level of classification (ie, 14 classes), our technique achieved a maximum
training accuracy of 88.52% and test accuracy of 66.634% by using ResNet50. We also found that when all the 16 classes were
classified at once, the overall accuracy for COVID-19 detection decreased, which in the case of ResNet50 was 88.92% for training
data and 71.905% for test data.
Conclusions: Our proposed pipeline can detect COVID-19 with a higher accuracy along with detecting 14 other chest diseases
based on x-ray images. This is achieved by dividing the classification task into multiple steps rather than classifying them
collectively.
(J Med Internet Res 2021;23(2):e23693) doi: 10.2196/23693
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 1
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
KEYWORDS
COVID-19; chest x-ray; convolutional neural network; data augmentation; biomedical imaging; automatic detection
Introduction
Background
The COVID-19 pandemic has been causing significant health
concerns since 2019. Symptoms of the disease include fever,
cough, headache, and severe respiratory complications, which
can subsequently lead to death. When this disease first started
to spread in December 2019, numerous unknown facts were
reported in Wuhan, China, where the first outbreak occurred
[1]. By early January 2020, the government of China and the
World Health Organization recognized SARS-CoV-2, the novel
coronavirus known to cause COVID-19, as a pathogenic virus
that belongs to the same family (Coronaviridae) as the virus
known to cause severe acute respiratory syndrome (SARS). A
SARS outbreak was previously reported in China in 2002-2003
[2].
Medical x-rays (short for x-radiation) are a form of visible light
rays but with higher energy that penetrate the body to generate
images of tissues and structures within the body, including
bones, chest, and teeth. X-ray imaging is a very effective
diagnostic tool and has been used for several decades by
specialists to detect fractures, certain tumors, pneumonia, and
dental problems [3]. In advanced cases, computed tomography
(CT) can be used to produce a series of body images, which is
later assembled into a 3D x-ray image that is processed by a
computer. However, the traditional x-ray is a lot faster, easier,
cheaper, and less harmful than a CT scan [4].
Research has shown that deep learning can be used to make
predictions based on medical images by extracting characteristic
features, including the shape and spatial rotation, from the
images. Convolutional neural networks (CNNs) have played a
very vital role in feature extraction and learning patterns that
enable prediction. For example, a CNN is used to improve
extraction high-speed video-endoscopy when the training data
is very limited [5]. Advancements in image processing tools
have brought about a radical change in the current techniques
for the detection of pulmonary diseases. Researchers are
employing traditional computer vision as well as deep learning
algorithms to achieve satisfactory performance [3]. Several
primary benefits are strongly correlated with the advancement
of radiographic image classification tools. For example, in rural
areas, owing to a shortage of doctors and places where doctors
cannot be reached, such tools can prove useful. Once these tools
become pervasive in the health care industry, radiologists, clinic
practitioners, and even patients may utilize radiographic image
classification tools to monitor and treat several diseases. As a
result, this can reduce the burden on radiologists all over the
world, by abolishing the requirement to examine every x-ray
image for anomalies. Instead, the doctors will only need to focus
on the patients whose x-ray images are flagged by this tool. The
use of such tools can also eliminate the subjective opinion of
doctors, increase the speed of early diagnosis of disease, and
identify the minor details that may be overlooked by the human
eye in some cases.
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
For this study, owing to computational restraints, we did not
build a model from scratch, as such models require extremely
high-end computers. Rather, we used CNN as a class of deep
neural networks to propose a model to classify COVID-19 x-ray
images from x-ray images of a wide range of chest diseases.
Although the x-ray images of the other diseases are inadequate
for proper training and to achieve state-of-the-art results, we
generalized the data by considering data augmentation. This
mainly rescales the x-ray images and flips them horizontally,
in addition to a few other functionalities such as shift range,
zooming, and rotation.
The strength of this study is that it classifies x-ray images at
two different stages. The first stage involves enhancing the
model to detect COVID-19–specific x-ray images at a faster
speed than x-ray images of other chest diseases. This will result
in a significant increase in the classification speed. Thus,
considering a part of a dataset of chest x-ray (CXR) images for
the analysis will result in low-quality output and unsatisfactory
diagnoses. Accordingly, if the case is not classified as “normal”
or “COVID-19” at this stage, then the classification is continued
to the second stage, which involves classification for 14 other
chest and related conditions (ie, atelectasis, cardiomegaly,
effusion, infiltration, mass, nodule, pneumonia, pneumothorax,
consolidation, edema, emphysema, fibrosis, pleural, and hernia).
This also saves processing power if the x-ray image has been
classified as “normal” or “COVID-19” in the first stage itself.
To further enhance the accuracy of detection, we used UNet to
complete lung and heart segmentation. Because we used a
pretrained model, we were able to independently train 5 different
models for each stage. Models with the best training and test
accuracy were then selected for further analyses.
Based on our findings, we found that ResNet50 is the best model
for classification in both scenarios: classifying 3 classes and 14
classes. Moreover, image segmentation helps in increasing the
classification accuracy by up to 5%. We also trained a model
for all 16 classes and found that classifying for a large number
of classes significantly reduces the overall accuracy of the
model.
The main contributions of this study are as follows:
1.
2.
3.
Introduction of new classification pipeline for more
accurate, automated classification in case of a large number
of classes, primarily to increase the accuracy of a specific
class.
Use of augmentation and semantic segmentation to increase
accuracy of the model.
Comparison between different deep learning models on the
basis of classification in cases of small and large number
of classes.
In this paper, we first review previous studies that used deep
neural networks for the detection of COVID-19 and other chest
diseases. Then, we discuss the datasets used for our experiments
as well as the study methodology, including data preprocessing,
data segmentation, and the setup for classification of the models.
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 2
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Finally, we present the results and analyses on the basis of the
models and dataset available.
Previous Work
Recently, with the rapid development of artificial intelligence,
an increasing number of researchers have begun to pay attention
to intelligent, deep learning−based diagnostic techniques. Some
of them have achieved significantly prominent results. In this
section, we first review the current, state-of-the-art techniques
concerning the application of artificial intelligence to chest
diseases in general, and then, we discuss the literature related
to COVID-19 detection using deep neural networks.
Detection of Chest Diseases Based on CXR Images by
Using Deep Neural Networks
Sivasamy and Subashini [6] used a Keras framework to classify
CXR images to predict lung diseases and reported an accuracy
of 86.14%. The accuracy of the model improved as the number
of epochs for training was increased. Wang et al [7] used
pixel-wise annotated digital reconstructed radiograph data to
train an unsupervised multiorgan segmentation model based on
x-ray images. In this case, the gaps in nodules annotated directly
on 2D x-ray images are quite challenging and time-consuming
due to the projective nature of x-ray imaging. Rajpurkar et al
[8] proposed a binary classifier for the detection of pneumonia
from frontal-view CXR images that achieved an f1 score of
0.435. Salehinejad et al [9] used a Deep Convolutional
Generative Adversarial Network (DCGAN) tailored model
designed for training with x-ray images wherein a generator is
trained to generate artificial CXR images. Their model obtained
its best accuracy when trained on an augmented dataset with
DCGAN-synthesized CXRs to balance the imbalanced real
dataset (D3). Chandra and Verma [10] used 5 different models
to identify pneumonia and reported 95.631% as the best
accuracy. The model is limited to analyzing only nonrigid,
deformable, registration-driven automatically lung regions and
segmented region of interest–confined feature extraction.
Previous studies using state-of-the-art techniques have achieved
effective results with one or two cardiothoracic diseases, but
these techniques could lead to misclassification.
A few techniques have targeted all 14 classes of chest diseases.
Wang et al [11] presented the largest publicly available dataset
of CXR images, which has provided a new dimension to the
research community. They achieved promising results using a
deep CNN and suggest that this dataset could be further
extended by using more disease labels. Smit et al [12] proposed
a deep learning−based technique to identify the 14 underlying
chest diseases. They trained the model to input a single-view
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
Albahli & Yar
chest radiograph and output the probability of each of the 14
observations. Several models were trained to identify the one
with the best accuracy. They used DenseNet121 for their
research and found that it yielded the best accuracy, but it was
limited to the CheXpert dataset and liable to overfitting. A
pretrained DenseNet121 model and feature extraction techniques
were used for accurate identification of 14 thoracic diseases in
the study by Ho and Gwak [13].
Detection of COVID-19 Cases Based on CXR Images
by Using Deep Neural Networks
There are several state-of-the-art studies on deep learning and
machine learning models for COVID-19 diagnosis. A study by
Apostolopoulos and Mpesiana [14] took advantage of CNNs
for the automatic detection of COVID-19 by using CXR images.
They adopted transfer learning to solve for the small image
dataset challenge. Their COVID-19 dataset consisted of 224
sample medical images. Despite the size limitation, their results
showed effective automatic detection of COVID-19−related
diseases. Abbas et al [15] used the CNN-based DeTraC
framework. They also used transfer learning to achieve the best
performance. This model achieved 95.12% accuracy and 97.91%
sensitivity. Chen et al [16] provided a prediction of patients
with or without COVID-19 by using the UNet++ based
segmentation model. Narin et al [17] classified CXR images
using the ResNet50 model and obtained the highest classification
performance with 98% accuracy, using a dataset comprising
only 50 COVID-19 and 50 normal samples. Li et al [18] also
used a ResNet50 model with a dataset comprising 468
COVID-19 samples, 1551 community-acquired pneumonia
samples, and 1445 non-pneumonia samples; this model achieved
90% sensitivity. Using deep learning approaches to extract and
transform features, Li et al proved their model’s efficacy in
COVID-19 diagnosis [18]. Furthermore, Sethy and Behera [19]
used deep learning to extract deep features from x-ray images
and then used state vector machine to classify them into
COVID-19–positive and COVID-19–negative classes; they
achieved an accuracy of 95%. Hemdan et al [20] used transfer
learning and fine-tuning on state-of-the-art networks like VGG
and ResNetV2 to classify COVID-19–positive and
COVID-19–negative x-ray images; they achieved an accuracy
of 90%. Wang et al [21] proposed the M-inception model, a
variant of the inception model. They detected only COVID-19
CT images from all available images and achieved an accuracy
of 82%. Table 1 presents a comparison of previously studies
models using radiographic imaging classification for COVID-19
cases, normal cases, and other chest diseases.
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 3
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
Table 1. Comparison of models detecting COVID-19 cases, normal cases, and other chest diseases based on medical images (data derived from [22]).
Reference
Medical image
Disease detected, n
COVID19
Normal
Other chest diseases
Accuracy (%)
Methodology
Apostolopoulos
and Mpesiana
[14]
X-ray
224
504
700
93
Used transfer learning on Used only 3 classes:
VGG19. MobileNetV2, COVID-19, pneumonia,
Inception, Xception, and and other
InceptionResNetV2
Wang et al [23]
X-ray
53
8066
5526
92
Introduced COVIDUsed only 3 classes:
Net—the first openCOVID-19, pneumonia,
source COVID-19 detec- and normal
tion system
Narin et al [17]
X-ray
50
50
N/Aa
98
Used 5 pretrained networks and applied 3 binary classifications for 4
classes of chest x-rays
Used only 3 classes:
normal, COVID-19, viral and bacterial pneumonia
250
3520
2753
97
Defined 2 models based
on VGG16: one to classify affected x-ray images
from healthy ones and
the other to classify
COVID-19 from affected
x-ray images. Then, they
localized the affected areas.
Although they used xray images of most diseases, they used only 3
classes: COVID-19,
healthy, and disease
Used only 3 classes:
COVID-19, bacterial
pneumonia, and healthy
Brunese et al [22] X-ray
Song et al [24]
CTb
777
708
N/A
86
Proposed DRE-Net and
compared its performance with VGG-16,
DenseNet, and ResNet
Zheng et al [25]
CT
313
229
N/A
90
Proposed DeCoVNet for Used only 2 classes:
classification
COVID-19–positive
and COVID-19–negative
Xu et al [26]
X-ray
219
175
224
86
Proposed ResNet-18
based CNNc network
a
Gaps in classification
Used only 3 classes:
COVID-19, InfluenzaA viral pneumonia, and
normal
Ozturk et al [27]
X-ray
250
1000
500
92
Proposed DarkCovidNet Used only 3 classes:
COVID-19, pneumonia,
and no findings
Ardakani et al
[28]
CT
510
N/A
510
99
Used 10 CNN networks Classified COVID-19
(ie, AlexNet and ResNet- class from
101) for classification of non–COVID-19 class
2 classes
Li et al [18]
CT
1296
1325
1735
96
Proposed COV-Net for
classifying 3 classes
Abbas et al [15]
X-ray
105
80
11
95.12
Proposed DeTracUsed only 3 classes:
ResNet18 CNN that uses normal, COVID-19,
Decompose, Transfer,
and SARS
and Compose architecture
Chen et al [16]
CT
51
N/A
55
95.24
Used UNet++ along with Used only binary classiKeras for segmentation
fication for COVID-19
and COVID-19 detection detection
Used only 3 classes:
COVID-19, community-acquired pneumonia,
and non-pneumonia
N/A: not applicable.
b
CT: computed tomography.
c
CNN: convolutional neural network.
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 4
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Methods
Dataset
The first step involved preprocessing of the data, which includes
segmentation of the lungs and the heart from the whole image,
as an x-ray image contains many unnecessary details. To
perform this segmentation task, we trained the UNet model on
segmented CXR data obtained by the Japanese Society of
Radiological Technology, which were downloaded from their
official website [29], and their corresponding masks, which
were downloaded from the SCR database [30]. This dataset
contains 247 images. For classification purposes, data for
Albahli & Yar
COVID-19 was collected from Cohen et al’s COVID Chest
X-ray dataset [31]. This dataset contains x-ray images of many
other diseases. Furthermore, x-ray images from the datasets
were separated using the available metadata file. Data for the
other 14 chest diseases were provided by the National Institute
of Health (NIH) and can be downloaded from the NIH Chest
X-ray Dataset of 14 Common Thorax Disease Categories [32].
Data available on the NIH Clinical Center website contains
112,120 images, belonging to 15 classes, which include 14
disease classes and 1 normal class—all of which were extracted
through the available metadata file. The number of images per
class is presented in Table 2.
Table 2. Number of images per class in the National Institute of Health Chest X-ray Dataset of 14 Common Thorax Disease Categories [32].
Model and class
Training set, n
Testing set, n
COVID-19
455
22
Normal
1995
405
Other
4600
730
Atelectasis
200
100
Cardiomegaly
200
100
Consolidation
200
100
Edema
200
100
Effusion
200
100
Emphysema
200
100
Fibrosis
200
100
Hernia
150
100
Infiltration
200
100
Mass
200
100
Nodule
200
100
Pleural thickening
200
100
Pneumonia
200
100
Pneumothorax
200
100
Model 1
Model 2
The data were randomly split into training and testing sets, as
there were very few data related to COVID-19. The idea was
to keep the training set as large as possible given the small
number of images present. Image augmentation compensated
for the lack of data. This was not an issue for model 2 images.
For model 1, however, the lack of data can cause a change in
testing accuracy. To compensate for this issue, we also applied
data augmentation while testing.
Data Preprocessing
Every x-ray image has a different contrast and illumination as
they are taken under different lighting conditions. Therefore,
in the first step of preprocessing, histogram equalization was
applied. CXR images also contain unnecessary details, such as
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
the collarbone, shoulders, neck, and torso region. To remove
these unnecessary details, lungs and heart segmentation were
applied. For this purpose, the UNet segmentation model was
trained on images from the Japanese Society of Radiological
Technology with their corresponding masks. The architecture
of the UNet model is shown in Table 3. The input image size
fed to the network was 256×256×3. The contraction part acts
as an encoder that extracts the context from the image using
downsampling through the max-pooling layer. The expansive
path acts as a decoder that precisely localizes the segmentation
part using transpose convolution layers. It is an end-to-end,
fully connected network and does not contain any dense layers.
It also restores the image through upsampling.
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 5
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
Table 3. Architecture of UNet model.
Path, layer, and type
Kernel size
Filters
Input Layer
N/Aa
N/A
2
Convolution
3×3
16
3
Dropout (0.1)
N/A
N/A
4
Convolution
3×3
16
5
MaxPooling
2×2
1
6
Convolution
3×3
32
7
Dropout (0.1)
N/A
N/A
8
Convolution
3×3
32
9
MaxPooling
2×2
1
10
Convolution
3×3
64
11
Dropout (0.2)
N/A
N/A
12
Convolution
3×3
64
13
MaxPooling
2×2
1
14
Convolution
3×3
128
15
Dropout (0.2)
N/A
N/A
16
Convolution
3×3
128
17
MaxPooling
2×2
1
18
Convolution
3×3
256
19
Dropout (0.3)
N/A
N/A
20
Convolution
3×3
256
21
Transposed convolution
2×2
128
22
Concatenate (21, 16)
N/A
N/A
23
Convolution
3×3
128
24
Dropout (0.2)
N/A
N/A
25
Convolution
3×3
128
26
Transposed convolution
2×2
64
27
Concatenate (26, 12)
N/A
N/A
28
Convolution
3×3
64
29
Dropout (0.2)
N/A
N/A
30
Convolution
3×3
64
31
Transposed convolution
2×2
32
32
Concatenate (31, 8)
N/A
N/A
33
Convolution
3×3
32
34
Dropout (0.1)
N/A
N/A
35
Convolution
3×3
32
36
Transposed convolution
2×2
16
37
Concatenate (36, 4)
N/A
N/A
38
Convolution
3×3
16
39
Dropout (0.1)
N/A
N/A
40
Convolution
3×3
16
1
Contraction Path
Expansive Path
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 6
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Path, layer, and type
41
a
Convolution (Sigmoid)
Albahli & Yar
Kernel size
Filters
1×1
1
N/A: not applicable.
Data Augmentation
Before feeding the data to the network, image augmentation
was applied to tackle the problem of fewer data, that is, in the
case of COVID-19. For applying augmentation, the rotation
range was set to 90°, the horizontal flip was set to true, and the
vertical flip was also set to true. For each iteration, the image
data generator used a different transformation of the original
images. In the case of COVID-19, we had 445 input images
and 20 iterations; therefore, the data generator used 8900 images
for training in this case.
Classification Models
The main objective of this study was to classify COVID-19
x-ray images from normal x-ray images and those of 14 other
chest diseases. When a single model is trained for classifying
16 different classes, its accuracy tends to decrease, and in the
case of COVID-19 detection, that is not acceptable. To solve
this problem, a new pipeline was formed, which is illustrated
in Figure 1. Two models were trained. The first model was
trained to classify 3 classes: COVID-19, normal, and some other
disease. The second model was trained to classify the 14 other
chest and related diseases. Both models were trained separately.
To automate the process, if the first model classified the x-ray
as “some other disease,” then the second model was called to
further classify the disease as one of 14 other chest diseases,
using a simple “IF” condition. This architecture makes the
classification process easy, as there are fewer features that need
to be classified at the first stage.
Figure 1. Proposed pipeline of classification.
Classifier 1 only needs to learn how to distinguish COVID-19
and normal x-ray images from those of all the other 14 chest
diseases. The rule is simple: the fewer the classes, the fewer
features there are to learn and distinguish, and the greater the
accuracy. This is critical because the classification of COVID-19
is much more important than that of other diseases during the
ongoing pandemic. Finally, the burden of classifying the other
14 x-ray diseases falls on classifier 2, which now has 14 classes
to classify instead of 16. Furthermore, the 2 most important
classes have already been classified by classifier 1. Moreover,
to support the statement that accuracy indeed decreases when
classifying into 16 classes, a third model was trained for
classification into all 16 classes.
For classification purposes, the following 5 models were trained
for both classifications:
NasNetLarge, Xception, InceptionV3, InceptionResNetV2, and
ResNet50.
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
NasNetLarge was proposed by “Google Brain” in 2018 [33]. It
has two types of architectures—CIFAR10 and ImageNet.
CIFAR10 architecture has N number of normal cells and one
reduction cell repeating after each other; in the end, it has the
SoftMax function. ImageNet has two strides of convolutional
layers with a 3×3 kernel size at the start, followed by two
reduction cells; thereafter, it has the same architecture as
CIFAR10.
Xception was proposed by Google in 2017 [34]. It consists of
one entry flow, eight middle flow, and one exit flow. Entry flow
consists of convolutional and max-pooling layers with ReLU
as the activation function. The middle flow consists of only
convolutional layers with the ReLU activation function. Exit
flow consists of convolutional, max pooling, and global average
pooling layers with the ReLU activation function; in the end,
it has fully connected layers for classification.
InceptionV3 was proposed by Google in 2015 [35]. The basic
architecture of the model is the same, as it consists of
convolutional and pooling layers; in addition, it has three
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 7
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
inception architectures as proposed previously [35]. Finally, at
the end, it has the logistic and SoftMax function for
classification into 1000 classes.
InceptionResNetV2 was proposed by Google in 2016 [36]. It
has the proposed inception and reduction blocks at the start,
and in the end, it has a pooling layer and dropout layer to prevent
overfitting. It classifies using the SoftMax function.
Models were taken from Keras library in Python, which were
initialized with ImageNet weights. These models can classify
1000 classes, but we only needed to classify 3, 14, and 16 classes
for classifier 1, classifier 2, and classifier 3, respectively.
Therefore, these models were fine-tuned, and additional layers
were added. Table 4 shows the fine-tuning layers added at the
end of each pretrained model. The input image size given to the
models was 331×331×3.
ResNet50 was proposed by Microsoft in 2015 [37]. It takes
residual learning as a building block and consists of
convolutional layers with an average pooling layer at the end.
Table 4. Fine-tuning layers for classifier 1, classifier 2, and classifier 3.
Type
a
Classifier 1
Classifier 2
Classifier 3
Output
Kernel
Output
Kernel
Output
Kernel
Average pooling
2048
2×2
2048
2×2
2048
2×2
Flatten
8192
N/Aa
8192
N/A
8192
N/A
Dense
1024
N/A
1024
N/A
1024
N/A
Dropout (0.5)
1024
N/A
1024
N/A
1024
N/A
Dense
1024
N/A
1024
N/A
1024
N/A
Dropout (0.5)
1024
N/A
1024
N/A
1024
N/A
Dense
3
N/A
1024
N/A
16
N/A
N/A: not applicable.
All the models explained in the Methods section were trained
and tested on Google Colab with 12 GB of RAM and GPU
(graphics processing unit) assigned by Google Colab.
Results
of 16.75%, training accuracy of 87.13%, validation loss of
12.27%, and validation accuracy of 89.64%.
Figure 2 shows some sample segmented CXR images. With
image segmentation, we achieved up to 5% increase in the
accuracy of our models.
Initially, the UNet model was trained for segmentation of lungs
and heart. After Training UNet, the model had a training loss
Figure 2. Sample segmented chest x-ray images.
After image segmentation was completed and new training data
were obtained, each training model was trained for 20 epochs
with a batch size of 8. The accuracy obtained from training is
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
shown in Table 5. The table shows that the maximum accuracy
for classifying the 3 classes, including COVID-19 was achieved
by using ResNet50 followed by NasNetLarge. These two models
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 8
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
yielded accuracy that competes with that of the available
state-of-the-art models for COVID-19 prediction.
Classifier 2 did not show promising results in classifying the
14 other diseases. The main reasons for this were the large
number of classes and the continued overfitting of the model.
The maximum test accuracy achieved was 65.63% with
ResNet50 followed by 61.47% with NasNetLarge.
classes. Table 5 shows that when the 16 classes were combined
and classified, the detection accuracy decreases. In all the cases
except in the case of NASNetLarge and ResNet50 models, the
test accuracy decreased when classifying 16 classes. Moreover,
in the case of the NASNetLarge model, the increase in accuracy
is not very notable. The maximum test accuracy was achieved
with ResNet50, with an average of 71.905% over 10-fold
cross-validation.
As described, our proposed model pipeline helps to increase
the accuracy of COVID-19 diagnosis when classifying 16
Table 5. Average training, validation, and test accuracy achieved by different models through a 10-fold cross-validation.
Model and classifier
Training accuracy (%) Training loss (%)
Validation accuracy
(%)
Validation loss (%) Test accuracy (%)
Test loss (%)
1st
91.80
33.78
91.25
31.32
89.66
32.028
2nd
84.67
52.45
61.22
127.83
61.47
127.99
Combined
79.72
68.36
63.68
123.42
63.58
121.39
1st
88.12
29.27
87.33
36.27
86.58
35.91
2nd
90.70
48.73
61.88
133.04
61.08
133.92
Combined
29.88
208.89
22.28
397.08
47.75
301.39
1st
65.87
69.25
63.43
57.38
63.19
5.233
2nd
83.91
56.46
53.57
142.22
53.75
139.97
Combined
65.52
110.33
38.31
176.24
38.83
174.97
1st
65.10
80.32
62.46
76.35
63.19
75.79
2nd
83.37
81.30
54.08
197.66
53.75
197.07
Combined
54.45
134.20
33.84
200.61
33.97
200.26
1st
96.32
9.84
94.16
23.09
92.52
20.32
2nd
87.83
35.85
67.55
105.63
65.63
108.24
Combined
88.92
26.16
73.14
87.05
71.91
88.95
NasNetLarge
Xception
InceptionV3
InceptionResNetV2
ResNet50
The results obtained by our proposed approaches compete with
that of state-of-the-art methods (shown in Table 1). Graphs
illustrating the training and validation accuracy and loss for
classifiers 1, 2, and 3 are shown in Figures 3, 4, and 5,
respectively. To further evaluate the results, the AUC (area
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
under the curve), sensitivity, and specificity results for all the
networks were studied (Table 6). We found that ResNet50
achieved the maximum AUC, sensitivity, and specificity scores
compared to any other model.
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 9
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
Figure 3. Graphs illustrating training and validation accuracy (left) and loss (right) over epochs for different models of classifier 1.
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 10
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
Figure 4. Graphs illustrating training and validation accuracy (left) and loss (right) over epochs for different models of classifier 2.
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 11
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
Figure 5. Graphs illustrating training and validation accuracy (left) and loss (right) over epochs for different models of classifier 3.
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 12
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
Table 6. Average training, validation, and test accuracy achieved by different models through the 10-fold cross-validation.
AUCa (%)
Sensitivity (%)
Specificity (%)
1st
97.61
90.73
93.42
2nd
82.15
75.33
81.1
Combined
93.88
90.15
93.85
1st
95.9
88.64
91.78
2nd
92.25
87.38
81.12
Combined
83.19
76.09
80.64
1st
89.28
83.2
85.51
2nd
91.69
79.53
81.25
Combined
89.85
83.29
93.7
1st
80.65
74.88
77.92
2nd
85.41
79.21
81.25
Combined
85.44
82.69
93.75
1st
98.73
93.14
95.22
2nd
94.6
85.64
81.03
Combined
96.9
93.4
93.72
Model and classifier
NasNetLarge
Xception
InceptionV3
InceptionResNetV2
ResNet50
a
Area under the curve.
Discussion
Principal Findings
In this study, we classified normal cases, COVID-19 cases, and
14 other chest diseases based on CXR images. We proposed a
novel, multiclass method for this purpose and used models that
were pretrained on ImageNet dataset to save training time and
resources. Our multilevel approach resulted in an increase in
the classification accuracy. We found that ResNet50 was the
best model for classification, yielding the highest accuracy.
Future Suggestions
This study tried to cover most aspects of detection of chest
diseases, but there is still work to be done. Most importantly,
there is a need for more data for patients with COVID-19, which
could help improve the accuracy of the model. At present, there
is a significant difference in the number of images per class for
the first level of classification.
This model can help in the first level of classification to
determine whether the person has COVID-19 or some other
chest disease, as x-rays are easier and less expensive than other
forms of radiographic imaging and can help determine the
severity of the disease. Although disease severity was not within
the scope of this study, future work in detecting the severity of
the disease can also be an important improvement in the
already-existing model. In addition, techniques such as the
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
Grad-Cam algorithm can be used to visualize the features in
radiographic images affecting the algorithm and to determine
disease severity. This algorithm will highlight which features
help the algorithm with the classification and which features
likely mislead the algorithm. This algorithm might also be the
key to investigating the low accuracy of the level-2 classifier
and can help improve its accuracy.
Conclusions
Deep learning has played a major role in medical image analysis
and feature extraction, which are applied to the detection of a
wide range of chest diseases. CNN architectures are popular for
their ability to learn mid- and high-level image representations
and to make predictions. Detecting the presence, or absence, of
COVID-19 in a patient is insufficient without addressing other
chest diseases. However, a deep learning system that is trained
to classify a large number of classes—16 in our case—has less
accuracy. This work aimed to deal effectively with this new
pipeline to help with a first-level differential diagnosis of
COVID-19 from other chest diseases. Subsequently, we applied
further enhancement to detect other chest diseases in order to
tackle multi-class chest classification in the detection of
anomalies on x-ray images. This approach yielded satisfactory
results.
Thus, we showed how our proposed models use state-of-the-art
deep neural networks to classify 16 cardiothoracic diseases by
training the models based on x-ray images in the database. Image
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 13
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
segmentation was applied to remove unnecessary details, and
both classifiers were independently trained on segmented data.
However, our model can classify not only COVID-19 but also
Albahli & Yar
14 other chest diseases, as well as normal x-ray images, with
satisfactory accuracy as compared with previous studies.
Acknowledgments
We would like to thank the Deanship of Scientific Research, Qassim University, for funding the publication of this project.
Conflicts of Interest
None declared.
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
Coronavirus disease 2019 (COVID-19) Situation Report – 94. World Health Organization. 2019. URL: https://www.who.int/
docs/default-source/coronaviruse/situation-reports/20200423-sitrep-94-covid-19.pdf [accessed 2021-02-04]
Yang X, Yu Y, Xu J, Shu H, Xia J, Liu H, et al. Clinical course and outcomes of critically ill patients with SARS-CoV-2
pneumonia in Wuhan, China: a single-centered, retrospective, observational study. The Lancet Respiratory Medicine 2020
May;8(5):475-481. [doi: 10.1016/s2213-2600(20)30079-5]
Albahli S. A deep neural network to distinguish COVID-19 from other chest diseases using x-ray images. Curr Med Imaging.
Epub ahead of print posted online on June 4, 2020 2020. [doi: 10.2174/1573405616666200604163954] [Medline: 32496988]
Haglin JM, Jimenez G, Eltorai AEM. Artificial neural networks in medicine. Health Technol 2018 Jul 27;9(1):1-6. [doi:
10.1007/s12553-018-0244-4]
Choe J, Lee SM, Do K, Lee G, Lee J, Lee SM, et al. Deep learning-based image conversion of CT reconstruction kernels
improves radiomics reproducibility for pulmonary nodules or masses. Radiology 2019 Aug;292(2):365-373. [doi:
10.1148/radiol.2019181960] [Medline: 31210613]
Sivasamy J, Subashini T. Classification and predictions of lung diseases from chest x-rays using MobileNet. The International
Journal of Analytical and Experimental Modal Analysis 2020 Mar;12(3):665-672 [FREE Full text]
Wang Y, Zhang Y, Xuan W, Kao E, Cao P, Tian B, et al. Fully automatic segmentation of 4D MRI for cardiac functional
measurements. Med Phys 2019 Jan;46(1):180-189 [FREE Full text] [doi: 10.1002/mp.13245] [Medline: 30352129]
Rajpurkar P, Irvin J, Zhu K, Yang B, Mehta H, Duan T, et al. CheXNet: Radiologist-level pneumonia detection on chest
x-rays with deep learning. arXiv. Preprint posted online on December 25, 2017 [FREE Full text]
Salehinejad H, Valaee S, Dowdell T, Colak E, Barfett J. Generalization of deep neural networks for chest pathology
classification in X-rays using generative adversarial networks. : IEEE; 2018 Presented at: 2018 IEEE International Conference
on Acoustics, Speech and Signal Processing (ICASSP); April 15-20, 2018; Calgary, AB, Canada p. 990-994. [doi:
10.1109/icassp.2018.8461430]
Chandra TB, Verma K. Pneumonia detection on chest x-ray using machine learning paradigm. In: Chaudhuri B, Nakagawa
M, Khanna P, Kumar S, editors. roceedings of 3rd International Conference on Computer Vision and Image Processing.
Advances in Intelligent Systems and Computing, vol 1022. Singapore: Springer; 2019:21-33.
Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. Hospital-scale chest X-ray database and benchmarks on
weakly-supervised classification and localization of common thorax diseases. : IEEE; 2017 Presented at: Proceedings of
the IEEE conference on computer vision and pattern recognition; July 21-26, 2017; Honolulu, HI p. 2097-2106. [doi:
10.1109/cvpr.2017.369]
Smit A, Jain S, Rajpurkar P, Pareek A, Ng AY, Lungren MP. Combining automatic labelers and expert annotations for
accurate radiology report labeling using BERT. In: ACL Anthology.: Association for Computational Linguistics; 2020
Presented at: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP);
November 2020; Online p. 1500-1519. [doi: 10.18653/v1/2020.emnlp-main.117]
Ho TK, Gwak J. Multiple feature integration for classification of thoracic disease in chest radiography. Applied Sciences
2019 Oct 02;9(19):4130. [doi: 10.3390/app9194130]
Apostolopoulos ID, Mpesiana TA. Covid-19: automatic detection from X-ray images utilizing transfer learning with
convolutional neural networks. Phys Eng Sci Med 2020 Jun;43(2):635-640 [FREE Full text] [doi:
10.1007/s13246-020-00865-4] [Medline: 32524445]
Abbas A, Abdelsamea M, Gaber MM. Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional
neural network. Appl Intell 2020 Sep 05;51(2):854-864. [doi: 10.1007/s10489-020-01829-7]
Chen J, Wu L, Zhang J, Zhang L, Gong D, Zhao Y, et al. Deep learning-based model for detecting 2019 novel coronavirus
pneumonia on high-resolution computed tomography: a prospective study. medRxiv. Preprint posted online on March 01,
2020 [FREE Full text] [doi: 10.1101/2020.02.25.20021568]
Narin A, Kaya C, Pamuk Z. Automatic detection of Coronavirus disease (covid-19) using X-ray images and deep
convolutional neural networks. arXiv. Preprint posted online on March 24, 2020 [FREE Full text]
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 14
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
Albahli & Yar
Li L, Qin L, Xu Z, Yin Y, Wang X, Kong B, et al. Using artificial intelligence to detect COVID-19 and community-acquired
pneumonia based on pulmonary CT: evaluation of the diagnostic accuracy. Radiology 2020 Aug;296(2):E65-E71 [FREE
Full text] [doi: 10.1148/radiol.2020200905] [Medline: 32191588]
Sethy PK, Behera SK. Detection of Coronavirus disease (covid-19) based on deep features. Preprints. Preprint posted online
on March 18, 2020 [FREE Full text] [doi: 10.20944/preprints202003.0300.v1]
Hemdan ED, Shouman MA, Karar ME. Covidx-net: A framework of deep learning classifiers to diagnose covid-19 in X-ray
images. arXiv. Preprint posted online on March 24, 2020 [FREE Full text]
Wang S, Kang B, Ma J, Zeng X, Xiao M, Guo J, et al. A deep learning algorithm using ct images to screen for Coronavirus
disease (covid-19). medRxiv. Preprint posted online on March 11, 2020 [FREE Full text]
Brunese L, Mercaldo F, Reginelli A, Santone A. Explainable deep learning for pulmonary disease and coronavirus COVID-19
detection from x-rays. Comput Methods Programs Biomed 2020 Nov;196:105608 [FREE Full text] [doi:
10.1016/j.cmpb.2020.105608] [Medline: 32599338]
Wang L, Lin ZQ, Wong A. COVID-Net: a tailored deep convolutional neural network design for detection of COVID-19
cases from chest X-ray images. Sci Rep 2020 Nov 11;10(1):19549 [FREE Full text] [doi: 10.1038/s41598-020-76550-z]
[Medline: 33177550]
Song Y, Zheng S, Li L, Zhang X, Ziwang Huang Z, Chen H, et al. Deep learning enables accurate diagnosis of novel
coronavirus (COVID-19) with CT images. medRxiv. Preprint posted online on February 25, 2020 [FREE Full text] [doi:
10.1101/2020.02.23.20026930]
Zheng C, Deng X, Fu Q, Zhou Q, Feng J, Ma H, et al. Deep learning-based detection for COVID-19 from chest CT using
weak label. medRxiv. Preprint posted online on March 26, 2020 [FREE Full text] [doi: 10.1101/2020.03.12.20027185]
Xu X, Jiang C, Ma C. Deep learning system to screen coronavirus disease 2019 pneumonia. arXiv. Preprint posted online
on February 21, 2020 [FREE Full text]
Ozturk T, Talo M, Yildirim EA, Baloglu UB, Yildirim O, Rajendra Acharya U. Automated detection of COVID-19 cases
using deep neural networks with X-ray images. Comput Biol Med 2020 Jun;121:103792 [FREE Full text] [doi:
10.1016/j.compbiomed.2020.103792] [Medline: 32568675]
Ardakani AA, Kanafi AR, Acharya UR, Khadem N, Mohammadi A. Application of deep learning technique to manage
COVID-19 in routine clinical practice using CT images: Results of 10 convolutional neural networks. Comput Biol Med
2020 Jun;121:103795 [FREE Full text] [doi: 10.1016/j.compbiomed.2020.103795] [Medline: 32568676]
JSRT chest X-ray dataset. Japanese Society of Radiological Technology. URL: http://db.jsrt.or.jp/eng-04.php [accessed
2021-02-04]
SCR Segmentation Maps for JSRT Chest X-ray Dataset. SCR Segmentation Maps for JSRT Chest X-ray Dataset. SCR
Database: Download. URL: http://www.isi.uu.nl/Research/Databases/SCR/download.php [accessed 2021-02-04]
Cohen JP, Morrison P, Dao L, Roth K, Duong TQ, Ghassemi M. COVID-19 image data collection: prospective predictions
are the future. arXiv. Preprint posted online on June 22, 2020. https://github.com/ieee8023/covid-chestxray-dataset [FREE
Full text]
NIH Chest X-ray Dataset of 14 Common Thorax Disease Categories. National Institutes of Health - Clinical Center. URL:
https://academictorrents.com/details/557481faacd824c83fbf57dcf7b6da9383… [accessed 2021-02-04]
Zoph B, Vasudevan V, Shlens J, Le QV. Learning Transferable Architectures for Scalable Image Recognition. : IEEE;
2018 Presented at: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition; June 18-23, 2018; Salt Lake
City, UT p. 8697-8710 URL: https://ieeexplore.ieee.org/document/8579005 [doi: 10.1109/cvpr.2018.00907]
Chollet F. Xception: Deep Learning with Depthwise Separable Convolutions. : IEEE; 2017 Presented at: 2017 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR); July 21-26, 2017; Honolulu, HI p. 1800-1807 URL:
https://ieeexplore.ieee.org/document/8099678 [doi: 10.1109/cvpr.2017.195]
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the Inception Architecture for Computer Vision. : IEEE;
2016 Presented at: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); June 27-30, 2016; Las
Vegas, NV p. 2818-2826 URL: https://ieeexplore.ieee.org/document/7780677 [doi: 10.1109/CVPR.2016.308]
Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. Inception-v4, inception-ResNet and the impact of residual connections on
learning. : AAAI Press; 2017 Presented at: AAAI'17: Proceedings of the Thirty-First AAAI Conference on Artificial
Intelligence; February 2017; San Francisco, CA p. 4278-4284 URL: https://dl.acm.org/doi/10.5555/3298023.3298188
He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. : IEEE; 2016 Presented at: 2016 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR); June 27-30, 2016; Las Vega, NV p. 770-778 URL:
https://ieeexplore.ieee.org/document/7780459 [doi: 10.1109/cvpr.2016.90]
Abbreviations
AUC: area under the curve
CNN: convoluted neural network
CT: computed tomography
CXR: chest x-ray
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 15
(page number not for citation purposes)
JOURNAL OF MEDICAL INTERNET RESEARCH
Albahli & Yar
DCGAN: Deep Convolutional Generative Adversarial Network
NIH: National Institute of Health
SARS: severe acute respiratory syndrome
Edited by R Kukafka, C Basch; submitted 19.08.20; peer-reviewed by I Apostolopoulos, I Gabashvili; comments to author 24.09.20;
revised version received 12.10.20; accepted 31.01.21; published 10.02.21
Please cite as:
Albahli S, Yar GNAH
Fast and Accurate Detection of COVID-19 Along With 14 Other Chest Pathologies Using a Multi-Level Classification: Algorithm
Development and Validation Study
J Med Internet Res 2021;23(2):e23693
URL: http://www.jmir.org/2021/2/e23693/
doi: 10.2196/23693
PMID: 33529154
©Saleh Albahli, Ghulam Nabi Ahmad Hassan Yar. Originally published in the Journal of Medical Internet Research
(http://www.jmir.org), 10.02.2021. This is an open-access article distributed under the terms of the Creative Commons Attribution
License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any
medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete
bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information
must be included.
http://www.jmir.org/2021/2/e23693/
XSL• FO
RenderX
J Med Internet Res 2021 | vol. 23 | iss. 2 | e23693 | p. 16
(page number not for citation purposes)