Your privacy, your choice

We use essential cookies to make sure the site can function. We also use optional cookies for advertising, personalisation of content, usage analysis, and social media.

By accepting optional cookies, you consent to the processing of your personal data - including transfers to third parties. Some third parties are outside of the European Economic Area, with varying standards of data protection.

See our privacy policy for more information on the use of your personal data.

for further information and to change your choices.

You are viewing the site in preview mode

Skip to main content
Download PDF
Download ePub

Reach, uptake, and psychological outcomes of two publicly funded internet-based cognitive behavioural therapy programs in Ontario, Canada: an observational study

International Journal of Mental Health Systems volume 18, Article number: 34 (2024) Cite this article

Abstract

Background

Access to traditional mental health services in Canada remains limited, prompting exploration into digital alternatives. The Government of Ontario initiated access to two internet-based cognitive behavioral therapy (iCBT) programs, LifeWorks AbilitiCBT and MindBeacon TAiCBT, for adults with mental health issues.

Methods

An uncontrolled observational study utilizing secondary retrospective program data was conducted to evaluate the reach, uptake, and psychological symptom changes among participants engaging with either iCBT program.

Results

Between May 2020 and September 2021, 56,769 individuals enrolled in LifeWorks AbilitiCBT, and 73,356 in MindBeacon TAiCBT. However, substantial exclusions were made: 56% of LifeWorks participants and 68% of MindBeacon participants were ineligible or failed to initiate treatment. Consequently, 25,154 LifeWorks participants and 23,795 MindBeacon participants were included in the analysis. Of these, 22% of LifeWorks and 26% of MindBeacon participants completed over 75% of iCBT treatment. On average, LifeWorks participants received 13 ± SD 7.1 therapist messages and sent 5 ± SD 10.3 messages, while MindBeacon participants received 25 ± SD 20.7 therapist messages and sent 13 ± SD 16.4 messages. LifeWorks included synchronous therapist contact averaging 1.4 ± SD 1.9 h per participant, while MindBeacon was purely asynchronous. Baseline severity of anxiety (37%) and depression symptoms (22%) was higher for LifeWorks participants compared to MindBeacon participants (24% and 10%, respectively). Clinically significant changes in anxiety and depression scores were observed: 22% of LifeWorks and 31% of MindBeacon participants exhibited reliable recovery in PHQ-9 scores, while 26% of LifeWorks and 25% of MindBeacon participants demonstrated reliable recovery in GAD-7 scores.

Conclusion

In conclusion, iCBT programs show promise for engaged participants with varying levels of severity in anxiety and depression symptoms. Future iterations of iCBT should consider adopting a broad entry criterion to iCBT programming to increase accessibility, especially for those with severe symptoms, alongside integrated intake care pathways, and potential payment structure adjustments for iCBT service providers. Taken all together, these factors could temper high dropout rates post-intake assessment. This evaluation underscores the potential and value of digital mental health interventions for individuals with mild to severe anxiety or depression symptoms, emphasizing the importance of addressing participant dropout.

Introduction

Over 5.3 million Canadians require some type of mental health care every year, out of which almost half reported their mental health needs were not met [1, 2]. While effective treatment options for mental health disorders exists, access to mental health services is limited due to long wait and referral times, geographical inequities, shortage of mental health professionals, cost of mental health services, lack of integration with other services, cultural and language barriers, and stigma related to help seeking behaviours [1, 3]. Given the increasing prevalence of mental health disorders, population-level prevention, promotion, and early intervention are required to minimize lasting negative impacts [4,5,6]. At the onset of the COVID-19 pandemic, a rise in virtual care was apparent which provided a unique opportunity to leverage existing digital platforms to address the increase in mental health concerns and reduce strain on an already over-burdened mental health care system [7, 8]. Digital mental health interventions like internet-based cognitive behavioral therapy (iCBT) were deployed at a large-scale, offering an accessible, preventive, and first-line treatment option [8]. As a general psychosocial intervention, iCBT can be an effective treatment for mental health conditions including mood disorders, social anxiety, panic disorders, phobias, substance use disorders, and obsessive-compulsive disorder [9,10,11]. In May 2020, the Ontario government provided funding to scale up and expand virtual mental health service offerings to the general public through two therapist-assisted iCBT programs: LifeWorks AbilitiCBT and MindBeacon TAiCBT [12].

It was imperative to understand who accessed and had change in symptoms to inform future policy and investment decisions. Several clinical trials suggest that iCBT is effective and broadly accessible [10, 11, 13], with growing evidence of successful iCBT implementation as part of routine care in Canadian provinces and other jurisdictions [14,15,16,17,18]. Small randomized controlled trials of iCBT have higher levels of adherence and lower dropout rates often provided to targeted and more symptomatic sub-populations compared to programs that are offered directly to the general population [19, 20]. RCTs can assess the impact of interventions in a more controlled setting, although they are not well suited to evaluate real-world implementation [21].

Real world implementations of iCBT exist such as Improved Access of Psychological Therapies (IAPT) in the United Kingdom, MindSpot, NewAccess, and This Way Up in Australia, and The Online Therapy Unit in Alberta, Canada [14,15,16,17,18]. Although these real-world implementations of iCBT differ in how they were offered whether as standalone courses tailored to address specific mental health issues or for sub-population groups (This Way up, MindSpot, Online Therapy Unit) or as part of stepped care models with a variety of low to high intensity mental health services (NewAccess and IAPT). It is noteworthy to mention that LifeWorks AbilitiCBT and MindBeacon TAiCBT were standalone guided programs but not part of a stepped care pathway. Evidence to inform policy decisions require innovative approaches using readily available data for timely, cost-effective, and robust evaluations [21]. This third-party evaluation intends to add to the growing literature with a head-to-head comparison of two standalone iCBT programs in a large and differentiated population in Ontario, Canada with the following objectives: [1] To determine reach of these iCBT programs to Ontario’s population [2], To describe uptake, completion, and engagement in both iCBT programs, and [3] To explore changes in psychological symptoms of individuals that initiated treatment.

Methods

Description of programs

LifeWorks AbilitiCBT and MindBeacon TAiCBT were predominantly designed as self-referral programs, advertised through Ontario’s provincial website, and through the LifeWorks and MindBeacon websites respectively. Referral through several participating hospitals across Ontario was another referral pathway to the programs though less common. LifeWorks AbilitiCBT and MindBeacon TAiCBT were available to adults in English and French, and are based on Beck’s Cognitive Triad, employing cognitive behavioural therapy principles and techniques [22]. Key program features included an intake assessment to assess primary mental health issues, patient-guided and goal-oriented program components, and communication with a trained therapist during treatment. Some mental health concerns, like pandemic-related anxiety and trauma support, are not classified in the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-IV) [23]. These concerns were only assessed by LifeWorks AbilitiCBT. Prior to commencing either iCBT program, intake assessments obtained baseline PHQ-9 and GAD-7 scores to determine suitability and identify individuals at imminent risk to themselves or others, or those requiring higher levels of care. Such individuals were promptly referred to immediate medical care or existing mental health and substance use services. Ongoing assessments of PHQ-9 and GAD-7 scores and client communication with therapists, enabled continuous evaluation for suicidality and the need for higher levels of care. Key differences between the two programs were how and by whom intake assessment was administered, type of therapist contact provided to participants (asynchronous vs. synchronous communication) and programmatic components based on identified mental health issues. Components of the two programs can be found in Table 1. More information on the clinical features and specific program components can be found on LifeWorks (https://go.lifeworks.com/en-ca/icbt-on) and MindBeacon websites ((https://www.mindbeacon.com/guided-cbt-programs).

Table 1 Program characteristics of LifeWorks AbilitiCBT and MindBeacon TAiCBT
Full size table

Design and ethics

A third-party evaluation was conducted comparing two iCBT programs using secondary retrospective data from individuals who registered for iCBT between May 2020 till September 2021. This study only included individuals who consented to having their data used for evaluation or research related to the programs. A restricted dataset was provided by LifeWorks, thus the number of individuals who initiated LifeWorks AbilitiCBT without consent is unknown. However, 3,460 individuals did not consent for their data to be used for evaluation and initiated MindBeacon TAiCBT. A retrospective study with two treatment groups was conducted. The Women’s College Hospital Institute for Health Care Solution and Virtual Care (WIHV) signed data sharing agreements with LifeWorks AbilitiCBT and MindBeacon TAiCBT respectively to obtain their data for evaluation and research purposes. The Centre for Digital Health Evaluation (CDHE) received approval from the Assessment Process for Quality Improvement Projects (APQIP) at Women’s College Hospital and obtained an exemption letter from the Women’s College Hospital Research Ethics Board under REB # 2021-0057-E in compliance with the Tri-Council Policy Statement (TCPS2).

Engagement

Both program platforms captured data on participant engagement, including patient-therapist communication, program completion rate, and duration of program. Patient-therapist communication was measured with two variables: total number of in application messages sent by each participant to their therapist and total number of in application messages received from therapist. In addition to these two variables, LifeWorks AbilitiCBT collected total number of clinical hours spent by therapist communicating with each patient. Program completion rate was calculated dividing the number of programmatic components a participant completed by the number of programmatic components assigned to a participant. Program completion was calculated and recoded into a categorical variable. It is important to note that MindBeacon TAiCBT had a unique number and combination of programmatic components provided to each participant dependent on their primary mental health issue identified during the intake assessment along with their changing needs and preferences throughout treatment. While completion rates were calculated for both iCBT programs, it is important to acknowledge the different programmatic features as outlined in Table 1. Consequently, completion rates may reflect variations in how participants engage with the distinct features of each program. While these completion rates offer insights, direct comparisons should be approached with consideration of the nuanced programmatic differences. Lastly, duration in program was calculated for each participant that initiated iCBT by subtracting the date and time when they initiated the program by the date and time when they exited or last interacted with the program.

Primary outcome measures

Primary outcome measures for the iCBT programs were change in the 9-Item Patient Health Questionnaire (PHQ-9) [24] and the Generalized Anxiety Disorder 7-Item Scale (GAD-7) [25]. Each participant completed the PHQ-9 and GAD-7 scales pre- and post-treatment. The pre-measurement was self-reported during the intake assessment, while the post-measurement comprised the final scores provided by participants before leaving the program. The PHQ-9 is a self-administered validated questionnaire comprising of nine statements that are based directly on the nine diagnostic criteria for major depressive disorder in the Diagnostic and Statistical Manual of Mental Disorder, 4th edition (DSM-IV) [23]. Research has shown the PHQ-9 to have good internal consistency, sensitivity to change, and strong correlation with subsequent major depression diagnosis [26]. Scores can range from 0 to 27, with a score between 0 and 4 indicating minimal depression, a score between 5 and 9 indicating mild depression, a score between 10 and 14 indicating moderate depression, a score of 15–19 indicating moderately severe depression, and a score between 20 and 27 indicating severe depression [24]. While the PHQ-9 tool is intended to assist health care professionals in screening for and identify depressive symptoms, it is not a substitute for a comprehensive diagnosis by a trained clinician [27]. Scores of 10 and above on the PHQ-9 were categorized as clinical cases [28]. The GAD-7 is a self-administered validated questionnaire comprising of 7 statements that are sensitive to the presence of generalized anxiety disorder along with social phobia, panic disorder and post-traumatic stress disorder [25]. Scores can range from 0 to 21, with a score between 1 and 4 indicating minimal anxiety, a score between 5 and 9 indicating mild anxiety, a score between 10 and 14 indicating moderate anxiety, and a score of 15–21 indicating severe anxiety [25]. Research has shown the GAD-7 to have good reliability and internal consistency, as well as convergent validity [29]. The GAD-7, like the PHQ-9, provides only probable diagnoses that should be confirmed with further evaluation by a trained clinician [25]. Scores of 8 and above on the GAD-7 were categorized as clinical cases [28].

Statistical analysis

Only participants meeting the following criteria were included in the full analysis: completion of an intake assessment, eligibility for iCBT as a suitable treatment, consent for research use of self-reported data, and initiation of iCBT through playlist or module completion. Participants failing to meet any of these criteria were excluded from the full analysis. In the dataset, 25,145 LifeWorks and 23,795 MindBeacon participants met the inclusion criteria, while 31,624 LifeWorks and 49,561 MindBeacon participants were excluded. No missing values were imputed prior to analysis. Descriptive statistics were conducted on demographic and clinical features of participants as well as for the three measures of engagement, including patient-therapist communication, program completion, and duration in program. Next, generalized estimating equation (GEE) models were fitted with PHQ-9 and GAD-7 scores as the dependent variables, and time (1 = pre and 2 = post), baseline PHQ-9 baseline GAD-7, and program completion (1 = Less than 25%, 2 = 26–50%, 3 = 51-74%, and 4 = More than 75%) as the independent variables. An unstructured correlation matrix and maximum likelihood estimation were used, and a Gaussian distribution with a linear response was specified as the primary outcome variables were symmetrical. Clinically significant changes were examined through percentage symptom changes from baseline, with Hedges g effect sizes calculated to measure magnitude of difference between pre- and post- scores using GEE estimated marginal means. Previously determined clinically reliable change cut-offs based on the Improving Access to Psychological Therapies (IAPT) program (GAD-7 change ≥ 4 and PHQ-9 change ≥ 6) were used to categorize each participant’s change in primary outcomes scores [28]. Reliable deterioration rate was calculated based on participants who showed reliable increase between pre and post scores (PHQ-9 score more than 6 and GAD-7 score more than 4) [28]. Recovery rate was calculated based on participants who were considered clinical cases at baseline but were not clinical cases at the end of the program [28]. Clinical cases were defined as participants exceeding accepted clinical thresholds for GAD-7 (≥ 8) or PHQ-9 (≥ 10) [28]. Data was analyzed using Stata BE version 17.

Results

Participant characteristics

Figure 1 details participant flow and final cohorts of participants included in full analysis. Between May 2020 and September 2021, 56,769 people registered for LifeWorks AbilitiCBT and 73,356 people registered for MindBeacon TAiCBT. Of those who registered, only individuals who completed the intake assessment, met eligibility and suitability, consented for their data to be use, and initiated iCBT by completing one or more playlist or module remained in the cohort, leading to a total of 25,145 (44.3%) participants in LifeWorks AbilitiCBT and 23,795 (32.4%) participants in MindBeacon TAiCBT that initiated iCBT and were included in our analyses. Among participants analyzed, 9% (n = 2,154) of LifeWorks and 10% (n = 2,360) of MindBeacon participants were missing pre-post self-reported PHQ-9 scores. Similarly, 14% (n = 3,409) of LifeWorks and 10% (n = 2,350) of MindBeacon participants were missing pre-post self-reported GAD-7 scores. Statistical techniques to impute for missing data were not employed.

Fig. 1
figure 1

Participant Flow Diagram & Cohort Inclusion. Illustrates the progression of LifeWorks and MindBeacon participants from registration through eligibility assessment, iCBT suitability determination, and research data consent, concluding with the formation of the iCBT-initiated participant cohort

Full size image

Demographic characteristics of participants who initiated iCBT through LifeWorks AbilitiCBT and MindBeacon TAiCBT are shown in Table 2 along with provincial or national estimates provided for comparison where available. For both programs, participants were on average between 35.5 ± SD 12.3 years old, predominantly made up of women (LifeWorks [74.7%], MindBeacon [76.3%]), who identified as ‘White’ (LifeWorks [40.5%], MindBeacon [70.8%]), and who resided in large and urban population centres (LifeWorks [65.4%], MindBeacon [71.1%]). Participants were mostly self-referred (LifeWorks [98.7%], MindBeacon [99.6%]), not healthcare workers (LifeWorks [86.8%], MindBeacon [83.8%]), and not post-secondary students (LifeWorks [66.1%], MindBeacon [52.0%]). The distribution of the participant pool was similar to national/provincial estimates with regards to age and location of residence, although it had a higher proportion of those whom identified as woman, ‘White’, healthcare workers, and post-secondary students compared to national/provincial estimates. Individuals characterized as having generalized anxiety disorder represented the largest group for both LifeWorks (57.9%) and MindBeacon (38.5%).

Table 2 Demographics, baseline measures, and engagement of participants that initiated iCBT between May 2020 and September 2021
Full size table

Baseline symptoms and measures of engagement

As shown in Table 2, there was a large proportion of participants at baseline with severe anxiety symptoms, (36.6% of LifeWorks participants and 24.3% of MindBeacon participants). As well, 43.2% of LifeWorks and 30.0% of MindBeacon participants had moderately severe to severe depressive symptoms at baseline. Overall, there was greater baseline severity in anxiety and depression symptoms among LifeWorks participants (36.6%) compared to MindBeacon participants (24.3%). On average, LifeWorks participants spent 13 ± SD 7.1 weeks in the program and MindBeacon participants spent 10 ± SD 4.1 weeks in the program. There was greater message engagement between participants and therapists among MindBeacon participants compared to Lifeworks participants (sent by participant to therapist: 13 ± 17.1 SD versus 5 ± 10.3 SD messages; sent by therapist to participant: 25 ± 20.7 SD versus 13 ± 16.4 SD messages). In terms of direct therapist and participant engagement, LifeWorks therapists spent on average 1.4 ± 1.9 SD clinical hours on each participant during treatment. Clinical hours were not collected for MindBeacon participants. In Fig. 2. Program Completion, there was similar level of completion for all participants, 22.2% (n = 5,526) for LifeWorks participants and 26.3% (n = 6,248) for MindBeacon participants completing more than 75% of the iCBT program.

Fig. 2
figure 2

Program Completion. Showcases program completion rates for participants in LifeWorks AbilitiCBT and MindBeacon TAiCBT programs

Full size image

Program outcomes

Table 3 reports estimated means and effect sizes between pre- and post- scores, percent changes, and reliable change in outcomes. Analysis revealed significant symptom reductions on all measures at post-treatment for both LifeWorks and MindBeacon. Hedges g effect sizes were calculated between pre-test to post-test scores. A small effect was observed on PHQ-9 scores (0.35) and GAD-7 scores (0.35) among LifeWorks AbilitiCBT participants and similarly for MindBeacon TAiCBT participants with PHQ-9 scores (0.34) and GAD-7 (0.32). The greatest percent change was in anxiety scores (17.4%) and depression scores (17.5%) for MindBeacon participants. A total of four GEE models were conducted to explore pre-post self-reported changes in depression and anxiety scores for LifeWorks and MindBeacon participants included in Table 4. GEE models revealed that change in depression scores were associated with greater program completion, higher baseline depression scores, more clinical hours spent by therapist for LifeWorks participants (p < 0.003) and more messages sent by therapists for MindBeacon participants (p < 0.000). GEE models revealed that change in anxiety scores were associated with greater program completion, higher baseline anxiety scores, more messages sent by LifeWorks participants (p < 0.002) and more messages sent by therapists for MindBeacon participants (p < 0.000). Regardless of program, greater program completion and more patient-therapist contact were associated with improvement in anxiety and depression scores.

Table 3 Pre-post change in GAD-7 and PHQ-9 scores for LifeWorks and MindBeacon participants that initiated iCBT between May 2020 and September 2021
Full size table
Table 4 Variables associated with pre-post change in GAD-7 and PHQ-9 scores (GEE models) for LifeWorks and MindBeacon participants that initiated iCBT between May 2020 and September 2021
Full size table

Discussion

In our uncontrolled observational study comparing LifeWorks AbilitiCBT and MindBeacon TAiCBT, we observed mild changes in anxiety and depression symptoms among all participants, which was associated with greater program completion, higher baseline anxiety and depression scores, and greater participant and therapist engagement, despite different therapist engagement offerings (i.e., synchronous & asynchronous vs. asynchronous). We also observed a similar demographic reach of participants in both programs among Ontario’s adult population, which were primarily women, middle-aged, predominantly white, self-referred, and living in a large urban area. At baseline, some participants among this sub-population were experiencing more severe levels anxiety or depression. In terms of engagement and program completion, participants from MindBeacon required more therapist engagement due to its asynchronous (messaging) model compared to participants in the LifeWorks program however, it should be noted that all participants had similar modest completion rates, with only approximately 20–25% of participants completing more than ¾ of their respective programs.

Noticeably, there was many individuals dropping out right after registering for the programs and during the intake assessment. Some plausible reasons for early dropout could be technical or platform difficulties, change in motivation, with participants potentially self-assessing their mental state at intake or determining the program was not suitable for themselves. Nevertheless, adherence to digital mental health therapies has been a longstanding issue, with adherence rates typically higher in study trials in comparison to real-world implementations and routine care [30]. A recent meta-analysis reviewing evidence from 64 randomised controlled trials on iCBT found adherence rates range from 6 to 100% [13]. In comparison, adherence rate for LifeWorks AbilitiCBT was 22.0% and MindBeacon TAiCBT was 26.3%. The variability in iCBT adherence and dropout is noticeable. Future work might consider determining participants that most likely benefit from iCBT from patient experience research and delve deeper into iCBT platform design and features that derive the most benefit in real-world implementation. Unlike randomized controlled trials, real-world implementation of iCBT does not closely control for and collect data on factors that could influence participant progress in the program e.g., motivators and facilitators to initiate iCBT, current psychotropic medication use, or prior and current use of other mental health services. In our study, for participants that disengaged from LifeWorks AbilitiCBT or MindBeacon TAiCBT, there was limited to no follow-up to divulge reasons for dropout and disengagement.

In terms of demographic reach for both LifeWorks and MindBeacon iCBT programs, the average age was 35.5 ± SD 12.3years old, which aligns with the increased prevalence of anxiety and depression among people in early to mid-adulthood in Canada during the pandemic [31, 32]. Only 22.4% of participants who initiated iCBT self-identified as men with previous research showing women having more positive perceptions of iCBT [33], while men had an increased risk of dropout and greater reluctance to seek help for mental health problems [33, 34]. Moreover, national surveys have reported that women had higher rates of screening positive for anxiety or depression compared to men during the pandemic [31, 32]. In line with our study, more women registered for iCBT through Lifeworks and MindBeacon programs, compared to men, while dropout rates were relatively similar among men and women in this study.

Although both programs were promoted to individuals experiencing mild to moderate depressive and/or anxiety symptoms, there was substantial severity at baseline among both programs with LifeWorks participants having greater severity in depression and anxiety symptoms at baseline. Interestingly, there were statistically significant changes observed in PHQ-9 and GAD-7 scores among program participants. For MindBeacon participants, more therapist messages significantly predicted change in PHQ-9 and GAD-7 scores. For Lifeworks participants, a greater number of clinical hours significantly predicted change in PHQ-9 scores and more participant messages significantly predicted change in GAD-7 scores. Some iCBT programs do not incorporate any form of contact with therapists, be it synchronous or asynchronous. Our findings demonstrate that increased contact with therapists correlates with greater clinical improvement over time implying that the presence of live therapists delivering messages is indeed a crucial component of iCBT programs.

The reliable recovery rates for LifeWorks AbilitiCBT (21.6–25.5%) and MindBeacon TAiCBT (24.8–30.7%) were lower compared to population-level digital mental health therapies in other jurisdictions like the Online Therapy Unit (36.3–45.0%), MindSpot (49.8–50.2%), NewAccess (64.0%) This Way Up (55.7%), but higher than IAPT (14.6–14.6%) when counting all participants who initiated treatment [14,15,16,17,18]. For fair comparisons, it is necessary to state programmatic differences between these real-world comparators. The Online Therapy unit offered a transdiagnostic course with five iCBT lessons [18], This Way Up and MindSpot offered a suite of low intensity therapies [14, 17], while IAPT and NewAccess are based on stepped care model with a mixture of low and high intensity therapies [15, 16]. MindBeacon participants exhibited statistically significantly (p < 0.001) higher rates of reliable recovery in PHQ-9 scores compared to LifeWorks participants, whereas the reliable recovery in GAD-7 scores between LifeWorks and MindBeacon participants did not statistically differ. This difference in PHQ-9 scores may be attributed to the nuanced programmatic approaches: LifeWorks AbilitiCBT uses standardized 10–12 modules, while MindBeacon TAiCBT offers personalized playlists tailored to mental health conditions and participants’ specific needs and preferences. The lower reliable recovery rates in both iCBT programs compared to some real-world comparators may stem from challenges like treatment adherence issues, heightened baseline severity, and potential gaps in integration into the larger mental health system. These iCBT implementations occurred in authentic, uncontrolled environments, introducing various confounding factors to reliable recovery rates.

Implications for policy, research, and practice

While these iCBT programs were free and theoretically easy to access, there was a gradual dropout with a small subset engaging with the programs. Optimistically with many participants registering and completing an intake assessment for both programs may indicate peoples’ self-awareness to recognize and assess their mental health state during the pandemic. More importantly, participants with low program completion had pre-post self-reported change in outcomes which indicates participants’ personal preferences on the frequency and intensity with which they choose to engage with iCBT. Additionally with majority of participants entering through the self-referral mode of entry speaks to the low barrier access needed for digital mental health services to be broadly available to the population especially for individuals that are reluctant to access mental health services due to stigma.

Secondly, a considerable proportion of people with severe baseline anxiety and/or depression registered for the iCBT programs with varying levels of program completion. iCBT is a viable service option for patients with mild to moderate mental health disorders [10, 35], but individuals with higher symptom severity also engaged with these programs in the absence of other options [12]. The literature is mixed in determining whether higher baseline severity is a positive or negative predictor of symptom improvement, but people with severe baseline depression or anxiety symptoms had improvements in pre-post self-reported scores while participating in these two programs [36]. Program suitability for individuals with severe baseline symptoms should be decided on a case-by-case basis taking into consideration prior mental health history, psychotropic medication use, willingness to engage in an online program, and case complexity to determine if iCBT could be a standalone treatment option or offered in tandem with other treatment [37]. A broad entry criterion should be adopted to ensure that individuals that would benefit from iCBT including those with severe symptoms get access.

Thirdly, the cost-effectiveness of large publicly funded iCBT programs depends on the payment models and arrangements made between government and service providers. As seen in both programs, there was a high level of drop out during initial stages of registration and intake assessment, with participants dropping out steadily even after initiating iCBT. Both programs were front-loaded with a high reimbursement to LifeWorks and MindBeacon for completed intake assessments, which meant paying for assessments that did not lead to initiation and/or program completion or referral to other external programs if the eligibility and program suitability criteria for participants for iCBT were not met. Publicly funded models should consider how to economically distribute costs across registration, intake assessment and completing iCBT program components.

Lastly, both iCBT programs were standalone external services with little to no integration into the larger mental health system. Around 99.1% of participants were self-referred to the iCBT programs with less than 1% being referred from hospitals and mental health organizations in Ontario. The intake assessments for both programs identified when participants required higher levels of care, were of risk to themselves or others, had accessibility or technological limitations, or lacked engagement early in the program. But there was no formal referral pathway for people deemed not eligible or suitable for the programs to existing mental health services in the larger mental health system - instead external mental health services were suggested, and participants were required to follow up on their own with no data on if they did. This was further supported by a qualitative study exploring patient and therapist experiences with these programs [39]. It should be noted that the standalone nature of both programs allowed for easier implementation and scaling up in a short time span, especially in jurisdictions where publicly accessible mental health services are in its early stages. Optimally, having a more comprehensive intake process with integration to the mental health system could directly triage people to a range of appropriate mental health services depending on their unique and changing mental health needs.

Limitations and strengths

As an uncontrolled observational study, there are several limitations and strengths. There was no active control group, so we cannot derive the true effectiveness of the programs. Future studies could use linkage to health administrative datasets to assess impact on overall health services utilization and compare outcomes to a control group that is matched on a variety of sociodemographic and clinical variables. However, it’s important to note some limitations in our approach. By not imputing missing values, our study may have introduced biases, including sampling bias and potentially decreasing statistical power. This approach could also lead to a biased estimate of parameters and a loss of valuable information on participants that were excluded from the analysis, thereby impacting the generalizability and precision of our results. Therefore, while our findings provide insights, they should be interpreted with caution considering these limitations. Additionally, participants in both iCBT programs may have simultaneously received medication, counselling, or other types of mental health care which may have contributed to improvement, but we do not have information on this from the data. Furthermore, a comprehensive account of the assessment process was not provided, and the programs did not confirm if mental health diagnosis was clinician-driven. It should also be noted that there may have been better suited psychiatric assessments beyond PHQ-9 and GAD-7 to measure change in symptoms for some of the primary mental health issues that were identified, which have impacted our psychological outcome findings. Lastly, paucity of data about reasons for dropout severely limits the interpretation and evaluation of the results. Direct comparisons between programs cannot be made without detailed data on the reasons for drop out which may or may not have been different between the two programs.

While there were limitations, this evaluation generated evidence on a population-level implementation of two iCBT programs with findings on the reach of both programs, levels of engagement and completion, and insights on improvement through change in PHQ-9 and GAD-7 scores. There are a number of strengths in this study including: (a) the total sample size of registered participants of both these programs is much larger than other iCBT programs like This Way Up, NewAccess Online Therapy Unit, (b) unlike other real world iCBT programs like This Way Up, MindSpot, NewAccess, IAPT, and Online Therapy Unit that were evaluated by respective program leads involved in program development and implementation, this evaluation was conducted by a neutral third-party evaluation team contracted by the provincial government, (c) apart from international comparators this is the largest iCBT implementation in Canada to date. Many of the findings and implications from this evaluation informed the successful integration of iCBT as a service offering into the Ontario Structured Psychotherapy (OSP) program which is coordinated by ten partner hospitals to offer iCBT across Ontario, Canada.

Conclusion

Publicly funded iCBT was offered throughout the pandemic in Ontario, free of charge through two programs and expanded rapidly to deliver mental health support during a time of dire need. The scale up of these two iCBT programs was one of the largest population-level implementations of iCBT in Canada. Across both programs, participants were predominantly female, White, in their mid-30s, and residing in large to urban population centres. Only a small subset of participants completed 75% of iCBT treatment through either program. Yet our study emphasizes clinically significant pre-post self-reported changes: 26% of LifeWorks and 25% of MindBeacon participants demonstrated reliable recovery in GAD-7 scores, and 22% of LifeWorks and 31% of MindBeacon participants exhibited reliable recovery in PHQ-9 scores.

Moving ahead, MindBeacon TAiCBT with another iCBT program were integrated in the Ontario Structured Psychotherapy (OSP) program to provide iCBT to up to 9,000 individuals [38]. Evaluation results were reported to the provincial government on an ongoing basis and used to inform next steps. Ultimately, it is important to highlight what worked, for whom, and what could have been improved for future large-scale digital mental health interventions and evaluations. There were several publicly funded iCBT programs offered across Canada during the pandemic to offset mounting mental health concerns and increase access to care, but there are no publicly available data on their performance. Future iterations of iCBT should consider the following: (1) Acknowledging that low barrier mental health services with participant-led engagement and completion lead to reduction in symptoms, (2) Adopting a broad entry criteria as initial results suggest individuals with severe anxiety or depression reported improvement but needs to be substantiated with more research, (3) Considering the high level of drop out after intake assessment, different funding models should be adopted to separate comprehensive assessment from single program intake so people can be directed to more suitable resources and incentivize retention and (4) Integrating and building seamless connectivity of iCBT programs to the larger mental health care system.

Data availability

The data that support the findings of this study are available on request from the corresponding author, B.K. The data are not publicly available due to restrictions whereby including that could compromise the privacy of research participants.

References

  1. ICES [Internet]. [cited 2023 Jul 3]. ICES | Opening Eyes, Opening Minds: The Ontario Burden of Mental Illness and Addictions Report. https://www.ices.on.ca/publications/research-reports/opening-eyes-opening-minds-the-ontario-burden-of-mental-illness-and-addictions-report/

  2. Mental health care needs. 2018 [Internet]. [cited 2023 Jul 3]. https://www150.statcan.gc.ca/n1/pub/82-625-x/2019001/article/00011-eng.htm

  3. Asmundson GJG, Blackstock C, Bourque MC, Brimacombe G, Crawford A, Deacon SH et al. Easing the disruption of COVID-19: supporting the mental health of the people of Canada—October 2020—an RSC Policy Briefing. Blais JM, editor. Facets. 2020;5(1):1071–98.

  4. Singh V, Kumar A, Gupta S. Mental Health Prevention and Promotion—A narrative review. Front Psychiatry. 2022;13:898009.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Shah N, Naik Y, Rajan S, O’Hagan K, Black M, Cartwright C, et al. National or population level interventions addressing the social determinants of mental health – an umbrella review. BMC Public Health. 2021;21(1):2118.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Almansa J, Alonso J, Angermeyer MC, Brugha TS, Codony M, De Girolamo G et al. Population level of unmet need for mental healthcare in Europe. Br J Psychiatry. 2018/01/02 ed. 2007;190(4):299–306.

  7. Kronfli K. COVID19 policy brief-realizing the full potential of virtual care in Ontario. Ontario Chambers of Commerce; 2020.

  8. Falk W. The State of Virtual Care in Canada as of Wave Three of the COVID-19 Pandemic.

  9. Strudwick G, Sockalingam S, Kassam I, Sequeira L, Bonato S, Youssef A, et al. Digital Interventions To Support Population Mental Health in Canada during the COVID-19 pandemic: Rapid Review. JMIR Ment Health. 2021;8(3):e26550.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Kumar V, Sattar Y, Bseiso A, Khan S, Rutkofsky IH. The effectiveness of internet-based cognitive behavioral therapy in treatment of Psychiatric disorders. Cureus. 2017;9(8):e1626.

    PubMed  PubMed Central  Google Scholar 

  11. Saad A, Bruno D, Camara B, D’Agostino J, Bolea-Alamanac B. Self-directed technology-based therapeutic methods for Adult Patients Receiving Mental Health Services: systematic review. JMIR Ment Health. 2021;8(11):e27404.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Ontario Ministry of Health [Internet]. [cited 2023 Jul 3]. Ontario Expands Virtual Mental Health Services During COVID-19. https://news.ontario.ca/en/release/56847/ontario-expands-virtual-mental-health-services-during-covid-19

  13. Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis. J Anxiety Disord. 2018;55:70–8.

    Article  CAS  PubMed  Google Scholar 

  14. Sharrock MJ, Mahoney AEJ, Haskelberg H, Millard M, Newby JM. The uptake and outcomes of internet-based cognitive behavioural therapy for health anxiety symptoms during the COVID-19 pandemic. J Anxiety Disord. 2021;84:102494.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Baigent M, Smith D, Battersby M, Lawn S, Redpath P, McCoy A. The Australian version of IAPT: clinical outcomes of the multi-site cohort study of NewAccess. J Ment Health Abingdon Engl. 2023;32(1):341–50.

    Article  Google Scholar 

  16. Richards DA, Borglin G. Implementation of psychological therapies for anxiety and depression in routine practice: two year prospective cohort study. J Affect Disord. 2011;133(1–2):51–60.

    Article  PubMed  Google Scholar 

  17. Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, et al. The first 30 months of the MindSpot Clinic: evaluation of a national e-mental health service against project objectives. Aust N Z J Psychiatry. 2017;51(12):1227–39.

    Article  PubMed  Google Scholar 

  18. Hadjistavropoulos HD, Peynenburg V, Thiessen DL, Nugent M, Karin E, Staples L, et al. Utilization, patient characteristics, and longitudinal improvements among patients from a provincially funded Transdiagnostic Internet-delivered cognitive behavioural therapy program: Observational Study of trends over 6 years. Can J Psychiatry. 2022;67(3):192–206.

    Article  PubMed  Google Scholar 

  19. van Ballegooijen W, Cuijpers P, van Straten A, Karyotaki E, Andersson G, Smit JH, et al. Adherence to internet-based and face-to-face cognitive behavioural therapy for Depression: a Meta-analysis. García AV. Editor PLoS ONE. 2014;9(7):e100674.

    Article  Google Scholar 

  20. Dedert E, McDuffie JR, Swinkels C, Shaw R, Fulton J, Allen KD et al. Computerized Cognitive Behavioral Therapy for Adults with Depressive or Anxiety Disorders [Internet]. Washington (DC): Department of Veterans Affairs (US); 2013 [cited 2023 Jul 3]. (VA Evidence-based Synthesis Program Reports). http://www.ncbi.nlm.nih.gov/books/NBK269001/

  21. Guo C, Ashrafian H, Ghafur S, Fontana G, Gardner C, Prime M. Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. NPJ Digit Med. 2020;3:110.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Gratzer D, Khalid-Khan F. Internet-delivered cognitive behavioural therapy in the treatment of psychiatric illness. Can Med Assoc J. 2016;188(4):263–72.

    Article  Google Scholar 

  23. Association AP. Desk reference to the diagnostic criteria from DSM-5[TM]. No Title [Internet]. [cited 2023 Jul 3]; https://cir.nii.ac.jp/crid/1130282272154863488

  24. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–13.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–7.

    Article  PubMed  Google Scholar 

  26. Kroenke K, Spitzer RL, Williams JBW, Löwe B. The Patient Health Questionnaire somatic, anxiety, and depressive Symptom scales: a systematic review. Gen Hosp Psychiatry. 2010;32(4):345–59.

    Article  PubMed  Google Scholar 

  27. Reynolds WM. The PHQ-9 works well as a screening but not diagnostic instrument for depressive disorder. Evid Based Ment Health. 2010;13(3):96–96.

    Article  PubMed  Google Scholar 

  28. NHS England ». The NHS Talking Therapies manual [Internet]. [cited 2023 Jul 3]. https://www.england.nhs.uk/publication/the-improving-access-to-psychological-therapies-manual/

  29. Plummer F, Manea L, Trepel D, McMillan D. Screening for anxiety disorders with the GAD-7 and GAD-2: a systematic review and diagnostic metaanalysis. Gen Hosp Psychiatry. 2016;39:24–31.

    Article  PubMed  Google Scholar 

  30. Baumel A, Edan S, Kane JM. Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs. Transl Behav Med. 2019;9(6):1020–33.

    PubMed  Google Scholar 

  31. Government of Canada SC. Gender differences in mental health during the COVID-19 pandemic [Internet]. 2020 [cited 2023 Jul 3]. https://www150.statcan.gc.ca/n1/pub/45-28-0001/2020001/article/00047-eng.htm

  32. Government of Canada SC. Canadians report lower self-perceived mental health during the COVID-19 pandemic [Internet]. 2020 [cited 2023 Jul 3]. https://www150.statcan.gc.ca/n1/pub/45-28-0001/2020001/article/00003-eng.htm

  33. Schmidt ID, Forand NR, Strunk DR. Predictors of Dropout in Internet-based cognitive behavioral therapy for Depression. Cogn Ther Res. 2019;43(3):620–30.

    Article  Google Scholar 

  34. Lynch L, Long M, Moorhead A, Young, Men. Help-Seeking, and Mental Health Services: exploring barriers and solutions. Am J Mens Health. 2018;12(1):138–49.

    Article  PubMed  Google Scholar 

  35. Young C, Campbell K, Dulong C. Internet-Delivered Cognitive Behavioural Therapy for Major Depression and Anxiety Disorders: A Review of Clinical Effectiveness [Internet]. Ottawa (ON): Canadian Agency for Drugs and Technologies in Health; 2018 [cited 2023 Jul 3]. (CADTH Rapid Response Reports). http://www.ncbi.nlm.nih.gov/books/NBK538735/

  36. Kambeitz-Ilankovic L, Rzayeva U, Völkel L, Wenzel J, Weiske J, Jessen F, et al. A systematic review of digital and face-to-face cognitive behavioral therapy for depression. NPJ Digit Med. 2022;5(1):144.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Duffy D, Enrique A, Connell S, Connolly C, Richards D. Internet-delivered cognitive behavior therapy as a prequel to face-to-face therapy for depression and anxiety: a naturalistic observation. Front Psychiatry. 2020;10:902.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Ontario cuts funding. by 85% for online therapy program introduced during pandemic | CBC News [Internet]. [cited 2023 Jul 3]. https://www.cbc.ca/news/canada/toronto/ontario-icbt-program-cuts-covid19-1.6548962

  39. Thapar S, Nguyen M, Khan BN, Fanaieyan R, Kishimoto V, Liu R, Bolea-Alamañac B, Leon-Carlyle M, O’Riordan A, Keresteci M, Bhattacharyya O. Patient and therapist perceptions of a publicly funded internet-based cognitive behavioral therapy (iCBT) program for Ontario adults during the COVID-19 pandemic: qualitative study. JMIR Formative Res. 2024;8(1):e50113.

    Article  Google Scholar 

  40. Population counts:. age and gender | 2016 census highlights | ontario.ca [Internet]. [cited 2023 Jul 3]. http://www.ontario.ca/document/2016-census-highlights/fact-sheet-3-population-counts-age-and-gender

  41. Table 3 Population by province and territory, Canada, 2010 [Internet]. [cited 2023 Jul 3]. Available from: https://www150.statcan.gc.ca/n1/pub/89-503-x/2010001/article/11475/tbl/tbl003-eng.htm

  42. Government of Canada SC. Census Profile, 2016 Census [Internet]. 2017 [cited 2023 Jul 3]. Available from: https://www12.statcan.gc.ca/census-recensement/2016/dp-pd/prof/index.cfm?Lang=E

  43. Archived - Health care sector injury statistics | Health care sector plan 2017-18 | ontario.ca [Internet]. [cited 2023 Jul 3]. Available from: http://www.ontario.ca/document/health-care-sector-plan-2017-18/health-care-sector-injury-statistics

  44. Government of Canada SC. Total population and farm population for rural areas and population centres classified by sex and age, Agriculture–Population Linkage, 2016 [Internet]. 2018 [cited 2023 Jul 3]. Available from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=3210001201

Download references

Funding

This work was funded through the Ontario Ministry of Health as funds provided to the Centre for Digital Health Evaluation at Women’s College Hospital, Toronto, Ontario. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Women’s College Hospital, 76 Grenville St, Toronto, ON, M5S 1B2, Canada

    Bilal Noreen Khan, Rebecca H. Liu, Cherry Chu, Blanca Bolea-Alamañac, Megan Nguyen, Serena Thapar, Roz Fanaieyan, Marisa Leon-Carlyle, Mina Tadrous, Anne O’Riordan, Maggie Keresteci & Onil Bhattacharyya

  2. Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada

    Bilal Noreen Khan, Megan Nguyen & Paul Kurdyak

  3. Department of Psychiatry, University of Toronto, Toronto, ON, Canada

    Blanca Bolea-Alamañac

  4. Department of Psychology, McGill University, Montreal, QC, Canada

    Serena Thapar

  5. Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, ON, Canada

    Mina Tadrous

  6. Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    Paul Kurdyak

  7. Centre for Addiction and Mental Health, Toronto, ON, Canada

    Paul Kurdyak

  8. Patient Advisors Network, Toronto, ON, Canada

    Anne O’Riordan & Maggie Keresteci

  9. Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada

    Onil Bhattacharyya

Authors
  1. Bilal Noreen Khan
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Rebecca H. Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Cherry Chu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Blanca Bolea-Alamañac
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Megan Nguyen
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Serena Thapar
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Roz Fanaieyan
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Marisa Leon-Carlyle
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Mina Tadrous
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Paul Kurdyak
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. Anne O’Riordan
    View author publications

    You can also search for this author inPubMed Google Scholar

  12. Maggie Keresteci
    View author publications

    You can also search for this author inPubMed Google Scholar

  13. Onil Bhattacharyya
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

B.K., C.C., M.N., R.L., R.F., and S.T., coordinated data transfer and assisted in interpretation, B.K., R.L., and and C.C. drafted the manuscript. B.K., R.L., B.BA., M.CL., and M.T. led in the conception of the study design and analysis. B.K. and C.C. conducted the analysis. B.K., R.L., and O.B. revised the manuscript. O.B. and P.K. supported the conception of the analytic approach. A.O. and M.K. provided formatting and grammar edits. All authors read and approved the final manuscript submitted for publication.

Corresponding author

Correspondence to Bilal Noreen Khan.

Ethics declarations

Ethical approval

The Centre for Digital Health Evaluation (CDHE) received approval from the Assessment Process for Quality Improvement Projects (APQIP) at Women’s College Hospital and obtained an exemption letter from the Women’s College Hospital Research Ethics Board under REB # 2021-0057-E. The Research Ethics Board (REB) of Women’s College Hospital ensured that all research activities conducted within or on behalf of Women’s College Hospital, met standards in accordance with the Tri-Council Policy Statement (TCPS2). Participants retroactively provided consent to both vendors: LifeWorks AbilitiCBT and MindBeacon TAiCBT, for their data to be used for evaluation and research purposes.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, B.N., Liu, R.H., Chu, C. et al. Reach, uptake, and psychological outcomes of two publicly funded internet-based cognitive behavioural therapy programs in Ontario, Canada: an observational study. Int J Ment Health Syst 18, 34 (2024). https://doi.org/10.1186/s13033-024-00651-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13033-024-00651-9

International Journal of Mental Health Systems

ISSN: 1752-4458