Mental Health Program Evaluation

Mental Health Program Evaluation: Effectiveness in Reducing Hospitalizations and Homelessness

John Bennett, M.D. and Ralph Aquila, M.D. March 23, 1995 This is an example of a program evaluation done in the Public Psychiatry Fellowship of the New York State Psychiatric Insitute/Columbia University College of Physicians and Surgeons. It is not intended to be a research study. Rather, it is an example of a management tool which can be done with existing databases and without additional funding. Comments would be appreciated. Please send them to jmr1@columbia.edu .

I. Introduction

A. Overview

Determining the prevalence of mental illness in the homeless population is difficult for many reasons (1,2). However, current estimates are that one-third of homeless persons suffer from mental illness (3). Bachrach noted that the literature underscores the need for service providers to approach homeless mentally ill individuals in nontraditional ways (1). Engagement of homeless mentally ill persons in community-based care does not mean that they will necessarily "stay put" in the residences offered to them. There is a widely recognized pattern of cycling between short-stay facilities and episodic homelessness. Clinically managed, functionally integrated housing programs are needed to meet the diverse needs of mentally ill persons who experience homelessness, at least episodically. Clinicians who treat the homeless mentally ill agree about the need to link housing and clinical services (1,4).

St. Luke's-Roosevelt Hospital Residential Community Services (RCS) program seeks to provide psychiatric services to homeless and formerly homeless mentally ill persons in a nontraditional manner. It is a hospital based program which has formed an alliance with several non-profit organizations that operate supportive residences and drop-in centers on the west side of Manhattan. The alliance provides a comprehensive program of housing, case management and psychiatric services including access to day treatment programs, emergency services and inpatient treatment. Eleven supportive residences and four drop-in centers contract with RCS for on-site psychiatric services. The level of support and supervision provided at each residence differs based upon the staff/client ratio, staff training and other resources available (i.e. prepared meals). One commonality is that the residents are formerly homeless and have a chronic psychiatric illness.

A primary objective of RCS is to develop a continuum of supportive housing and treatment programs in order to improve care, decrease hospitalizations, and prevent return to homelessness. A review by the program director in December, 1993 estimated that chronic psychiatric illness was present in 95% of the patients. Schizophrenia, schizoaffective and bipolar disorder was the diagnosis in 90% of the persons, with substance abuse present in 60 - 75% of the patients. The drop- out rate from the residences was 3% per year with a re- hospitalization rate of less than 5% per year. The hospitalization rate is the total number of hospitalizations for all patients over the course of a year. The drop-out rate reflects those who left the residences and returned to homelessness. That is, it could not be confirmed that they moved to another residence, in with friend or family, or independent living. It is the anecdotal experience of another psychiatrist who works in the program, Steve Theccanat M.D., that RCS does not significantly decrease hospitalizations.

B. Literature review:
A number of studies evaluate the impact of supportive housing programs on hospitalizations and homelessness.

1. McCarthy and Nelson described 34 residents in 7 supportive housing programs in Ontario. They compared the two years prior to entry into a housing program and one year after entry and found a reduction in hospitalization days. Two years prior to entry into a housing program the mean number of hospitalization days was 36+/-88 days per resident. One year prior to entry the mean was 53+/-71 days per resident. One year after entry into a housing program the mean number of hospitalization days was reduced to 0.5+/-2.5 days per resident (5).

2. Caton et. al. looked at 42 homeless mentally ill men who were in a shelter program in New York City which provided psychosocial counseling, medication maintenance, and housing placement. Eighty-one percent had a prior psychiatric hospitalization, 76% had a diagnosis of schizophrenia or schizoaffective disorder, and 67% had a substance abuse diagnosis. A comparison between the 6 months before entering the program and 6 and 18 months after placement in community housing showed hospitalization rates went from 29% before to 26% at 6 months and 38% at 18 months. The percentage who lived in a shelter was 67% in the before period, 10% at 6 months and 44% at 18 months. Sixty-five percent of those who returned to families or their own apartment became homeless again at 18 months in contrast to 18% (3 of 17) discharged to supportive housing (6,7).

3. Hawthorne et. al. retrospectively studied two transitional community residential facilities in California for 104 patients with "multiple failures at community tenures." One year mean number of hospital admissions and days in a hospital or crisis center during the two years before program entry were compared with the year after completing the program. Admissions and days were significantly reduced during the follow-up year. Mean number of hospitalizations decreased from 2.28 plus or minus 1.26 per year to .56 plus or minus .62 per year and mean days in the hospital or crisis center went from 73.09 to 6.95 (8).

4. Lipton et. al. randomized 49 homeless chronic mentally ill patients who were selected at time of inpatient admission to either placement in a residential treatment program or "routine discharge planning" (i.e. discharge to shelter, transfer to state hospital, discharge to friend). The residence provided such services as individual case management, medication monitoring, referrals to psychosocial and rehabilitation programs and on-site psychiatric treatment. One year after study admission, 69% of the experimental group were in permanent housing compared to 30% of the control group. Experimental subjects spent a mean of 55 nights of the study year in a hospital compared to 168 nights for the controls. However, index hospitalization for experimental group was 22 nights and for control group was 52 nights. No significant difference was found in the mean number or length of readmissions to psychiatric hospitals or in the proportion of the total year spent undomiciled. However, when looking at the proportion of the nights spent homeless after the index discharge, experimental subjects were homeless for 6% of nights compared with 46% for controls (9).

5. Summary: The data can be difficult to compare because the housing programs and clinical interventions are variable. The following table provides a summary of the information available after involvement in the study programs:



 Study   Hospitalization  Mean Hospital Days       Drop-out Rate

             Rates          per Resident         

     

McCarthy

                              

2 years pre     -            36+/-88      		   -

1 year  pre     -            53+/-71      		   -

1 year  post   9%(1)          0.5+/-2.5    		   -



Caton                                                  (in shelter)



6 months pre    29%             -  			  67%

6 months post   26%             -  			  10%

18 months post  38%             -  			  44%



Hawthorne                                               (homeless)



Avg 2 years pre  228%         73.09     		   22%

1 year post       56%          6.95 			    5%



Lipton                                  



Experimental         -             33 (2)	            31%

Control              -            116  (2)		    70%

(1) Estimated based on the mean hospitalization days of 0.5 for 34 residents. This would be a total of 17 hospitalization days. Assuming this accounted for 3 hospitalizations the rate would be 9%.

(2) Days not including index hospitalization

The review supports the concept that supportive housing programs decrease the rate of hospitalization, the total hospitalization days and return to homelessness. However, there were some inconsistent findings. For instance, the study by Lipton et. al. did not show a significant difference in the mean number or length of readmissions between a group in a supported housing program and a group who received "routine" discharge planning. There was, however, a decrease in total nights homeless after the index hospitalization. Caton et. al. found that the benefits of a shelter support program prior to housing placement were no longer apparent 18 months after placement. They did find that supportive housing appeared to decrease the chance of returning to homelessness as compared to living independently or with family. There was also significant variability in the mean hospitalization rates and mean days spent hospitalized in a year as noted in the above table. The studies noted the limitation of small sample size, case attrition, lack of comparison and control groups, and selection bias in community housing programs.

C. Objective: The purpose of this program evaluation will be to determine the hospitalization rate and drop out rate for clients served by the RCS program.

II. Methods and Design

For purposes of this evaluation, data was collected on the 11 supported housing programs. The 4 drop-in centers are excluded as they are transitional in nature and this is a different population. The RCS psychiatrists were surveyed to determine how many patients were hospitalized from 1/1/94 to 12/30/94. Multiple hospitalizations for one patient were counted as separate episodes in determining the mean number of hospitalizations. This appears to be consistent with studies reviewed in the literature. The clinical directors at each of the residential sites were contacted to verify the number of hospitalizations and determine the number of clients who leave. Information obtained also included the length and outcome of hospitalizations (i.e. Did the person return to the residence?). It should be noted that at some of the sites, there are residents who receive psychiatric services at a community clinic or are currently not involved in treatment. All clients at a residence have been included in this evaluation as RCS psychiatrists are often involved in crisis intervention and facilitating hospitalization. The drop out rates from the residence were obtained as a measure of return to homelessness. Drop outs were those persons who left the residence and returned to the streets or shelter system or for whom location could not be determined.

III. Results



Site	No.Patients	Hospitalizations  Hospital Days   Drop-outs



Sun          34                6              89                3

Cluster      35                4                        

Travelers    30                1              32                2

RCCA         55                5                        

Senate       50                4              73                1

Wanaque      40                7             133          

37th         39                4              28           

FH Storefront35                1                        

Metro        70                5                        

Euclid       60                1                        

Encore       35                1              20           



Totals       483              39 

           

Overall Hospitalization Rate   8%

Mean hospitalization days per resident for the 6 residences with available information as noted above is 1.6 days.

IV. Discussion

There are a number of limitations inherent to this evaluation including the lack of a comparison group, limitations of retrospective evaluations and selection bias of who is admitted to a residence. The RCS psychiatrist is involved in the screening process at the residences and admission is often dependent upon a willingness by the individual to continue in treatment and possibly participate in a day treatment program. Another method of evaluation would be to look at the year prior to admission to the residence and compare this to the first year in the residence in a pre-test post-test design.

Comparison between the results of this program evaluation and the studies reviewed is difficult and limited because of the variability in the housing programs, the clinical services provided and the study designs. All studies did appear to involve persons with SPMI. Based upon the data currently obtained the hospitalization rate for Residential Community Services is 8%. This is higher than the estimate by the program director one year earlier of 5%. Possible reasons include the increased number of residences now contracting with RCS or more accurate data since the clinical directors at the residence were involved. The 8% figure is still lower than that found in the studies reviewed which ranged from 9% to 56%. The mean hospitalization days for the data currently available is 1.6. This is similar to the McCarthy and Hawthorne studies which were 0.5 and 6.95, respectively.

The study by Lipton et. al. deserves further discussion as it is the only randomized controlled study and appears to be a program most similar to the RCS in that on-site psychiatric services were provided. The residence in this study was St. Frances supported housing program in Manhattan. The study did not give the hospitalization rates and counted the index admissions in the total number of days spent hospitalized in the first year. However, when removing the index hospitalization, the mean number of hospitalization days during the first year was 33. This is considerably higher than that obtained for RCS. A possible explanation may be the difference in the patient population. The population in the Lipton study was admitted to a housing program after an index hospitalization while homeless. Many of the residents served by RCS have been in housing for longer than the year studied. It will be interesting to compare the drop-out rates for these programs once the information is obtained.

VI. Bibliography

1. Bachrach LL: What We Know About Homelessness Among Mentally Ill Persons: An Analytical Review and Commentary. Hospital and Community Psychiatry 43: 453-464, 1992

2. Hamid WA, Wykes T, Stansfeld S: The Homeless Mentally Ill: Myths and Realities. The International Journal of Social Psychiatry 38: 237- 254, 1993

3. Fischer PJ, Nuehring EM, Bestman EW: Mental Health Problems Among Homeless Persons: A Review of Epidemiological Research From 1980 to 1990. Treating the Homeless Mentally Ill. pp 75-94, APA Press, 1992

4. Bebout RB, Harris M: In Search of Pumpkin Shells: Residential Programming for the Homeless Mentally Ill. Treating the Homeless Mentally Ill. pp 159-182, APA Press, 1992.

5. McCarthy J, Nelson G: An Evaluation of Supportive Housing for Current and Former Psychiatric Patients. Hospital and Community Psychiatry 42: 1254-1256, 1991

6. Caton CLM, Wyatt RJ, Grunberg J, Felix A: An Evaluation of a Mental Health Program for Homeless Men. American Journal of Psychiatry 147: 286-289, 1990

7. Caton CLM, Wyatt RJ, Felix A, Grunberg J, Dominguez, B: Follow- Up of Chronically Homeless Mentally Ill Men. American Journal of Psychiatry 150: 1639-1642, 1993

8. Hawthorne WB, Fals-Stewart W, Lohr JB: A Treatment Outcome Study of Community-Based Residential Care. Hospital and Community Psychiatry 45: 152-155, 1994

9. Lipton FR, Nutt S, Sabatini A: Housing the Homeless Mentally Ill: A Longitudinal Study of a Treatment Approach. Hospital and Community Psychiatry 39: 40-45

		Click on these buttons to visit our journals