Treatment Integrity: Why it is Important Regardless of Discipline

Denys Brand, PhD
University of Kansas

In 1988, Van Houten and colleagues published an article in the Journal of Applied Behavior Analysis outlining six fundamental rights of individuals receiving services based on the principles of behavior analysis. One of these included the right to receive the most effective procedures available. There are several methods to ensure that this fundamental right is honored. One method is to implement procedures for which there exist a robust body of empirical literature documenting the efficacy and effectiveness of that procedure (Behavior Analyst Certification Board, 2014). Another is to ensure high levels of treatment integrity while implementing those effective procedures. Treatment integrity (also known as procedural integrity, procedural fidelity, and treatment fidelity) is the extent to which interventionists implement procedures in a manner consistent with their prescribed protocols (e.g., DiGennaro Reed & Codding, 2014, Peterson, Homer, & Wonderlich, 1982). This article discusses treatment integrity and the importance thereof, not just in the field of behavior analysis but across disciplines.

In preparation for this article, a PsycINFO® search was conducted to identify the number of peer-reviewed articles, regardless of discipline, with the terms treatment integrity, treatment fidelity, procedural integrity, and procedural fidelity in the title of the article. The search identified 225 articles, which included experimental work, literature reviews, and commentaries. A closer look at the articles that the search identified indicates that treatment integrity is a topic of empirical interest across numerous disciplines, including intellectual and developmental disabilities, dementia care, special education, mental health, addiction and substance abuse, social work, psychotherapy, and cognitive therapy, just to name a few. The search also indicated that since the year 2000, there has been a sharp increase in the number of articles published on treatment integrity. Of the 225 articles identified, 205 (91%) have been published since the year 2000. It is important to provide a rationale for why studying treatment integrity is important, regardless of discipline, since it is a topic to which much recent empirical research has been dedicated.

Implications for Research

An overview of the published literature on treatment integrity reveals that it is an important methodological concern, regardless of discipline, for two main reasons: research and practice (Hagermoser Sanetti & Kratochwill, 2009). From a research perspective, high levels of treatment integrity are essential in reaching accurate conclusions regarding functional relations between dependent (i.e., outcomes) and independent variables (i.e., procedures). That is, high levels of treatment integrity increase the internal validity of a study and minimize the possibility that variables not related to the study are effecting outcomes (Cook & Campbell, 1979). Internal validity is an important indicator regarding the quality and rigor of a study (Bruhn, Hirsch, & Lloyd, 2015). Compromised treatment integrity can lead to erroneous conclusions regarding treatment effectiveness. For example, compromised levels of treatment integrity can produce outcomes suggesting that effective procedures are ineffective and vice versa (Gresham, Gansle, & Noel, 1993).

Either one of these conclusions can have detrimental effects on the development of effective procedures. For example, if a procedure is implemented with the aim of increasing prosocial behavior in a child with autism, and no increases in behavior is shown, it may lead the interventionist to conclude that the procedure was ineffective and that more intensive training is required. However, without measuring the integrity with which the procedure was administered, it remains unclear whether the procedure itself was ineffective, or whether the procedure would have been effective had prescribed protocols been followed. If the interventionist concludes that the procedure was ineffective, it becomes less likely that the procedure will be used with other clients in the future. Low levels of treatment integrity also adversely affect external validity (i.e., the degree to which the results from a study can be generalized across people, settings, and behaviors) and the ability of other researchers to replicate the procedures and find similar results, which can also have a detrimental effect on the development of effective procedures (Baer, Wolf, & Risley, 1968).

Unfortunately, reporting quantifiable measures of treatment integrity within published literature is not the norm. Several literature reviews have been conducted across various disciplines to assess the frequency with which treatment integrity data are reported. The results from these reviews paint a somewhat grim picture and should motivate researchers and journals to pay more attention to the reporting of treatment integrity data. Monchar and Prinz (1991) conducted a review involving journals across four disciplines (psychiatry, clinical psychology, behavior therapy, and marital and family therapy) for articles published between 1980 and 1988. Of the 359 articles reviewed, only 67 (19%) reported treatment integrity data, although the review showed that articles published towards the end of the review period were more likely to report treatment integrity. Wheeler, Baggett, Fox, and Blevins (2006) reviewed articles across a range of behavior analytic journals involving treatments for people with intellectual and Development Disabilities (I/DD). Out of the 60 articles included in the review, 41 (67%) did not provide any treatment integrity data. Other reviews involving psychotherapy (Perepletchikova, Treat, & Kazdin, 2007), correctional treatment programs (Andrews & Dowden, 2005), and aphasia treatments (Hinckley & Douglas, 2013) have found similar results. For example, the Hinckley and Douglas (2013) review found that of the 149 aphasia treatment studies included in the review, only 21 (14%) reported treatment integrity. Despite these numbers, it is encouraging to see that various disciplines have begun the process of self-evaluation with respect to reporting treatment integrity data. For some related fields, such as gerontology, occupational therapy, and performance management, literature reviews outlining the frequency with which treatment integrity data are reported, to the best of our knowledge, do not exist (Bruhn et al., 2015). Treatment integrity literature reviews usually contain recommendations calling for journals and authors to measure and report treatment integrity data as often as possible and with increased frequency. I wish to add my voice to these recommendations.

Implications for Clinical Practice

Treatment integrity is also important from a clinical practice perspective. Research investigating the relationship between treatment integrity and client outcomes are either correlational (e.g., Downs, Downs, & Rau, 2008), or experimental (e.g., St. Peter Pipkin, Vollmer, & Sloman, 2010). Correlational studies involve recording data on how changes in client behavior coincide with changes in treatment integrity. Experimental studies involve arranging conditions in which treatment integrity is systematically manipulated and assessing how client outcomes are affected under these varying treatment integrity conditions. Experimental studies allow for functional conclusions with respect to how varying levels of treatment integrity affect client behavior, whereas correlational studies do not.

Durlak and Dupre (2008) published a review article in the American Journal of Community Psychology investigating the relationship between treatment integrity and client/program outcomes. The review included more than 500 articles covering areas such as mental health, alcohol and substance abuse, social skills, and physical health promotion. The review found that procedures implemented with high levels of integrity are more effective compared to procedures that are implemented with lower levels of integrity. For interested readers, similar reviews involving drug prevention studies (Tobler, 1986), anti-bullying programs (Smith, Schneider, Smith, & Ananiadou, 2004), and mentoring (DuBois, Holloway, Valentine, & Cooper, 2002) have also been conducted.

Research investigating the relationship between treatment integrity and client outcomes have also involved people with I/DD. I/DD is characterized by deficits in adaptive (e.g., social and interpersonal skills), and intellectual functioning (e.g., reason, judgement) and is often diagnosed before age 18 (American Psychiatric Association, 2013). Unfortunately, the I/DD literature does not contain a review article similar to that of Durlak and Dupre (2008) that synthesizes the results from experimental studies in which levels of treatment integrity were systematically manipulated to assess the effects on client behavior. The I/DD literature would greatly benefit from such a review. The following paragraph contains a brief summary of some of the conclusions reached by authors in the field of I/DD when conducting experimental treatment integrity research.

A consistent finding in the I/DD literature is that higher levels of treatment integrity are associated with best client outcomes (e.g., DiGennaro Reed, Reed, Baez, & Maguire, 2011). Specifically, client outcomes are best when procedures are implemented with perfect treatment integrity. Although implementing procedures with perfect integrity every time a procedure is carried out in the natural environment may not be feasible, interventionists should strive to implement procedures with high levels of integrity as often as possible. An interesting finding is that when treatment integrity decreases, client behavior becomes less predictable (e.g., Groskreutz, Groskreutz, & Higbee, 2011). That is, for some clients, decreasing levels of treatment integrity do not adversely affect their behavior as much as others. Another finding is that skill acquisition is possible when treatment integrity is compromised (e.g., Jenkins, Hirst, & DiGennaro Reed, 2015). However, skill acquisition despite compromised levels of treatment integrity applies only to a minority of clients, and skill mastery is often delayed when treatment integrity is low. Next, initial exposure to procedures consisting of treatment integrity errors appear to have idiosyncratic long-term effects on client behavior (e.g., Leon, Wilder, Majdalany, Myers, & Saini, 2014). Research of this kind involves exposing participants to initial low-integrity conditions before getting exposure to high-integrity conditions. For some clients, skill acquisition is delayed but not for others. The exact source of this variability across clients has not yet been determined. Finally, research conducted by St. Peter Pipkin et al. (2010) found that treatment integrity errors are less detrimental with respect to client behavior if procedures are initially implemented with perfect integrity.

Enhancing Treatment Integrity

The treatment integrity literature contains several examples of empirically validated evidence-based methods of training interventionists to administer procedures with high levels of integrity. I will briefly describe three training procedures used to enhance treatment integrity: behavioral skills training (Miltenberger, 2004), performance feedback (e.g., Leblanc, Ricciardi, & Luiselli, 2005), and video modeling (e.g., Catania, Almeida, Liu-Constant, & DiGennaro Reed, 2009; Weldy, Rapp, & Capocasa, 2014). These training procedures typically result in skills maintenance (i.e., high levels of treatment integrity are maintained for extended periods following the completion of the training), and can be relatively time and cost effective to implement.

Behavioral skills training (BST) consists of providing trainees with detailed instructions on how to correctly perform target skills, modeling or demonstrating to trainees how to correctly administer target skills, providing trainees the opportunity to practice target skills, and providing feedback regarding their performance once they practiced performing target skills. For example, Sarokoff and Sturmey (2004) used BST to train special education teachers to implement discrete-trial training (DTT) procedures with high levels of integrity. Initially, all teachers implemented more than half of all DTT procedures incorrectly. Following the implementation of BST, all teachers implemented DTT with close to perfect integrity. Other research examples of when BST was used to enhance treatment integrity includes Rosales, Stone, and Rehfeldt (2009), and Miles and Wilder (2009).

Performance feedback consists of providing trainees with direct feedback about the integrity with which procedures are implemented. Performance feedback can also consist of goal setting and graphing trainee progress with respect to progress on their goals (Hagermoser Sanetti & Kratochwill, 2008). In the Leblanc, Ricciardi, and Luiselli (2005) study, performance feedback consisted of providing interventionists with positive feedback for skills performed correctly, corrective feedback for skills performed incorrectly, and answering any interventionist questions regarding the procedure. Results showed that treatment integrity increased following training and that performance gains were maintained up to 11 weeks following training. For further discussion on performance feedback, readers are referred to an excellent article by Hagermoser et al. (2008).

Video modeling involves trainees watching a video that demonstrates how to administer procedures correctly within a given context (Catania, Almeida, Liu-Constant, & DiGennaro Reed, 2009). Research suggests that video modeling is most effective when a wide variety of potential interventionist behaviors are used as exemplars within the video (e.g., Moore & Fisher, 2009). Other research has shown video modeling to be even more effective when combined with performance feedback (e.g., DiGennaro Reed, Codding, Catania, & Maguire, 2010). Some examples from the literature of when video modeling was used to increase treatment integrity includes Catania et al. (2009; training interventionists to implement DTT procedures), and Neef, Trachtenberg, Loeb, & Sterner (1991; training respite-care workers).

In conclusion, I am greatly encouraged by the increased number of published research articles in recent years with respect to treatment integrity. Treatment integrity is an interesting and exciting area of research. This article discussed reasons why treatment integrity is important, regardless of discipline, from both research and clinical perspectives. I hope this article encourages researchers and interventionists to measure and report treatment integrity data with greater frequency given the reasons outlined here.

References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Andrews, D. A., & Dowden, C. (2005). Managing correctional treatment for reduced recidivism: A meta-analytic review of programme integrity. Legal and Criminological Psychology, 10, 173-187.

Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91-97.

Behavior Analyst Certification Board. (2014). Professional and ethical compliance code for behavior analysts.

Bruhn, A. L., Hirsch, S. E., & Lloyd, J. W. (2015). Treatment integrity in school-wide programs: A review of the literature (1993-2012). The Journal of Primary Prevention, 36, 335-349.

Catania, C. N., Almeida, D., Liu-Constant, B., & DiGennaro Reed, F. D. (2009). Video modeling to train staff to implement discrete-trial instruction. Journal of Applied Behavior Analysis, 42, 387-392.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimental design and analysis issues for field settings. Boston, MA: Houghton Mifflin.

DiGennaro Reed, F. D., Codding, R., Catania, C. N., & Maguire, H. (2010). Effects of video modeling on treatment integrity of behavioral interventions. Journal of Applied Behavior Analysis, 43, 291-295.

DiGennaro Reed, F. D., Reed, D. D., Baez, C. N., & Maguire, H. (2011). A parametric analysis of errors of commission during discrete-trial training. Journal of Applied Behavior Analysis, 44, 611-615.

DiGennaro Reed, F. D., & Codding, R. S. (2014). Advancements in procedural fidelity assessment and intervention: Introduction to the special issue. Journal of Behavioral Education, 23, 1-18.

Downs, A., Downs, R. C., & Rau, K. (2008). Effects of training and feedback on discrete trial teaching skills and student performance. Research in Developmental Disabilities, 29, 235-246.

DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157-198.

Dupak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350.

Gresham, F. M., Gansle, K, A., & Noel, G. H. (1993). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26, 257-263.

Groskreutz, N. C., Groskreutz, M. P., & Higbee, T. S. (2011). Effects of varied levels of treatment integrity on appropriate toy manipulation in children with autism. Research in Autism Spectrum Disorders¸ 5, 1358-1369.

Hagermoser Sanetti, L. M., & Kratochwill, T. R. (2008). Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and Therapy, 4, 95-114.

Hagermoser Sanetti, L. M., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445-459.

Hinckley, J. J., & Douglas, N. F. (2013). Treatment fidelity: Its importance and reported frequency in aphasia treatment studies. American Journal of Speech-Language Pathology, 22, 279-284.

Jenkins, S. R., Hirst, J. M., & DiGennaro Reed, F. D. (2015). The effects of discrete-trial training commission errors on learner outcomes: An extension. Journal of Behavioral Education, 24, 196-209.

Leblanc, M., Ricciardi, J. N., & Luiselli, J. K. (2005). Improving discrete trial instruction by paraprofessional staff through an abbreviated performance feedback intervention. Education and Treatment of Children, 28, 76-82.

Leon, Y., Wilder, D. A., Majdalany, L., Myers, K., & Saini, V. (2014). Errors of omission and commission during alternative reinforcement of compliance: The effects of varying levels of treatment integrity. Journal of Behavioral Education, 23, 19-33.

Miles, N. I., & Wilder, D. A. (2009). The effects of behavioral skills training on caregiver implementation of guided compliance. Journal of Applied Behavior Analysis, 42, 405-410.

Miltenberger, R. (2004). Behavior modification: Principles and procedures (3rd ed.). Belmont, CA: Wadsworth/Thomson Learning.

Monchar, F. J., & Prinz, F. J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247-266.

Moore, J. W., & Fisher, W. W. (2009). The effects of videotape modeling on staff acquisition of functional analysis methodology. Journal of Applied Behavioral Analysis, 40, 197-202.

Neef, N. A., Trachtenberg, S., Loeb, J., & Sterner, K. (1991). Video-based training of respite care workers: An interactional analysis of presentation format. Journal of Applied Behavior Analysis, 24, 473–486.

Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: Analysis of studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 6, 829-841.

Peterson, L., Homer, A. L., & Wonderlich, S. A. (1982). The integrity of independent variables in behavior analysis. Journal of Applied Behavior Analysis, 15, 477-492.

Rosales, R., Stone, K., & Rehfeldt, R. A. (2009). The effects of behavioral skills training on implementation of the picture exchange communication system. Journal of Applied Behavior Analysis, 42, 541-549.

Sarokoff, R., & Sturmey, P. (2004). The effects of behavioral skills training on staff implementation of discrete-trial teaching. Journal of Applied Behavior Analysis, 37, 535–538.

Smith, J. D., Schneider, B. H., Smith, P. K., & Ananiadou, K. (2004). The effectiveness of whole-school antibullying programs: A synthesis of evaluation research. School Psychology Review, 33, 547-560.

St. Peter Pipkin, C., Vollmer, T. R., & Sloman, K. N. (2010). Effects of treatment integrity failures during differential reinforcement of alternative behavior: A translational model. Journal of Applied Behavior Analysis, 43, 47-70.

Tobler, N. S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537-567.

Van Houten, R., Axelrod, S., Bailey, J. S., Favell, J. E., Foxx, R. M., Iwata, B. A., & Lovaas, I. (1988). The right to effective behavioral treatment. Journal of Applied Behavior Analysis, 21, 381-384.

Weldy, C. R., Rapp, J. T., & Capocasa, K. (2014). Training staff to implement brief stimulus preference assessments. Journal of Applied Behavior Analysis, 47, 214-218.

Wheeler, J. J., Baggett, B. A., Fox, J., & Blevins, L. (2006). Treatment integrity: A review of intervention studies conducted with children with autism. Focus on Autism and Other Developmental Disabilities, 21, 45-54.

Citation for this article:

Brand, D. (2014). Treatment Integrity: Why it is important regardless of discipline. Science in Autism Treatment, 14(2), 6-7, 9-11.

 

#BehaviorAnalysts #Faculty #OTs #SLPs #Researchers #Teachers