IRB: Ethics & Human Research

Knowledge of Regulations Governing Pediatric Research: A Pilot Study

A growing awareness of the paucity of evidence for most treatments in pediatric medicine¹ has led to a rapid increase in the number of clinical trials involving children.² This means that some Institutional Review Boards (IRBs) are increasingly called upon to review pediatric research protocols. Because children’s cognitive, psychological, and social immaturity limits their ability to understand and make decisions about participation in clinical research, federal regulations governing research with children require IRBs to implement additional protections beyond those afforded to adults who are recruited to participate in clinical research. Moreover, because children generally lack legal standing to authorize their own participation in research through the mechanism of informed consent, IRBs must ensure that regulatory requirements for proxy permission and child assent—as well as for upper limits on the acceptable degree of research-related risk—are met.³ Application of these extra protections to specific studies is complicated by the heterogeneity of children who might participate in pediatric research.⁴ Given these factors, review and oversight of pediatric research is a specialized area of IRB practice.⁵

Several studies of IRB practices reveal inconsistency among IRBs when applying regulations governing clinical research, including specific provisions governing research with children.⁶ Although this inconsistency is likely due in part to legitimate differences in interpreting regulatory requirements and ethical standards,⁷ it might also stem from some IRB members’ lack of necessary expertise regarding pediatric research ethics and regulations.⁸ In its 2004 report,Ethical Conduct of Clinical Research Involving Children, the Institute of Medicine (IOM) highlighted the need to educate IRB members about regulatory requirements and ethical standards for conducting research with children.⁹

In this pilot study, we assessed knowledge of pediatric research regulations among IRB members who review protocols involving children. We also evaluated members’ training and experience regarding the practice and oversight of pediatric research. To our knowledge, this study represents the first attempt to evaluate IRB members’ knowledge of these important topics. 

Study Methods

Using the member list of the National Association of Children’s Hospitals and Related Institutions (NACHRI),¹⁰ we identified institutions within the United States likely to conduct research involving children. We focused on centers providing comprehensive pediatric care. Shriner’s hospitals, rehabilitation centers, and disease-specific specialty hospitals were excluded from our sample. We also excluded institutions located outside the United States, as these operate under different regulatory requirements.
One hundred fifty-six institutions met criteria for inclusion in this study. We randomly selected 12 institutions from this list for our pilot study group. We contacted IRB officials at these centers and asked them to provide contact information for all active members of their institution’s IRB. Five centers reporting a total of 72 members (range 8–20 per IRB) agreed to participate.

For four of the institutions, we mailed the study packet directly to all active IRB members. At the request of the fifth institution, we sent all study packets to the IRB administrator, who distributed them to members. The study packet contained a cover letter, an anonymous questionnaire, a separate coded postcard to notify us that the respondent had returned the questionnaire or preferred not to participate, and a return envelope. It was not possible to link questionnaires to individual respondents or to their institutions. Upon receipt of the postcard indicating return of the questionnaire, respondents were entered into a drawing for a gift certificate to a local restaurant. Four weeks after the initial mailing, we mailed a second questionnaire to IRB members from whom we had not received a response postcard. Seventy-two individuals were contacted. One survey recipient who reported not being an IRB member was excluded. After two mailings, 58% (41 out of 71) of eligible IRB members returned their questionnaires.Our pilot study sought to assess two fundamental constructs: 1) self-reported preparedness for the tasks involved in IRB review, both overall and specifically with respect to pediatric protocols, and 2) knowledge of pediatric research regulations. The 12-question preparedness section asked respondents “How prepared do you consider yourself to be when performing each of the following IRB member tasks?” Responses were on a five-point ordinal scale, ranging from “not prepared” to “very well prepared,” and included a “not applicable” response option.

The five-question knowledge section of the survey assessed understanding of federal regulations governing pediatric clinical research (e.g., those related to risk categorization, parental permission, and participant assent). We designed questions based on the regulatory requirements governing research with children (45 CFR 46, Subpart D),¹¹ the IOM report,¹² and Kornetsky et al.’sStudy Guide for Institutional Review Board Management and Function .¹³ Questions were multiple-choice format, with one best answer, and included a “not sure” response option. Instructions accompanying the knowledge items asked respondents to answer the questions without the use of reference materials.

Additional items asked about respondents’ professional backgrounds, their previous experience with conduct and review of clinical research, the structure and workload of their IRBs, and the general and pediatric-specific education regarding IRB review that they had received.

Prior to distribution, a three-member panel of experts on pediatric research oversight reviewed a draft questionnaire. We also pilot-tested the draft questionnaire with five individuals of varying professional backgrounds who are IRB members at the Dana-Farber Cancer Institute (DFCI). We revised or omitted questions that the panel review and pilot test showed to be confusing or ambiguous.

We calculated a knowledge score for each respondent by assigning one point for each best answer selected.A priori sample size calculations indicated that 40 respondents would allow estimation of the mean knowledge score within B1 0.32 standard deviations with 95% confidence.

We calculated an overall self-assessment score by averaging responses to the 12 questions regarding preparedness for elements of protocol review, and then we normalized the average to a 0–100 point scale. Similarly, we calculated a “pediatric-specific” self-assessment score by averaging the responses to the subset of seven questions regarding preparedness for elements of pediatric protocol review, then we normalized the average to a 0–100 point scale.In bivariate analyses, we tested potential correlates of dependent variables (i.e., knowledge score, overall self-assessment score, and pediatric-specific self-assessment score) using Wilcoxon rank-sum tests, Kruskal-Wallis tests, or Spearman rank correlation coefficients as appropriate to the data. We considered associations to be potentially significant if two-sided p was less than 0.05. Because we viewed the analyses as exploratory, we did not adjust for multiple comparisons. Statistical analyses were performed using Stata 8 for Windows (Stata Corp., College Station, Texas).

The study was approved by the DFCI IRB, which waived the requirement for documentation of informed consent.

IRB Knowledge

Respondent Characteristics

Table 1shows that most respondents were affiliated with the medical center their IRB served. Approximately half reported professional experience related to child health or development. Almost all had at least one year of experience as an IRB member; almost half had at least five years experience. Six respondents (15%) were currently serving as IRB chairs. Respondents reported exposure to a variety of training methods, with computer- and lecture-based methods being most common. Most (78%) reported receiving ongoing training, and most (68%) had received specific training in review of pediatric research.

Preparedness for Protocol Review

Levels of self-reported preparedness to perform tasks involved in reviewing research protocols were high (Figure 1 ). The median self-assessment score was 79 (interquartile range [IQR] 71–85); the median pediatric-specific self-assessment score was 75 (IQR 64–93). Internal consistency reliabilities of the overall self-assessment and pediatric-specific self-assessment scales were high (Cronbach’s a = 0.93 and 0.92, respectively). Respondents felt least prepared to evaluate the need for a child’s assent (66% responding “well prepared” or “very well prepared”) and to determine whether the research involved a minor increase over minimal risk (68% responding “well prepared” or “very well prepared”).
Several facets of experience and training correlated with self-reported preparedness for IRB tasks (Table 2 ). Overall, IRB chairs considered themselves better prepared than did other members, although this difference was not apparent when asking specifically about review of pediatric protocols. Respondents who received ongoing training in IRB review and those who had received specific training in the review of pediatric protocols considered themselves better prepared than did other respondents. Respondents who attended at least 12 IRB meetings annually considered themselves better prepared than did those who attended fewer meetings.

Knowledge of Pediatric Research Regulations

The median knowledge score was one (IQR 0.5–2, range 0–4). Ten respondents (25%) did not select the best answer for any question. For individual questions, 8–40% of respondents selected the best answer (Table 3; an appendix presenting comprehensive results is available here ).

Respondent characteristics associated with increased knowledge are shown in Table 4. IRB chairs were more knowledgeable than other members (p = 0 .04), respondents who had received lecture-based training were more knowledgeable than those who had not received such training (p = 0.02), and respondents who had received pediatric-specific training were more knowledgeable than those who had not (p = 0.04). Other characteristics, including affiliated/unaffiliated status, prior professional experience in child health, prior experience as a clinical investigator, years of IRB service, number of protocols reviewed, number of IRB meetings attended per year, percent of protocols reviewed that enrolled primarily children, and receipt of ongoing education regarding IRB review were not statistically associated with knowledge of regulations governing pediatric research review.

There was no significant correlation between knowledge score and either overall self-assessment score (Spearman’s r = 0.26, p = 0.11) or pediatric-specific self-assessment score (r = 0.14, p = 0.39).

Discussion

IRB review of research involving children requires careful attention to regulations governing pediatric research. According to the IOM, however, “no systematic documentation exists on the extent to which IRB members understand and fulfill their responsibilities in reviewing studies that include children.”¹⁴ We therefore evaluated self-reported preparedness for clinical research review, both in general and with respect to pediatric protocols, among IRB members at institutions that self-identify as children’s hospitals or related institutions. We also assessed members’ knowledge of federal regulations governing research with children. Despite high levels of self-reported preparedness, our data suggest limited knowledge of regulatory requirements specific to pediatric research.
Problems with knowledge of regulations governing pediatric research may partly explain the consistent heterogeneity in IRB review of research involving children.¹⁵ In a survey of IRB chairs, Shah et al. observed marked variability in assessments of risk level of various procedures (i.e., minimal risk, minor increase over minimal risk)—a determination that plays a critical role in IRBs’ decisions regarding the approvability of pediatric research protocols.¹⁶ In a different analysis based on the same survey, Whittle et al.¹⁷ noted that IRBs differ markedly in the standards used to determine whether child assent is required and in their attitudes towards payment to research participants or their parents. Kimberly et al.¹⁸ recently reported substantial variation among IRBs at hospitals participating in three multicenter trials regarding the requirement for and documentation of assent as well as the presence, amount, and form of payment for participation. Smith Rogers et al.¹⁹ found similar variability in risk assessment and assent requirements among 11 IRBs reviewing an identical study involving adolescents. Finally, Mammel et al.²⁰ found variability among IRB chairs in their willingness to waive the requirement for parental permission for both observational and intervention research involving adolescents. Other studies in the adult context show striking heterogeneity in IRBs’ handling of common multicenter protocols.²¹ As the IOM notes, this variability in IRB decisions may have legitimate causes—including limited data on the risks of research procedures, lack of precision in the regulations, and reasonable disagreement among reviewers about interpretation of regulations—as well as less justifiable causes, such as inadequate reviewer education.²² Whether this heterogeneity fosters inadequate protection or impedes ethical scientific inquiry is unknown.

We emphasize the pilot nature of this study and note that it has several limitations. First, though the development of the survey involved expert review and cognitive pilot testing, the potential for ambiguity in questions or response options remains. Furthermore, as noted previously, the regulations are themselves often imprecise and subject to interpretation. “Incorrect” responses to some survey questions could therefore indicate defensible alternative interpretations of the regulations, rather than lack of knowledge. For example, the regulations leave room for judgment about what role IRBs should play in determining the capability of individual children to provide assent. IRB members’ views on this issue may have influenced their responses to the question about when children are considered capable of assent. Second, because respondents were asked to complete the knowledge questions without the use of reference materials, our results may underestimate their performance during actual research reviews when reference materials are presumably available. Third, the study’s small size, limited number of participating institutions, and exploratory nature indicate that the findings should be viewed as preliminary and in need of confirmation. Finally, the imperfect individual and institutional response rates raise the possibility that the generalizability of the results may be affected by response bias at both levels. Though selected at random from the NACHRI list, the IRBs that agreed to participate in this survey may not be representative of all IRBs at NACHRI institutions. Participating IRBs served children’s hospitals in the eastern (one), midwestern (three), and western (one) United States. Nonparticipating institutions were also geographically diverse, representing the eastern (three), midwestern (one), and western (three) United States. The participating IRBs tended to have fewer members (mean 14, standard deviation [SD 5] versus mean 28 [SD 15]) and to serve smaller pediatric institutions and services (mean 130 [SD 62] versus 177 [SD 82] pediatric beds) than the seven nonparticipating IRBs. Two participating IRBs were at freestanding children’s hospitals, while three were at children’s hospitals that are part of larger hospitals, or “hospitals within hospitals.” Among nonparticipating IRBs, three served freestanding children’s hospitals, while four served hospitals within hospitals.

Given the small size of the survey and the multiple tests of association performed, our results could also be affected by false negative or false positive tests of association between IRB members’ knowledge and their individual characteristics. In addition, the anonymous design of the survey, used to maximize individual and IRB willingness to participate, precludes our ability to investigate differences between IRBs or to control for clustering within IRBs. For all these reasons, a larger, more comprehensive survey of IRB members’ knowledge that preserves links to institutions is urgently needed.

Finally, at a conceptual level, our study presupposes that all IRB members who review studies involving children should be knowledgeable about regulations governing pediatric research. This view receives support from the IOM, which concluded that “To be effective, IRB members should understand the ethics and history of research with humans, the current structure and funding of research projects, and the regulatory structure of research, including local laws.”²³

An alternative model might view knowledge of regulations as the province of a specialized subset of IRB members, or even just of the chair along with committee staff. If such a model were adopted, the roles of IRB members who are not responsible for knowledge of regulations should be clarified. Nonetheless, our study raises the concern that there may be inadequate understanding of pediatric research regulations among IRB members at institutions that conduct research with children. Additional studies to corroborate these findings and to identify modifiable factors at the individual or institutional level that are associated with IRB members’ knowledge are needed. If confirmed, our results suggest a pressing need for education of IRB members in the specialized task of reviewing pediatric research protocols.

Acknowledgments

We thank Jeffrey Botkin, E. Francis Cook, Celia Fisher, Alan Fleischman, and Tina Gelsomino for their invaluable advice and assistance with study and survey development and execution.

This work was supported by funding from the Alexandra J. Miliotis Pediatric Oncology Research Fund and the Harvard-MIT Division of Health Sciences and Technology. Dr. Joffe received support from the National Cancer Institute (K01 CA096872).

Disclosure

Susan Kornetsky serves as a site visitor for the Association for the Accreditation of Human Research Protection Programs (AAHRPP), for which she is paid on a per diem basis. She also receives financial compensation to teach seminars for IRB members (IRB 101) and administrators (IRB Administrator 101) on behalf of Public Responsibility in Medicine and Research (PRIM&R).
Annemarie Stroustrup, MD, is Fellow, Division of Newborn Medicine, Kravis Children’s Hospital, Mount Sinai Medical Center, New York, NY;Susan Kornetsky, MPH, CIP, is Director, Clinical Research Compliance, Children’s Hospital Boston, Boston, MA; andSteven Joffe, MD, MPH, is Assistant Professor of Pediatrics, Children’s Hospital Boston, and Dana-Farber Cancer Institute, Boston, MA.

References

1. Steinbrook R. Testing medications in children.NEJM2002;347:1462-1470.

2. The 108th United States Congress.Pediatric Research Equity Act of 2003. Washington D.C.: Government Printing Office, 2003; The 107th United States Congress.Best Pharmaceuticals for Children Act. Washington D.C.: Government Printing Office, 2002; Baylis F. Mandating research with children.IRB: A Review of Human Subjects Research1999;21(1):10-11.

3. Department of Health and Human Services. 45 CFR 46. Protection of Human Subjects; Jonsen AR. Research involving children: Recommendations of the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research.Pediatrics1978;62(2):131-136.

4. Santelli JS, Smith Rogers A, Rosenfeld WD, et al. Guidelines for adolescent health research. A position paper of the Society for Adolescent Medicine.The Journal of Adolescent Health2003;33(5):396-409.

5. National Institutes of Health.Policy and Guidelines on the Inclusion of Children as Participants in Research Involving Human Subjects.Bethesda, MD: Government Printing Office, 1998; National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research.The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. Washington D.C.: Government Printing Office, 1979.

6. Bennett CL, Sipler AM, Parada JP, et al. Variations in institutional review board decisions for HIV quality of care studies: A potential source of study bias.Journal of Acquired Immune Deficiency Syndromes2001;26(4):390-391; Stair TO, Reed CR, Radeos MS, et al. Variation in institutional review board responses to a standard protocol for a multicenter clinical trial.Academic Emergency Medicine2001;8(6):636-641; Hirshon JM, Krugman SD, Witting MD, et al. Variability in institutional review board assessment of minimal-risk research.Academic Emergency Medicine2002;9(12):1417-1420; Wendler D, Belsky L, Thompson KM, et al. Quantifying the federal minimal risk standard: Implications for pediatric research without a prospect of direct benefit.JAMA2005;294(7):826-832; Kimberly MB, Hoehn KS, Feudtner C, et al. Variation in standards of research compensation and child assent practices: A comparison of 69 institutional review board-approved informed permission and assent forms for three multicenter pediatric clinical trials.Pediatrics2006;117(5): 1706-1711; Shah S, Whittle A, Wilfond B, et al. How do institutional review boards apply the federal risk and benefit standards for pediatric research?JAMA2004;291:476-482; Whittle A, Shah S, Wilfond B, et al. Institutional review board practices regarding assent in pediatric research.Pediatrics2004;113(6):1747-1752.

7. See ref. 6, Wendler et al. 2005.

8. See ref. 6, Kimberly et al. 2006; Shah et al. 2004; Whittle et al. 2004.

9. Field M, Behrman R, eds.The Ethical Conduct of Clinical Research Involving Children. Washington, D.C.: National Academies Press, 2004.

10. National Association of Children’s Hospitals and Related Institutions. Member Institutions of NACHRI.https://www.childrenshospitals.net.

11. See ref. 3, Department of Health and Human Services 2001.

12. See ref. 9, Field et al. 2004.

13. Kornetsky S, Davis A, Amdur R.Study Guide for Institutional Review Board Management and Function. Sudbury, MA: Jones and Bartlett Publishers, 2003.

14. See ref. 9, Field et al. 2004, p. 255.

15. See ref. 6, Kimberly et al. 2006; see ref. 6, Shah et al. 2004; see ref. 6, Whittle et al. 2004.

16. See ref. 6, Shah et al. 2004.

17. See ref. 6, Whittle et al. 2004.

18. See ref. 6, Kimberly et al. 2006.

19. Smith Rogers A, Schwartz DF, Weissman G, et al. A case study in adolescent participation in clinical research: Eleven clinical sites, one common protocol, and eleven IRBs.IRB: A Review of Human Subjects Research1999:21(1):6-10.

20. Mammel KA, Kaplan DW. Research consent by adolescent minors and institutional review boards.The Journal of Adolescent Health1995;17(5):323-330.

21. See ref. 6, Bennett et al. 2001; see ref. 6, Stair et al. 2001; see ref. 6, Hirshon et al. 2002.

22. See ref. 9, Field et al. 2004.

23. Institute of Medicine.Responsible Research: A Systems Approach To Protecting Research Participants. Washington, DC: National Academy Press, 2002, p. 63.

Annemarie Stroustrup, Susan Kornetsky, and Steven Joffe, “Knowledge of Regulations Governing Pediatric Research: A Pilot Study,”IRB: Ethics and Human Research 30, no. 5 (2008): 1-7.