- IRB: ETHICS & HUMAN RESEARCH
Assessing the Readability of Non-English-Language Consent Forms: The Case of Kiswahili for Research Conducted in Kenya
One of the fundamental ethical requirements for research with humans is that individuals provide valid informed consent to participate in research studies.1A key issue regarding the validity of informed consent is whether individuals understand the information presented in the consent form. How easy or challenging it is for individuals to understand a consent form depends on their knowledge and ability to comprehend and on the text itself. Factors affecting the reader’s ability to comprehend include her socioeconomic status and how many years of education she has. For instance, several studies assessing comprehension of consent forms have shown that research participants with more years of education had significantly higher comprehension levels than those with fewer years of education.2In another study, participants who had difficulty reading and those who were illiterate benefited more from a simplified consent form than their more educated counterparts did.3Text factors related to readability include a form’s overall length, legibility of print, illustration, color, vocabulary, conceptual difficulty, syntax, and organization.4
The Kenya Medical Research Institute (KEMRI) is the national body responsible for carrying out health research in Kenya. The KEMRI national ethical review committee (KEMRI/NERC) has a general requirement that consent forms be of a readability level suitable for the research participants, although what this means is not specified. To assess acceptable readability, the KEMRI/NERC examines English-language consent forms; however, it never determines whether the forms translated into Kiswahili—which, with English, is one of the two official languages of Kenya—are comparable in readability to the English-language forms. Rather, the KEMRI/NERC assumes that a lower level of readability for the English-language consent form results in an equal level of readability for the Kiswahili-language form. Bearing in mind that the literacy rate in Kenya is 61%, it is important that the translated forms are at the lowest readability level possible.
To date, there has not been a single empirical study to document the readability of Kiswahili forms or to determine whether readability formulas previously developed for English text applies to Kiswahili text. Indeed, we are not aware of any studies that compare the readability of English-language forms to their translated forms in any language. To address this lacuna in knowledge, our study was designed to measure and compare the text difficulty of English-language consent forms to the version of those forms translated into Kiswahili. Knowing whether translated forms resemble the original English-language form in readability is important given that ethics review committees around the world generally make the assumption that a simplified English-language form will result in a simplified local language form. Without empirical data to determine whether this assumption is accurate, however, research ethics review committees may need other mechanisms to determine the readability of local language consent forms.
Readability, or reading ease, is a function of texts, while comprehension is a function of readers. Nonetheless, the concepts are inextricably linked. Readability formulas and methods, therefore, should not merely predict whether the reader can read the words, but whether the sentences have meaning to the reader. There currently are over 40 formulas that can be used to determine readability levels of English prose text. Although a smaller number of readability formulas exist for languages other than English, there are no readability formulas for Kiswahili.5
One method designed to measure readability of documents written in any language is called the cloze procedure. Briefly, it requires that the tester delete words from the passage of interest at regular intervals—for example, every fifth word. The tester then asks the reader to make his or her best guess to fill in the missing words from the given passage of text. The number of correctly guessed words serves as an index of how well the reader understands the meaning of the passage. Thus, the cloze procedure examines the extent to which the reader can predict the intended words based on the surrounding context. This ability to predict and make sense of textual meaning and context is consistent with how the reading process ordinarily occurs. Normal reading is not a letter-by-letter decoding, but rather a mixture of perception and guesswork.6The cloze procedure attempts to mimic this guesswork in a controlled manner.
In a cloze test, two characteristics significantly determine how well the reader is able to decipher a passage of text with missing words. The first is the reader’s own “redundancy utilization.” This is the reader’s ability to use his or her preexisting knowledge of vocabulary, linguistic conventions, and general knowledge to make sense of a given passage of text. The second is the “information load” of the passage itself, meaning the volume and complexity of the information provided. Thus, the second factor is a feature of the message and not of the reader.
Unlike other readability formulas, the method by which the cloze procedure predicts the difficulty of text is not based on the length of words or sentences. Rather, it measures the reader’s ability to make sense of the text directly, thereby acknowledging the relationship between individual background knowledge and text complexity in a way that word-based, predictive formulas cannot.7
Consent Forms.We used 10 pairs of consent forms (each pair containing one English form and its Kiswahili translation), or 20 forms in total, for this study. We drew English-language consent forms and their Kiswahili-translated forms from 10 clinical studies submitted to and approved by the KEMRI/NERC from 2004 to 2006. We selected studies randomly and required that they have KEMRI/NERC-approved consent forms written in English and Kiswahili.
We removed all information from the consent forms that might have identified the investigators or the institutions where the studies were conducted. We then divided each consent form into three parts: the beginning, the middle, and the end. Finally, starting from the beginning of each of the three parts, we deleted every fifth word until we had deleted 50 words in total in each part to facilitate the cloze procedure.
Study Participants.Thirty census fieldworkers from the Demographic Surveillance Survey study at the Centre of Geographic Medicine Coast in Kilifi consented to participate in this study. They were selected because they were available to do the procedure at the designated time and had completed 12 years of schooling. They agreed to conduct the cloze procedure over a total of two hours a day for a total of 15 days. The field workers completed consent forms anonymously.
Data Collection.Each subject received an English-language and Kiswahili-language consent form from every study that we looked at. However, we did not provide the same study’s consent forms in both languages consecutively, nor did we provide them in any specific order; thus, the participant might receive the Kiswahili form before the English one, and vice versa. We deleted 50 words from each of the three parts of every consent form, replacing the deleted words with a standard-length blank space.
All of the participants reviewed all of the consent forms. We instructed participants to attempt to fill in each blank with one word only, and we told them that spelling errors would not be marked wrong. We also told them that they were not being tested; rather, we told them it was the difficulty of the passage that the study was measuring. We measured participants’ comprehension scores by how many missing words each participant correctly inserted across all three passages of a form. We scored consent forms and entered data into Stata, a data analysis and statistical software program.
Data Analysis.Using the cloze procedure, we calculated an average score for each of the three sections of the 20 consent forms (10 in English, and 10 in Kiswahili). This average score would control for variations within passages and within the individual’s entire response. Based on these data, we calculated a final average score for each of the 20 forms.
We compared the average score for each English-language form with its Kiswahili translation using a paired t-test testing a null hypothesis of no difference (meaning that each individual answered English and Kiswahili at the same rate of accurate completion) versus an alternative hypothesis that the mean difference is statistically not equal to zero (meaning that each individual answered English and Kiswahili at varying rates of accurate completion). We estimated mean differences along with their 95% confidence intervals and corresponding p-values. A p-value of less than 0.05 was considered to be statistically significant. We then subtracted the Kiswahili score from the English score. We repeated this process for all passages from the 10 pairs of consent forms.
Test of Symmetry.A cloze pass mark is described in the literature as a score above 38%. This pass mark is based on a study by Bormuth8in which he compared information gain scores and cloze test results. Information gain tests measure how much information a reader gains from a passage by testing his or her comprehension of a passage’s subject matter before he or she sees the passage, then giving the same test to the reader after he or she sees the passage and subtracting the first score from the second. Bormuth concluded from the study that at approximately the level of 38% correct cloze items there was a noticeable increase in information gain scores.
For the purpose of our study, consent forms were categorized in the following way: forms that scored less than 38% were determined to be incomprehensible (i.e., the sentence did not make sense because it did not have adequate information load and therefore failed to communicate); those that scored 38% and above were determined to be comprehensible; and those that scored 66% and above were determined to be very easy to comprehend.
Mean Difference.Seven out of 10 pairs of consent forms had a statistically significant mean difference in favor of English, suggesting that they were more comprehensible in English than Kiswahili. Only one of the 10 forms had a statistically significant mean difference in favor of Kiswahili, meaning that this one Kiswahili form was more comprehensible than its English counterpart. Consent forms D and I were nearly equally comprehensible in both English and Kiswahili. These results are summarized inTable 1andFigure 1.
Test of Symmetry Categorization.Table 2shows the actual number and percentage by score of each form for English and Kiswahili, andFigure 2shows the percentage of participants that scored below 38% in the cloze procedure (therefore “failing” readability according to standard cloze procedure scoring).
For six of the 10 forms, a greater percentage of respondents “failed” the Kiswahili form than the English form. Consent forms B and J, for example, demonstrate notable differences between the percentages of respondents who failed the Kiswahili form compared to the English form: 35% versus 10% and 40% versus 10%, respectively (seeTable 2). In forms A, H, and I, all 30 field workers received passing scores between 38% and 66%, or the “comprehensible” category, in both languages, suggesting that these forms were equally readable in either language. None of the respondents scored higher than 66% correct comprehension on any form, regardless of whether it was written in English or Kiswahili.
The wide variation in the degree to which the consent forms written in Kiswahili corresponded to the English language forms in readability raises questions about how and by whom consent forms are translated. Informal discussions with the investigators who allowed their consent forms to be used for this study indicated that a wide range of persons—from university lecturers to secondary school teachers—were asked to translate English-language consent forms into the local language. In one case, the fieldworkers translated a consent form as a team. Of note, only one of the 10 consent forms used in this study was back-translated.
Another question raised by this study is the degree to which local languages that are traditionally spoken rather than written (which is true for Kiswahili) lend themselves to written format, as the formality of consent forms requires. It may be that if the consent forms written in Kiswahili had been read to the educated fieldworkers who participated in our study, scores and comprehension would have been higher, even if the researchers recited the language on the existing forms. Indeed, in many countries, including Kenya, formal education is conducted in English, and local languages are generally oral rather than written. Conducting consent discussions in local languages can be critical; but where participants are literate, it may be appropriate to provide them with written forms in English, as their reading skills may be more advanced in English even if their oral fluency is greater in the local language.
This study had two significant limitations. First, while the cloze procedure has been validated in dozens of languages, it is not clear that it is valid for Kiswahili, despite the fact that it is designed to be usable for all languages. Second, we selected the 10 consent forms as a convenience sample and only in one setting. Since this is the first study of its kind, it is difficult to determine if such a sample is adequate to draw generalizable conclusions.
Further research is needed to explore the reasons for the low readability of the consent forms written in Kiswahili before recommendations can be made to improve or standardize the translation process. It is also important to ensure that local language consent forms are comprehensible to readers. Finally, ethics review committees should pause before asking only for simplified English language forms without scrutinizing the language or translation approach that will be used to translate those forms into the local language.
Caroline Kithinji, Msc,is ERC Manager/Administrator at Kenya Medical Research Institute, Nairobi, Kenya; andNancy E. Kass, ScD,is Phoebe R. Berman Professor of Bioethics and Public Health, Bloomberg School of Public Health and Berman Institute of Bioethics, Johns Hopkins University, Baltimore, MD, USA.
1. Emanuel EJ, Wendler D, Grady C. What makes clinical research ethical?JAMA2000;283(20):2701-2711.
2. Freeman T, O’Brien-Pallas LL. Factors influencing job satisfaction on specialty nursing units.Canadian Journal of Nursing Administration1998;11(3):25-51; Muss HB, White DR, Michielutte R, et al. Written informed consent in patients with breast cancer.Cancer1979; 43(4):1549-1556; Taub HA: Informed consent, memory and age.Gerontologist1980;20(6):686-690; Young DR, Hooker DT, Freeberg FE. Informed consent documents: Increasing comprehension by reducing reading level.IRB: A Review of Human Subjects Research1990;12(3):1-5. Doak CC, Doak LG, Root JH.Teaching Patients with Low Literacy Skills. 2nd ed. Philadelphia, PA: JB Lippincott, 1996.
3. Dawson L, Kass N. Views of U.S. researchers about informed consent in international collaborative research.Social Science and Medicine2005;61:1211-1222.
4. Harrison C, ed.Readability in the Classroom. New York: Cambridge University Press, 1980.
5. See ref. 3, Dawson and Kass 2005.
6. Bormuth JR. Empirical determination of the instructional reading level. In: Figurel JA, ed.Reading and Realism. Newark, DE: International Reading Association, 1969.
7. See ref. 4, Harrison 1980.
8. See ref. 6, Bormuth 1969.
Caroline Kithinji and Nancy E. Kass, “Assessing the Readability of Non-English-Language Consent Forms: The Case of Kiswahili for Research Conducted in Kenya,”IRB: Ethics & Human Research32, no. 4: 10-15.