IRB: Ethics & Human Research

Assessing the Readability of Non-English-Language Consent Forms: The Case of Kiswahili for Research Conducted in Kenya

One of the fundamental ethical requirements for research with humans is that individuals provide valid informed consent to participate in research studies.¹A key issue regarding the validity of informed consent is whether individuals understand the information presented in the consent form. How easy or challenging it is for individuals to understand a consent form depends on their knowledge and ability to comprehend and on the text itself. Factors affecting the reader’s ability to comprehend include her socioeconomic status and how many years of education she has. For instance, several studies assessing comprehension of consent forms have shown that research participants with more years of education had significantly higher comprehension levels than those with fewer years of education.²In another study, participants who had difficulty reading and those who were illiterate benefited more from a simplified consent form than their more educated counterparts did.³Text factors related to readability include a form’s overall length, legibility of print, illustration, color, vocabulary, conceptual difficulty, syntax, and organization.⁴ 

The Kenya Medical Research Institute (KEMRI) is the national body responsible for carrying out health research in Kenya. The KEMRI national ethical review committee (KEMRI/NERC) has a general requirement that consent forms be of a readability level suitable for the research participants, although what this means is not specified. To assess acceptable readability, the KEMRI/NERC examines English-language consent forms; however, it never determines whether the forms translated into Kiswahili—which, with English, is one of the two official languages of Kenya—are comparable in readability to the English-language forms. Rather, the KEMRI/NERC assumes that a lower level of readability for the English-language consent form results in an equal level of readability for the Kiswahili-language form. Bearing in mind that the literacy rate in Kenya is 61%, it is important that the translated forms are at the lowest readability level possible.

To date, there has not been a single empirical study to document the readability of Kiswahili forms or to determine whether readability formulas previously developed for English text applies to Kiswahili text. Indeed, we are not aware of any studies that compare the readability of English-language forms to their translated forms in any language. To address this lacuna in knowledge, our study was designed to measure and compare the text difficulty of English-language consent forms to the version of those forms translated into Kiswahili. Knowing whether translated forms resemble the original English-language form in readability is important given that ethics review committees around the world generally make the assumption that a simplified English-language form will result in a simplified local language form. Without empirical data to determine whether this assumption is accurate, however, research ethics review committees may need other mechanisms to determine the readability of local language consent forms.

Assessing Readability

Readability, or reading ease, is a function of texts, while comprehension is a function of readers. Nonetheless, the concepts are inextricably linked. Readability formulas and methods, therefore, should not merely predict whether the reader can read the words, but whether the sentences have meaning to the reader. There currently are over 40 formulas that can be used to determine readability levels of English prose text. Although a smaller number of readability formulas exist for languages other than English, there are no readability formulas for Kiswahili.⁵

One method designed to measure readability of documents written in any language is called the cloze procedure. Briefly, it requires that the tester delete words from the passage of interest at regular intervals—for example, every fifth word. The tester then asks the reader to make his or her best guess to fill in the missing words from the given passage of text. The number of correctly guessed words serves as an index of how well the reader understands the meaning of the passage. Thus, the cloze procedure examines the extent to which the reader can predict the intended words based on the surrounding context. This ability to predict and make sense of textual meaning and context is consistent with how the reading process ordinarily occurs. Normal reading is not a letter-by-letter decoding, but rather a mixture of perception and guesswork.⁶The cloze procedure attempts to mimic this guesswork in a controlled manner.

In a cloze test, two characteristics significantly determine how well the reader is able to decipher a passage of text with missing words. The first is the reader’s own “redundancy utilization.” This is the reader’s ability to use his or her preexisting knowledge of vocabulary, linguistic conventions, and general knowledge to make sense of a given passage of text. The second is the “information load” of the passage itself, meaning the volume and complexity of the information provided. Thus, the second factor is a feature of the message and not of the reader.

Unlike other readability formulas, the method by which the cloze procedure predicts the difficulty of text is not based on the length of words or sentences. Rather, it measures the reader’s ability to make sense of the text directly, thereby acknowledging the relationship between individual background knowledge and text complexity in a way that word-based, predictive formulas cannot.⁷

Study Methods

Consent Forms.We used 10 pairs of consent forms (each pair containing one English form and its Kiswahili translation), or 20 forms in total, for this study. We drew English-language consent forms and their Kiswahili-translated forms from 10 clinical studies submitted to and approved by the KEMRI/NERC from 2004 to 2006. We selected studies randomly and required that they have KEMRI/NERC-approved consent forms written in English and Kiswahili.

We removed all information from the consent forms that might have identified the investigators or the institutions where the studies were conducted. We then divided each consent form into three parts: the beginning, the middle, and the end. Finally, starting from the beginning of each of the three parts, we deleted every fifth word until we had deleted 50 words in total in each part to facilitate the cloze procedure.

Study Participants.Thirty census fieldworkers from the Demographic Surveillance Survey study at the Centre of Geographic Medicine Coast in Kilifi consented to participate in this study. They were selected because they were available to do the procedure at the designated time and had completed 12 years of schooling. They agreed to conduct the cloze procedure over a total of two hours a day for a total of 15 days. The field workers completed consent forms anonymously.

Data Collection.Each subject received an English-language and Kiswahili-language consent form from every study that we looked at. However, we did not provide the same study’s consent forms in both languages consecutively, nor did we provide them in any specific order; thus, the participant might receive the Kiswahili form before the English one, and vice versa. We deleted 50 words from each of the three parts of every consent form, replacing the deleted words with a standard-length blank space.

All of the participants reviewed all of the consent forms. We instructed participants to attempt to fill in each blank with one word only, and we told them that spelling errors would not be marked wrong. We also told them that they were not being tested; rather, we told them it was the difficulty of the passage that the study was measuring. We measured participants’ comprehension scores by how many missing words each participant correctly inserted across all three passages of a form. We scored consent forms and entered data into Stata, a data analysis and statistical software program.

Data Analysis.Using the cloze procedure, we calculated an average score for each of the three sections of the 20 consent forms (10 in English, and 10 in Kiswahili). This average score would control for variations within passages and within the individual’s entire response. Based on these data, we calculated a final average score for each of the 20 forms.

We compared the average score for each English-language form with its Kiswahili translation using a paired t-test testing a null hypothesis of no difference (meaning that each individual answered English and Kiswahili at the same rate of accurate completion) versus an alternative hypothesis that the mean difference is statistically not equal to zero (meaning that each individual answered English and Kiswahili at varying rates of accurate completion). We estimated mean differences along with their 95% confidence intervals and corresponding p-values. A p-value of less than 0.05 was considered to be statistically significant. We then subtracted the Kiswahili score from the English score. We repeated this process for all passages from the 10 pairs of consent forms.

Test of Symmetry.A cloze pass mark is described in the literature as a score above 38%. This pass mark is based on a study by Bormuth⁸in which he compared information gain scores and cloze test results. Information gain tests measure how much information a reader gains from a passage by testing his or her comprehension of a passage’s subject matter before he or she sees the passage, then giving the same test to the reader after he or she sees the passage and subtracting the first score from the second. Bormuth concluded from the study that at approximately the level of 38% correct cloze items there was a noticeable increase in information gain scores.

For the purpose of our study, consent forms were categorized in the following way: forms that scored less than 38% were determined to be incomprehensible (i.e., the sentence did not make sense because it did not have adequate information load and therefore failed to communicate); those that scored 38% and above were determined to be comprehensible; and those that scored 66% and above were determined to be very easy to comprehend.

Study Results

Mean Difference.Seven out of 10 pairs of consent forms had a statistically significant mean difference in favor of English, suggesting that they were more comprehensible in English than Kiswahili. Only one of the 10 forms had a statistically significant mean difference in favor of Kiswahili, meaning that this one Kiswahili form was more comprehensible than its English counterpart. Consent forms D and I were nearly equally comprehensible in both English and Kiswahili. These results are summarized inTable 1andFigure 1.

Test of Symmetry Categorization.Table 2shows the actual number and percentage by score of each form for English and Kiswahili, andFigure 2shows the percentage of participants that scored below 38% in the cloze procedure (therefore “failing” readability according to standard cloze procedure scoring).

For six of the 10 forms, a greater percentage of respondents “failed” the Kiswahili form than the English form. Consent forms B and J, for example, demonstrate notable differences between the percentages of respondents who failed the Kiswahili form compared to the English form: 35% versus 10% and 40% versus 10%, respectively (seeTable 2). In forms A, H, and I, all 30 field workers received passing scores between 38% and 66%, or the “comprehensible” category, in both languages, suggesting that these forms were equally readable in either language. None of the respondents scored higher than 66% correct comprehension on any form, regardless of whether it was written in English or Kiswahili.

Discussion