Development and Evaluation of Content Validity Knowledge Questionnaire for Patients with Diabetes Mellitus

Background: Currently in the Czech Republic there is lack of questionnaire that would verify the level of knowledge of patients with diabetes. Such a questionnaire is needed because the numbers of diabetic patients are increasing and need to be able to evaluate the effectiveness of the implementation of education, which is staffed and very time consuming. The questionnaire assessing patient knowledge is one of the ways you can evaluate the effectiveness of education. Aim: The aim was to create a knowledge questionnaire for people with diabetes. The objective was to propose major themes, areas and specific items of the newly formed knowledge questionnaire, to assess the content validity of the proposed main areas of proposed specific questionnaire items and individual parts of the entire questionnaire. Methods: For questionnaire creation itself and for assessing its content validity was established cooperation with 12 field experts. On the basis of this cooperation through research methods Delphi, the study literature and publications have been proposed major themes of the questionnaire, its main areas and specific items. Content validity was evaluated by calculating the content validity index. Results: The final version of the questionnaire is made up of four main topics, regions 16 and 53 items. Content validity index (CVI) for each area of the questionnaire ranged from 0.50 to 1.00, for each item in the range from 0.33 to 1.00. CVI for each part of the questionnaire (S-CVI/Ave) ranged from 0.90 to 0.98. Conclusion: The newly created measurement tools to assess knowledge in patients with diabetes was assessed as highly valid content. In the next step of research will be examined further psychometric properties of the questionnaire and its comprehensibility for respondents.


INTRODUCTION
Diabetes mellitus (diabetes) is characterized by hyperglycemia, which is due to the insufficient effect of insulin in case of its absolute or relative deficiency (1).Diabetes affects not only the patients themselves and their close persons but also the system of health and social care.The disease involves numerous new situations that affect the quality of the patient's everyday life (2).
The number of people with diabetes is constantly rising for all types of diabetes.According to data provided by the Institute of Healthcare Information and Statistics of the Czech Republic, 861,647 persons with diabetes lived in the Czech Republic in 2013.The number of patients has increased by more than 2.5 times since 1980.If the number of patients continues to grow at a similar, it can be assumed that in 2035 one tenth of the Czech population will suffer from diabetes regardless of age.In 2012, the number of patients treated only by a diet dropped from 120 to 107 thousand persons, indicating the increasing use pharmacotherapy, which is often relatively costly (3,4).
One of the first insulin-treated patients in the then Czechoslovakia who had been living with diabetes for 60 years said: "The more diabetic patients know about their illness, the more opportunities they have to adjust their condition and live to a high age.The doctor's role consists in managing the treatment; the actual treatment is realized by the very patient himself " (5).The knowledge about the illness is very closely related to self-monitoring, self-management of one's own treatment and thus also to the compensation of diabetes.As Trunček (6) suggests, the nature of knowledge is based upon the human insight and results from the interaction between real environment and the actual human.The interest in knowledge as a problem area is not a new phenomenon as it may seem; only the application and use of these concepts has increased significantly in recent years.We need to learn how to handle, create, work with and evaluate such knowledge.
The strategies for assessing knowledge in patients with diabetes are constantly improving, and it is also increasingly being enforced even by the actual healthcare providers.Care providers are lacking verified measurement tools to assess the knowledge of patients (7).Assessing and influencing the knowledge of patients is an essential part of the educational process.Given that diabetes involves certain limitations that patients should be familiar with, the success of disease compensation directly depends on the patients' knowledge.Education provided as part of healthcare should be effective because it consumes a lot of healthcare professionals' time.There is currently no proven tool in the Czech Republic to assess the knowledge about diabetes.As for proven tools in place abroad, we may mention for instance the Diabetes Knowledge Test (DKT), created in 1998 (8), the Diabetes Knowledge Questionnaire (ADKnowl), the latest version of which was updated in 2009 (9), the Diabetes Knowledge Questionnaire (DKQ), created in 2011 (7), or the 24-item Diabetes Knowledge Questionnaire (DKQ-24), derived from the 60-item original questionnaire in 2001 (10).In the discussions with experts it was agreed that the existing tools are not suitable for use in the Czech Republic and, on the other hand, it would be useful for Czech patients and healthcare professionals to develop a new tool evaluating the knowledge of patients with diabetes.This is based on several reasons: the age of the existing questionnaires, their non-correlating content with current recommendations as well as the cultural non-transferability of the items (for instance due to patients' different lifestyle and social conditions).
At the beginning of the creation of a new measuring tool, it is necessary to determine the main themes to be addressed by such tool, i.e. to determine its construct, main areas, and finally also its specific items.This process takes place in cooperation with the authors of the questionnaire and experts in the given field.Every newly created measuring tool should have its psychometric properties verified, including validity and reliability.Developing knowledge assessment tools is primarily about evaluating their content validity.Content validity assesses to what extent the selection of items is adequate to the characteristics to be measured.In the process of such assessment, we refer to the experts' opinion (11,12).A calculation of the content validity index is applied for evidencing the content validity.This index is calculated based on an evaluation of the relevance of individual themes or items in the questionnaire by experts.For the calculation of the content validity index for each item in the questionnaire, experts are mostly asked to rate the relevance of each item using a 4 point scale (13).

OBJECTIVE OF THE STUDY
The objective of the present paper was to create a questionnaire for persons with diabetes.Part of this objective was to suggest the key themes, areas and specific items of a newly developed knowledge questionnaire, to assess the content validity of the main suggested areas, specific suggested items and the various sections of the entire questionnaire.

METHODOLOGY
The questionnaire was developed in the following phases: 1. Selection of experts.2. Defining the questionnaire construct (i.e. the main themes of the questionnaire).2.1.Defining the main questionnaire areas and calculating their content validity index.3. Defining and selecting the questionnaire items and calculating their content validity index.4. Calculating the content validity index for the various parts of the generated questionnaire.
In the first phase of the research, a targeted selection of experts took place to provide diversity of this group.All of the experts contacted provided their curriculum vitae documenting their expertise.Specifically, these experts were 5 physicians with experience in diabetology, 6 general nurses with experience with diabetes patients, and one nutritional therapist, all of whom had at least 3 years of experience in diabetes care.
The tool design definition phase included a proposal of the questionnaire structure.The design was worked up based on a detailed study of literature, professional publications and consultation with experts in diabetology.The following phase of the research involved the designing of the main areas, i.e. components forming the questionnaire and expressing the major aspects of type 1 and type 2 diabetes, using the Delphi method applied by the above-mentioned experts.The Delphi method consists in obtaining information in stages from a group of experts using a series of questionnaires.This technique takes place in two or more rounds, while mutual anonymity of the experts is maintained (14).In the first round, the experts were asked to suggest areas that, in their view, should be included in the questionnaire structure.In the second round, the list of suggested themes was sent to experts, who then used a 4 -point scale (1 -definitely not, 2 -rather not, 3 -rather yes, 4 -definitely yes) to rate the themes in terms of applicability in the questionnaire.Based on the experts' judgement, the content validity index (CVI) was calculated for each proposed area, by dividing the number of experts who rated the various items with 3 or 4 points on the 4-point scale by the the total number of assessing experts.Following recommendations, 0.8 was set as the minimum acceptable value of the content validity index for including the area in the questionnaire.Where this value was not reached, the suggested area was removed from the questionnaire.
After completing the questionnaire construct design phase, in the 3rd round of the Delphi method, experts were asked to design specific items for the individual suggested areas of the questionnaire.The wording of the items should allow to be answered by "Yes, I agree", "No, disagree" or "I don't know".Option "I don't know" was included in the answer options to eliminate mere guessing the correct answer where the respondents do not know or are unsure about the correct answer.All the proposed items were evaluated in the last, fourth round, and subsequently, items suggested multiple times and suggested non-knowledge--related items were eliminated.All experts were also asked to comment on the clarity and formulation of the various items.The wording of the items was adjusted to correlate with current recommendations.The experts used a 4-point scale to rate the content validity of the proposed questionnaire, as was the case of the main areas of the questionnaire.Following the recommendations by Polit and Beck (15), 0.78 was selected as the minimum acceptable value of the content validity index of individual items.Where this value was not reached, the suggested item was removed from the questionnaire.
After finalizing the questionnaire by the experts, the content validity index was calculated for the various parts of the questionnaire (S-CVI/Ave), applying 0.9 as the qualification criterion, in compliance with the procedure by Polit and Beck (15).

RESULTS
Following the study of professional literature, publications and consultations with experts, the construct of the newly created questionnaire intended for type 1 and type 2 diabetes treated patients taking either peroral anti-diabetic agents, insulin, or a combination thereof, was determined in the first stage of the survey, structured as follows: • Part A: general (for patients with type 1 and type 2 diabetes).• Part B: only for patients with type 1 diabetes.
• Part C: only for female patients with type 1 diabetes.
• Part D: only for patients with type 2 diabetes.
• Part E: only for insulin-treated patients.
In the second phase of the research, the main areas to be addressed by the questionnaire were suggested in cooperation with the experts.9 areas were proposed for the general part of the questionnaire, 3 for the part intended for patients with type 1 diabetes, 1 area for women with type 1 diabetes, 3 areas for the part intended for patients with type 2 diabetes and 1 area for insulin-treated patients.Content validity index was calculated for each proposed area.The area called the Social Aspects of Diabetes from the general part of the questionnaire, did not reach the set qualification minimum, achieving only a content validity value of 0.5.The content validity results are visualised in Table 1 below.
In the third phase of the survey, 67 items were proposed for a general part of the questionnaire, 17 for the part intended for type 1 diabetes, 8 for women with type 1 diabetes -concerning area of pregnancy, 15 items for type 2 diabetes items and 7 items for the insulin treatment-related part.After eliminating duplicities in proposed items and non-knowledge-related items, the questionnaire consisted of 41 items in the general section, 3 items in the area focused on women with type 1 diabetes, 14 items for type 1 diabetes area, 11 items for type 2 diabetes area and 5 items for insulin treatment related area.In the fourth phase of the survey, experts were asked to evaluate the relevance of the individual items for the inclusion in the final version of the questionnaire on a 4-point scale.The content validity index, calculated for each item of the questionnaire, ranged from 0.33 to 1.0.16 items from the general part, 2 items from the part for patients with type 1 diabetes and 3 items from the part for type 2 diabetes patients did not meet the minimum qualification value of content validity index defined as 0.78.The number of items proposed and the final number of items for each area of the questionnaire are shown in Table 1.
In the final phase of the research, the content validity index of the various questionnaire parts was evaluated using the S-CVI/Ave calculation method.The general part of the questionnaire reached the value of 0.90, part for type 1 diabetes patients 0.95, part for women with a type 1 diabetes 0.92, part for patients with type 2 diabetes 0.98 and a part for insulin--treated patients 0.90.Detailed results are presented in Table 1.

DISCUSSION
Knowledge evaluation is one of the essential parts of education.As reported by Šmahelová and Lášticová ( 16), the main purpose of education is to provide patients with information in such a form and to an extent that makes them understand the nature and treatment of their disease.Patients are expected to take greater responsibility for their health and its support, but this depends largely on their skills and knowledge (17).The developed questionnaire was designed to determine the level of knowledge of diabetic patients in defined areas and to possibly identify areas where the patients need to be educated in a greater extent or in a different form.It can also be used for identifying the patients' educational needs, representing an important aspect of the educational process.Educational needs should be defined at the very beginning of education in order to determine Legend: CVI -content validity index, S-CVI/Ave -scale content validity index/averaging method the extent to which the education itself should be provided.When identifying the educational needs, it is also possible to use questionnaires and other measurement techniques, but it is always necessary to give the patient or his/her close persons additional room for commenting on other possible educational needs (18).The aim of the present paper was to assess the content validity of the newly created tool for assessing the patients' knowledge in 4 rounds, using the Delphi method.Verification of content validity is one of the most important steps in creating a new measurement tool and is an indispensable factor in structural validity assessment (19).The achieved results show that the newly created tool has a high content validity.
For possible comparison, we refer to knowledge questionnaires used abroad, such as the ADKnowl questionnaire created by Clare Bradley (9).This questionnaire consists of 120 items, divided into a general part, a part for insulin-treated persons and a part for persons treated using medication.The above-mentioned questionnaire also consists of items with answer options "Yes", "No" and "Don't know".The length of the questionnaire and hence also the amount of time to be spent by the patients when completing it is considered its greatest weakness.Another questionnaire in use abroad is the Diabetes Knowledge Questionnaire (DKQ), which was designed in the form of a test with multiple choices of answers (7).The general part of this questionnaire consists of 12 questions; 2 questions are addressed only to those who are pharmacologically treated and one question is addressed only to persons with type 1 diabetes.Its authors state that the questionnaire wording is in line with international recommendations provided in expert literature, but it needs to be regularly revised, always in line with the latest recommendations and standards.This limitation should be taken into account in the development of knowledge assessment tools, hence also in regards to the questionnaire created for the Czech population.The items wording should be regularly revised in order to maintain the content validity of the tool.Another knowledge evaluation tool for diabetic patients is the Diabetes Knowledge Questionnaire -24, derived from the English original and translated into Spanish language (10).This questionnaire also consists of 24 items with answer options "Yes", "No" and "Don't know".As compared to tools giving possible responses options or tests containing Likert scales, a tool with these response options is simpler to create and also better suitable for bilingual application.Another important foreign questionnaire is the Diabetes Knowledge Test (DKT), consisting of 23 items, 14 of which are intended for people without insulin therapy and 9 for persons with insulin therapy (8).The authors specify approximately 15 minutes as the estimated time required to complete the questionnaire.The expected time for completing our Czech questionnaire is 20 minutes, which is considered to be an optimal amount of time; however, it always depends on the research aim and the interviewee's relation to the theme (20).
Since questionnaires evaluating diabetic patients' knowledge are created for various countries and languages, always in somewhat different areas of diabetology, with different numbers of items and, of course, different choices of answers, it is very difficult to compare the results of research surveys applying different measuring instruments.It should also be noted that for no one of the above-mentioned measuring instruments the authors state the results of content validity assessment.
Calculating the content validity index for the entire measurement tool is an important component of questionnaire content validity assessment.According to Polit and Beck (15), many authors do not mention this important value in their studies.The content validity for the entire measuring tool can be evaluated using two methods, either as S/CVI/UA (scale content validity index/universal agreement) or as S-CVI/Ave (scale content validity index/averaging method).The S-CVI/UA calculation method has not been used because its application may potentially cause problems with the increasing number of experts working on validity assessment as the likelihood of concordance among these experts decreases.As mentioned above, the S-CVI/Ave value was evaluated as sufficient for the entire tool.
Using the generally accepted Delphi method, and based on the collaboration of the questionnaire authors and the team of experts, a unique tool for assessing diabetic patients' basic knowledge in selected areas was developed in the Czech Republic.It was created based on the generally accepted and predetermined research methodology, supported by foreign studies focusing on the same or similar themes.

CONCLUSION
Based on collaboration with diabetes experts, a questionnaire was developed to assess the knowledge of patients with diabetes as part of the process of their education.This questionnaire can be a useful tool for evaluating the effectiveness of educating diabetic patients and may indicate areas, where the education process requires improvement or change.A proof of effective education may also increase the motivation of healthcare payers to reimburse specific forms of education.The questionnaire was created on the basis of the most recent insights and state-of-the-art processes for the development of measuring instruments, and the Delphi method was used in its preparation.The content validity of the various questionnaire parts and specific items was rated as high.Questionnaire areas and items that failed to reach the minimum qualification value of the content validity index were disqualified.It can be stated that the generated questionnaire achieves the quality level of foreign instruments owing to a detailed description of the methodology of its development.After verifying all psychometric properties, the questionnaire might find its application primarily in the educational domain of care for diabetes patients, both for assessing the educational needs and the effectiveness of education provided.
The next development phase of this questionnaire will involve an assessment of the comprehensibility of the various items for the respondents, and further psychometric characteristics of the questionnaire will be verified.

Table 1
Questionnaire structure with evaluation by I-CVI and S-CVI/Ave