| Home | E-Submission | Sitemap | Contact us |  
Acute and Critical Care > Volume 33(2); 2018 > Article
Cho, Lee, Lee, Park, Kim, Kim, Lee, Park, Cho, Yoon, Lee, Lee, and Lee: Transcultural Adaptation and Validation of Quality of Dying and Death Questionnaire in Medical Intensive Care Units in South Korea



Providing palliative care to dying patients in the intensive care unit (ICU) has recently received much attention. Evaluating the quality of dying and death (QODD) is important for appropriate comfort care in the ICU. This study aimed to validate the Korean version of the QODD questionnaire.


This study included decedents in the ICUs of three tertiary teaching hospitals and one secondary hospital from June 2016 to May 2017. ICU staff members were asked to complete the translated QODD questionnaire and the visual analogue scale (VAS) questionnaire within 48 hours of patient death. The validation process consisted of evaluating construct validity, internal consistency, and interrater reliability.


We obtained 416 completed questionnaires describing 255 decedents. The QODD score was positively correlated with the 100-VAS score (Pearson correlation coefficient, 0.348; P<0.001). An evaluation of the internal consistency presented favorable results (calculated Cronbach’s alpha if a given item exceeded 0.8 in all items). The interrater reliability revealed no concordance between doctors and nurses.


The QODD questionnaire was successfully translated and validated in Korean medical ICUs. We hope further studies that use this valuable instrument will be conducted in Korea.


Since many patients admitted to intensive care units (ICUs) worldwide are critically ill, these units have a high mortality rate; thus, the burden of critical care is a global problem [1,2]. As a result, much attention has recently been paid to providing comfort care to dying patients in the ICU. Pain and symptom management, death preparation, achieving a sense of completion, and making decisions about treatment preferences are important factors in comfort care [3]. However, these points have not been identified as important issues in actual practice [4,5].
Providing comfort care is one of the major goals in palliative settings [6]. Maintaining an acceptable quality of life is important for dying patients in this regard. The quality of dying and death (QODD) is defined as the degree to which a person’s preferences for dying and the moment of death agree with observations of how the person actually died. Patrick et al. [7] proposed a model for evaluating the QODD that consists of 31 aspects that can be rated by clinicians after a patient dies. The QODD score, an average of the ratings of each completed item, represents the degree of a patient’s QODD as perceived by raters. Since the original QODD model was proposed, various versions of similar questionnaires intended for different raters (e.g., family members, medical staffers) and in different healthcare settings (e.g., outpatient, hospice, ICU) have been developed [8-10].
Many studies have been conducted in ICU settings using instruments like the QODD model, particularly in the United States and Europe. Currently, there is no validated questionnaire in South Korea to evaluate the quality of dying ICU patients. In this study, we translated the QODD questionnaire version 3.2 into Korean to validate the Korean version of the QODD questionnaire.


Study Design and Data Collection

This study was conducted prospectively in the medical ICUs at three tertiary teaching hospitals and one secondary hospital from January 2015 to February 2016. Eligible patients were those who died in a medical ICU. Attending doctors (resident, fellow, or attending physician) and nurses caring for these patients at the time of death were asked to complete the translated QODD questionnaire within 48 hours of the death. The raters were permitted to complete the questionnaires only a limited number of times per month (a maximum of three times per month for a doctor and two times per month for a nurse).
All completed questionnaires were collected in a designated storage box in the ICU. The raters were also asked to complete a brief questionnaire about their own information (e.g., age, sex, religion, education, length of experience). The decedents’ demographic characteristics (e.g., sex, age, previous medical history) and clinical information (e.g., reason for ICU admission, severity, hospital day, length of ICU stay, life support equipment, resuscitations within 24 hours) were obtained from their medical records.

QODD Questionnaire

We translated version 3.2 of the QODD questionnaire, which is designed for self-administration by a healthcare professional who cared for a patient who later died in a hospital or ICU setting [11]. We obtained permission to use the questionnaire from one of the developers prior to commencement of the study. Two experts translated the original questionnaire into Korean individually and then integrated their individual translations into a single translation. This version was retranslated into English by two additional experts, and an English language expert compared that version with the original. The final Korean version of the QODD questionnaire is included in the Supplementary Material.
The QODD questionnaire has four subcategories with items to be rated: patient experiences at the end of life (15 items), medical care at the end of life (four items), experience at the moment of death (three items), and overall ratings of care (two items). For each item, all raters were asked to evaluate the quality of each experience on a scale of 0 (“terrible”) to 10 (“almost perfect”). Twenty designated items are used for calculating the QODD score, which involves summing valid ratings, dividing by the number of valid items and finally multiplying the result by 10. The QODD score ranges from 0 to 100, with a higher score indicating a better quality of death for the patient [11].

Visual Analogue Scale Questionnaire

The raters were also asked to complete a visual analogue scale (VAS) questionnaire to validate the QODD questionnaire. The VAS questionnaire consisted of two items for rating: “effect on the patient’s quality of death by medical treatment and the patient’s experience in the ICU” and “overall quality of ICU care by medical professionals.” Each item was rated on a scale from 1 (excellent) to 10 (inadequate). The mean value of valid items multiplied by 10 was designated as the “VAS score.”

Validation and Statistical Analysis

All data were presented as median (range) or median (interquartile range) for continuous variables and number (percent) for categorical variables. We used IBM SPSS version 22.0 (IBM Corp., Armonk, NY, USA) for the statistical analyses. The validation process consisted of construct validity, internal consistency, and interrater reliability. Construct validity is defined as the extent to which a test assesses the concept intended to be measured [12]. To assess construct validity, we calculated the Pearson correlation coefficient between the QODD score and the 100-VAS score. Internal consistency estimates were related to item homogeneity or the degree to which the items on a test jointly measure the same construct [13]. We assessed internal consistency by calculating the Cronbach’s alpha if a given item was deleted. A Cronbach’s alpha that exceeded 0.8 indicated good internal consistency. The interrater reliability is the extent of agreement among raters. The interrater reliability between doctors and nurses was assessed by calculating intraclass correlation coefficients (ICCs). ICCs that exceed 0.75 indicate an excellent agreement, while those below 0.40 suggest poor agreement. A P-value <0.05 was considered statistically significant.

Research Ethics

This study was approved by the Institutional Review Board of each hospital. Written informed consent was obtained from medical staff at the time of each patient’s death. The requirement for written consent from the patients was waived because their personal information was not used. We conducted this study in compliance with the Declaration of Helsinki.


Baseline Characteristics of Patients and Medical Staff Members

Of the 255 total patients with a mean age of 66.6 years, 39.6% had an underlying malignancy at the time of ICU admission (Table 1). Respiratory failure was the leading cause of ICU admission, and 20.0% of patients were admitted for post-resuscitation care. Forty-eight (18.8%) had received cardiopulmonary resuscitation within 24 hours of death. Documented do-not-resuscitate orders were in place for 206 patients (80.8%).
In total, 177 medical staff members completed 416 questionnaires during the 12-month study period, including 66 doctors: 52 residents (78.8%), nine fellows (13.6%), and five attending physicians (7.6%). The characteristics of the respondent ICU medical staff members are detailed in Table 2.

Validation Process

The results of the assessment of construct validity are presented in Table 3. The QODD score and subcategorical items were positively correlated with the 100-VAS score (Pearson correlation coefficient, 0.348; P<0.001).
Table 4 contains the results of the assessment of internal consistency. The calculated Cronbach’s alpha exceeded 0.8 for all items, which showed good internal consistency and item heterogeneity.
Three hundred six questionnaires that were completed by both doctors and nurses and described 153 patients were used for assessing interrater reliability. the ICC for the QODD score was 0.077, and there was no statistical significance (P = 0.312) (Table 5).


We conducted this study because an instrument that can objectively evaluate the quality of life of dying patients should be available for use in Korean ICUs. An assessment of construct validity demonstrated a positive correlation between the 100-VAS score and the QODD score. The calculated scores for the subcategories also were generally correlated with the 100-VAS score. In the evaluation of internal consistency, all items presented high Cronbach’s alpha (>0.8) values. These data suggest that QODD scores calculated from the translated questionnaire could provide a useful and objective evaluation of the quality of life of dying patients, as perceived by ICU staff members in Korea.
However, in terms of construct validity, there were relatively low correlations between the QODD and 100-VAS scores despite the extremely low P-value (Table 3). This finding may be explained by the number of extreme scores. For example, 50.8% of the raters scored 0 for the item “Being able to feed oneself” (47.0% in “Having a spiritual service or ceremony before death” and 42.6% in “Having one or more visits from a religious or spiritual advisor”). Many raters might not have fully understood what was being asked or lacked clarity regarding several questions of the QODD questionnaire. A revised version of the QODD questionnaire that presents the actual status of Korean ICUs should be discussed in further studies.
In the present study, a considerable number of medical staff members had no experience with comfort care education (Table 2). Currently, comfort care education has been not incorporated into standard medical school curricula in Korea. Still, the ICU is a space where medical personnel are likely to encounter dying patients, and most ICU deaths are preceded by a decision to withhold or withdraw life-sustaining care [14,15]. Hence, comfort care education is important for medical professionals who provide comfort care. Data from a U.S. national survey showed that medical students’ sense of preparedness for comfort care increases with more coursework and bedside teaching [16]. Extended opportunities for comfort care education in formal medical curricula should therefore be provided.
Analysis of interrater reliability showed no concordance in QODD score between doctors and nurses (Table 5). We could not determine which factors were responsible for this finding. However, we suspect that actual differences between how doctors and nurses perceive the quality of life of a dying patient may be related to the amount of time spent in direct contact with the patient. Levy et al. [17] reported discordance in the QODD rating between resident physicians and nurses. A study by Breen et al. [18] also identified conflicts among staff members in 48% of cases when withdrawing life-sustaining care. In Korean medical ICUs, as is the case worldwide, nurses generally spend more time in direct contact with and closely caring for patients than do doctors. Therefore, nurses might gain a more in-depth understanding of the experiences and wishes of dying patients than doctors.
Although a good quality of death is one of the most important goals of proper comfort care, this concept is unfamiliar to Korean ICU workers. Very few studies have objectively evaluated patients’ quality of death. Many dying patients still receive meaningless treatments [19], and very limited ICU visiting policies can prohibit them from spending much time with their families. Patient autonomy is often overlooked when discussing whether life-sustaining care should be provided [20]. Thus, it can be assumed that the quality of death in Korean ICUs is relatively low. Further studies that use our validated and reliable tool can demonstrate the current status of the quality of death in Korean ICUs and offer suggestions for improving the quality of death.
This study had several limitations. We did not include patients from surgical, neurologic, or other types of ICUs where comfort care is also frequently required. Samples from just four medical ICUs of acute-care-based hospitals may not be enough to validate the assessment tool for all cases. An extended validation study in various ICU settings should be conducted in the future to confirm generalizability.
In conclusion, a translated QODD questionnaire version 3.2 was successfully validated in Korean medical ICUs. We hope that further studies conducted in Korea will include this useful instrument.


No potential conflict of interest relevant to this article was reported.

Supplementary Materials

The online-only Supplement data are available with this article online: https://doi.org/10.4266/acc.2017.00612.

Table 1.
Baseline characteristics of patients died in ICUs
Variable Total patients (n = 255)
Male sex 159 (62.4)
Age (yr) 69 (18–94)
Malignancy 101 (39.6)
Reason for ICU admission
 Sepsis 59 (23.1)
 Respiratory failure 148 (58.0)
 Heart failure 22 (8.6)
 Renal failure 35 (13.7)
 Hemorrhagic shock 7 (2.7)
Post-resuscitation care 51 (20.0)
APACHE II score 27 (0–71)
SOFA score 12 (1–24)
Hospital day 13 (0–196)
Length of ICU stay (day) 4 (0–117)
Life support equipment
 Mechanical ventilator 228 (89.4)
 Continuous renal replacement treatment 124 (48.6)
 Extracorporeal membrane oxygenation 17 (6.7)
 Others 9 (3.5)
CPR within 24 hours prior to death 48 (18.8)
DNR documentation prior to death 206 (80.8)

Values are presented as number (%) or median (range).

ICU: intensive care unit; APACHE: Acute Physiology and Chronic Health Evaluation; SOFA: Sequential Organ Failure Assessment; CPR: cardiopulmonary resuscitation; DNR: do-not-resuscitate.

Table 2.
Baseline characteristics of medical staffs
Variable Doctor (n = 66)a Nurse (n = 111)
Age (yr) 31 (25–47) 27 (23–42)
 Male 33 (50) 11 (9.9)
 Female 33 (50) 100 (90.1)
 Catholicism 15 (22.7) 14 (12.6)
 Christianity 15 (22.7) 26 (23.4)
 Buddhism 5 (7.6) 10 (9.0)
 No religion 31 (47.0) 61 (55.0)
Experience (mo)
 Medical personnel 42 (24–252) 24 (0–216)
 ICU staff 12 (0–120) 24 (0–180)
Experience of EOL care education
 Yes 46 (69.7) 27 (24.3)
 No 20 (30.3) 84 (75.7)
No. of completed questionnaires 213 203

Values are presented as median (range) or number (%).

ICU: intensive care unit; EOL: end of life.

a Sixty-six doctors were participated: resident (n = 52, 78.8%), fellow (n = 9, 13.6%), and attending physician (n = 5, 7.6%).

Table 3.
Construct validity
Variable Pearson correlation coefficient P-value
QODD score 0.348 <0.001
Patient’s experience at the end of life 0.261 <0.001
Medical care at the end of life 0.332 <0.001
Experience at moment of death 0.234 <0.001
Overall ratings of care 0.346 <0.001

QODD: quality of dying and death.

Table 4.
Internal consistency
Item Rating Cronbach’s αa
Patient’s experience at the end of life
 Having pain under control 5 (3–7) 0.834
 Having control over what is going on around you 2 (1–3) 0.835
 Being able to feed oneself 0 (0–2) 0.837
 Being able to breath comfortably 2 (1–5) 0.836
 Feeling at peace with dying 2 (1–4) 0.834
 Feeling unafraid of dying 3 (1–5) 0.835
 Being able to laugh and smile 0 (0–2) 0.837
 Keeping one’s dignity and self-respect 2 (0–3) 0.836
 Spending time with family and friends 3 (1–7) 0.835
 Spending time alone 2 (0–5) 0.837
 Being touched or hugged by loved ones 3 (0–6) 0.835
 Saying goodbye to loved ones 1 (0–3) 0.834
 Clearing up bad feelings 1 (0–3) 0.837
 Having one or more visits from a religious or spiritual advisor 1 (0–4) 0.836
 Having a spiritual service or ceremony before death 1 (0–3) 0.836
Medical care at the end of life
 Experience of receiving mechanical ventilation 3 (2–5) 0.834
 Experience of receiving dialysis 4 (2–7) 0.837
 Receiving right amount of sedation 6 (3–8) 0.834
 Having discussed end-of-life care wishes with your doctor 2 (0–5) 0.836
Experience at moment of death
 Having desired people present at the time of one’s death 7 (3–8) 0.835
 State at moment of death (i.e., asleep, awake, unconscious) 4 (2–7) 0.833
 Overall quality of one’s dying 3 (2–5) 0.834
Overall ratings of care
 By all medical personnel 7 (5–8) 0.837
 By assigned doctor 6 (5–8) 0.836

Values are presented as median (interquartile range).

a Calculated Cronbach’s alpha if a given item deleted.

Table 5.
Interrater reliability
Variable Intraclass correlation coefficient P-value
QODD score 0.077 0.312

QODD: quality of dying and death.


1. Adhikari NK, Fowler RA, Bhagwanjee S, Rubenfeld GD. Critical care and the global burden of critical illness in adults. Lancet 2010;376:1339-46.
crossref pmid
2. Vincent JL, Marshall JC, Namendys-Silva SA, Francois B, Martin-Loeches I, Lipman J, et al. Assessment of the worldwide burden of critical illness: the intensive care over nations (ICON) audit. Lancet Respir Med 2014;2:380-6.
crossref pmid
3. Steinhauser KE, Christakis NA, Clipp EC, McNeilly M, McIntyre L, Tulsky JA. Factors considered important at the end of life by patients, family, physicians, and other care providers. JAMA 2000;284:2476-82.
crossref pmid
4. Hofmann JC, Wenger NS, Davis RB, Teno J, Connors AF Jr, Desbiens N, et al. Patient preferences for communication with physicians about end-of-life decisions. Ann Intern Med 1997;127:1-12.
crossref pmid
5. Connors AF, Dawson NV, Desbiens NA, Fulkerson WJ, Goldman L, Knaus WA, et al. A controlled trial to improve care for seriously ill hospitalized patients. The study to understand prognoses and preferences for outcomes and risks of treatments (SUPPORT). JAMA 1995;274:1591-8.
crossref pmid
6. Beckstrand RL, Callister LC, Kirchhoff KT. Providing a “good death”: critical care nurses’ suggestions for improving end-of-life care. Am J Crit Care 2006;15:38-45.
crossref pmid
7. Patrick DL, Engelberg RA, Curtis JR. Evaluating the quality of dying and death. J Pain Symptom Manage 2001;22:717-26.
crossref pmid
8. Glavan BJ, Engelberg RA, Downey L, Curtis JR. Using the medical record to evaluate the quality of end-of-life care in the intensive care unit. Crit Care Med 2008;36:1138-46.
crossref pmid pmc
9. Treece PD, Engelberg RA, Crowley L, Chan JD, Rubenfeld GD, Steinberg KP, et al. Evaluation of a standardized order form for the withdrawal of life support in the intensive care unit. Crit Care Med 2004;32:1141-8.
crossref pmid
10. Curtis JR, Patrick DL, Engelberg RA, Norris K, Asp C, Byock I. A measure of the quality of dying and death: initial validation using after-death interviews with family members. J Pain Symptom Manage 2002;24:17-31.
crossref pmid
11. University of Washington School of Medicine. Quality of dying and death questionnaire for nurses: version 3.2. Healthcare professional after-death self-administered questionnaire [Internet] Seattle (WA), University of Washington School of Medicine. [cited 2018 May 11]. Available from: http://depts.washington.edu/eolcare/pubs/wp-content/uploads/2011/08/nurseqodd3_2.pdf.

12. Bagozzi RP, Yi Y, Phillips LW. Assessing construct validity in organizational research. Adm Sci Q 1991;36:421-58.
13. Henson RK. Understanding internal consistency reliability estimates: a conceptual primer on coefficient alpha. Meas Eval Couns Dev 2001;34:177-89.
14. Ferrand E, Robert R, Ingrand P, Lemaire F, French LATAREA Group. Withholding and withdrawal of life support in intensive-care units in France: a prospective survey. French LATAREA Group. Lancet 2001;357:9-14.
crossref pmid
15. Prendergast TJ, Puntillo KA. Withdrawal of life support: intensive caring at the end of life. JAMA 2002;288:2732-40.
crossref pmid
16. Billings ME, Engelberg R, Curtis JR, Block S, Sullivan AM. Determinants of medical students’ perceived preparation to perform end-of-life care, quality of end-of-life care education, and attitudes toward end-of-life care. J Palliat Med 2010;13:319-26.
crossref pmid pmc
17. Levy CR, Ely EW, Payne K, Engelberg RA, Patrick DL, Curtis JR. Quality of dying and death in two medical ICUs: perceptions of family and clinicians. Chest 2005;127:1775-83.
crossref pmid
18. Breen CM, Abernethy AP, Abbott KH, Tulsky JA. Conflict associated with decisions to limit life-sustaining treatment in intensive care units. J Gen Intern Med 2001;16:283-9.
crossref pmid pmc
19. Bae JM, Gong JY, Lee JR, Heo DS, Koh Y. A survey of patients who were admitted for life-sustaining therapy in nationwide medical institutions. Korean J Crit Care Med 2010;25:16-20.
20. Shin K, Mok JH, Lee SH, Kim EJ, Seok NR, Ryu SS, et al. The current status of medical decision-making for dying patients in a medical intensive care unit: a single-center study. Korean J Crit Care Med 2014;29:160-5.
Editorial Office
#805-806, Yongseong Biztel, 109 Hangang-daero, Yongsan-gu, Seoul 04376, Korea
TEL: +82-2-2077-1533   FAX: +82-2-2077-1535   E-mail: acc@accjournal.org
About |  Browse Articles |  Current Issue |  For Authors and Reviewers
Copyright © The Korean Society of Critical Care Medicine.                 Developed in M2PI
Close layer
prev next