3 Rubric & Answers - Semrau Question 1 Is the study population identified and described? Are eligibility criteria specified? Are the sample selection procedures clearly delineated? Study Population Eligibility criteria 2 3 What approach do the authors use to establish and content validity? What type of sampling plan was used? Would an alternative sampling plan have been preferable? Was the sampling plan one that could be expected to yield a representative sample? Sampling plan. Make sure to include the sampling plan for each of the different studies conducted. Alternative sampling plan Sampling plan yielding representative sample 4 5 If sampling was stratified, was a useful stratification variable selected? If a consecutive sample was used, was the time period long enough to address seasonal or temporal variation? Stratification Consecutive sampling If cluster sampling was utilized, what were the clusters and how was sampling done within clusters? Clusters Sampling method within clusters How were people recruited into the sample? Does the method suggest potential biases? Recruitment Answers Possibl e Points General adult population in humanitarian settings of conflict or other disasters in low- and middle income countries. 5 Individuals in humanitarian settings of conflict or other disasters in low- and middle income countries 5 9 7 2 5 3 3 5 5 3 Potential biases 3 6 7 8 9 Did some factor other than the sampling plan affect the representativeness of the sample? Are possible sample biases or weaknesses identified by the researchers themselves? Are key characteristics of the sample described (e. g., mean age, percent female)? If yes, please provide that information. 5 5 5 Was the sample size justified on the basis of a power analysis or other rationale? 5 What approach do the authors use to establish criterion validity? 5 11 What approach do the authors use to establish interrater reliability? 5 12 What approach do the authors use to establish testretest reliability? 5 10 13 Does the sample support inferences about external validity? To whom can the study results reasonably be generalized? External validity Generalizability Source: Nursing Research: Generating and Assessing Evidence for Nursing Practice, 9th edition, p.289. 5 5 Assignment 3 Rubric & Answers - Semrau Question 1 Is the study population identified and described? Are eligibility criteria specified? Are the sample selection procedures clearly delineated? Study Population Answers General adult population Phase 1- Possibl e Points 5 Eligibility criteria 2 3 What approach do the authors use to establish and content validity? What type of sampling plan was used? Would an alternative sampling plan have been preferable? Was the sampling plan one that could be expected to yield a representative sample? Sampling plan. Make sure to include the sampling plan for each of the different studies conducted. Individuals in humanitarian settings of conflict or other disasters in low- and middle income countries 5 The HESPER Scale was modeled on a mental health instrument, the Camberwell Assessment of Need Short Appraisal Schedule (CANSAS),19 which has well-established reliability and validity. The CANSAS has been modied successfully for numerous populations22---25 and adapted for use in several countries.26 It has been used on a wide range of populations, including asylum seekers and refugees in the United Kingdom27,28 and torture victims in centers of the International Rehabilitation Council for Torture Victims in several countries. Assessment tools in the humanitarian eld tend to have unknown psychometric properties (i.e., indices of validity and reliability). Without published psychometric properties, it is unknown to what extent assessment tools are t for purpose. They developed a method and instrument to rapidly and quantitatively assess perceived needs in emergency-affected populationsthe Humanitarian Emergency Settings Perceived Needs (HESPER) Scale. They describe the development and psychometric properties of the scale. 9 We employed different sampling methods in the 3 sites according to what was appropriate and feasible. Phase 1 (2008): sampled general and psychosocial humanitarian experts across the world (24 men and 19 women), as well as 6 national aid workers in Sierra Leone. development of a draft scale through a process of item generation and item reduction, based first on a literature review and second on a survey with humanitarian experts. Phase 2 (2009): convenience sampling to recruit participants in the 3 7 Alternative sampling plan Sampling plan yielding representative sample 4 If sampling was stratified, was a useful stratification variable selected? If a consecutive sample was used, was the time period long enough to address seasonal or temporal variation? Stratification pilot sites, with interviewers identifying and selecting participants. The following were interviewed: 40 Iraqis displaced following the 2003 invasion of Iraq (interviewed in Amman, Jordan in June 2009), 40 members of the local population in Gaza City (October 2009), and 42 members of the local population in Juba, Sudan (December 2009). All participants were at least 18 years old. pilot-testing of the draft scalein Jordan with displaced Iraqi people, in Gaza and Sudan with the general adult population, and in the United Kingdom with refugees from the Democratic Republic of the Congoto assess the scale's feasibility, intelligibility, and cultural applicability, and to establish the suitability of training materials. Phase 3 (2010): larger samples in 3 relevant humanitarian settings to assess its psychometric properties and to estimate the level of perceived needs in these settings (here we focus on the scale's psychometric properties only). In total, 269 Iraqi participants displaced following the 2003 invasion of Iraq were interviewed in Jordan (Amman, Zarqa, Irbid, and Madaba) in July 2010, 279 people living in displacement camps following the January 2010 earthquake were interviewed in Haiti (Champs de Mars and Bolosse camps in Port-au-Prince and Pinchinat camp in Jacmel) in September 2010, and 269 Bhutanese refugees were interviewed in Nepal (Beldangi-II camp in Jhapa district) in October and November 2010. field-testing of the revised draft scalein Jordan with displaced Iraqi people, in Haiti with people living in postearthquake displacement camps, and in Nepal with Bhutanese refugeesto assess its psychometric properties (i.e., validity and reliability). employed different sampling methods in the 3 sites according to what was appropriate and feasible. Phase 2 No. Because the convience sample doesn't usually yield a representative sample. Haiti, we purposively selected 3 displacement camps as study sites to fit in with the implementing agency's programs. Within camps, we selected participants by using a 2-stage systematic random sampling method, the first stage being households and the second stage being individuals within households. 2 5 3 Consecutive sampling If cluster sampling was utilized, what were the clusters and how was sampling done within clusters? Clusters Sampling method within clusters 5 3 Iraqi participants in Jordan were recruited through a multistage cluster sampling design, involving 30 clusters of city districts. The sample was geographically representative of Iraqis living in Jordan, with around 75% of the sample in Amman (23 clusters) and around 25% (7 clusters) in other governorates (4 in Zarqa, 2 in Irbid, and 1 in Madaba). 5 Jordan and Haiti, we employed random-walk methods to recruit households within clusters or camps; we then randomly selected individuals within chosen households by using a random-number Kish How were people recruited into the sample? Does the method suggest potential biases? Recruitment Potential biases 5 3 3 6 7 8 Did some factor other than the sampling plan affect the representativeness of the sample? Are possible sample biases or weaknesses identified by the researchers themselves? Are key characteristics of the sample described (e. g., mean age, percent female)? If yes, please provide that information. 5 5 Male Total-305 (37.3) Jordan-116 (43.1) Haiti-50 (17.9) Nepal-139 (51.7) Female Total-512 (62.7) Jordan-153 (56.9) Haiti-229 (82.1) Nepal-130 (48.3) Age, y Total-37.09 13.5 Jordan-40.24 13.36 Haiti-34.22 12.31 Nepal-36.92 14.15 5 9 10 11 Was the sample size justified on the basis of a power analysis or other rationale? What approach do the authors use to establish criterion validity? What approach do the authors use to establish interrater reliability? The sample size was justified on the basis of a power analysis. The authors calculated for test-retest reliability on the basis of previous psychometric testing of the different CANSAS versions. This showed the required minimum sample size for test-retest reliability to be 69 per site to give power (1 - ) of 0.8, using a P value of .05, a minimum acceptable level of test-retest reliability (intraclass correlation coefficient) of 0.6, and a predicted test-retest reliability (intraclass correlation coefficient) of 0.7. This sample size also allowed the detection of correlations for criterion (concurrent) validity of at least r=0.3 with power (1 - ) of 99%, or r=0.2 with power (1 - ) of 83%. They performed a calculation for interrater reliability on the basis of findings made during previous pilot-testing of the HESPER Scale. This showed the required minimum sample size for interrater reliability to be 39 per site in order to give power of 0.8, using a P value of .05, a minimum acceptable level of interrater reliability of 0.7, and a predicted interrater reliability of 0.8. 5 They established criterion (concurrent) validity of the HESPER Scale by comparing 15 of its 26 individual need items, as well as the total number of unmet needs, to similar questions of an established quality-of-life instrument, the World Health Organization Quality of Life-100 (WHOQOL100), (77 participants in Jordan, 79 in Haiti, and 269 in Nepal). For the remaining 11 HESPER items, there was no comparable external criterion available. the total WHOQOL-100 score as was predicted before data collection (i.e., Pearson's correlation was within 1 order-ofmagnitude step of the predicted value, where 0.1-0.3 represented a low correlation, 0.3-0.5 represented a medium correlation, and 0.5- 1.0 represented a high correlation) in all 3 settings (r=-0.629 in Jordan, -0.417 in Haiti, and -0.469 in Nepal), as well as with the WHOQOL-100 question \"How would you rate your quality of life?\" (r =-0.501 in Jordan, -0.302 in Haiti, and -0.286 in Nepal). The authors use a second interviewer to measure the HESPER Scale's interrater reliability. The second interviewer acted as silent rater for 46 participants in Jordan, 44 in Haiti, and 42 in Nepal. Intraclass correlation coefficients (absolute agreement) for interrater reliability of total number of unmet needs were 0.998 in Jordan, 5 5 12 What approach do the authors use to establish testretest reliability? 0.986 in Haiti, and 0.995 in Nepal; across sites it was 0.998. Percentage agreements for interrater reliability of need ratings of individual HESPER items ranged between 95.3% and 100%, and Cohen's ranged between 0.66 and 1.0 across the 3 field-testing sites To assess test-retest reliability of the scale, 70 and 73 participants in Jordan and Nepal, respectively, were interviewed a second time 1 week after the first interview by the same interviewer who had interviewed them before. They did not assess test-retest reliability in Haiti as it was considered too burdensome for local people in this intense humanitarian setting Retest interviews were conducted between 6 and 8 days following the first interview in Jordan, and between 5 and 8 days later in Nepal; the means were 6.9 days (SD=0.3) and 6.5 days (SD=0.8), respectively. Intraclass correlation coefficients (absolute agreement) for test- retest reliability of total number of unmet needs were 0.961 in Jordan and 0.773 in Nepal; across the 2 sites it was 0.907. Percentage agreements for test-retest reliability of need ratings of individual HESPER items ranged between 66.7% and 100%, and Cohen's ranged between 0.07 and 1.0 across the 2 sites. Test-retest 5 reliability was not measured in Haiti, as it was not considered appropriate in this setting. 13 Does the sample support inferences about external validity? To whom can the study results reasonably be generalized? External validity Generalizability Source: Nursing Research: Generating and Assessing Evidence for Nursing Practice, 9th edition, p.289. 5 5 Assignment 3 Rubric & Answers - Semrau Question 1 Is the study population identified and described? Are eligibility criteria specified? Are the sample selection procedures clearly delineated? Study Population Eligibility criteria 2 3 What approach do the authors use to establish and content validity? What type of sampling plan was used? Would an alternative sampling plan have been preferable? Was the sampling plan one that could be expected to yield a representative sample? Sampling plan. Make sure to include the sampling plan for each of the different studies conducted. Answers Yes, the study population were identified and described. Eligibility criteria include the 23 medical-surgical units and 6 ICUs, all nurses from these units were considered. Patients who were in respiratory isolation were excluded. Patients and nurses from the hematology/oncology floor was also excluded because they were not scheduled to participate in the initial phase of the eMAR implementation. General adult population Phase 1- Possibl e Points 5 Individuals in humanitarian settings of conflict or other disasters in low- and middle income countries 5 The HESPER Scale was modeled on a mental health instrument, the Camberwell Assessment of Need Short Appraisal Schedule (CANSAS),19 which has well-established reliability and validity. The CANSAS has been modied successfully for numerous populations22---25 and adapted for use in several countries.26 It has been used on a wide range of populations, including asylum seekers and refugees in the United Kingdom27,28 and torture victims in centers of the International Rehabilitation Council for Torture Victims in several countries. Assessment tools in the humanitarian eld tend to have unknown psychometric properties (i.e., indices of validity and reliability). Without published psychometric properties, it is unknown to what extent assessment tools are t for purpose. They developed a method and instrument to rapidly and quantitatively assess perceived needs in emergency-affected populationsthe Humanitarian Emergency Settings Perceived Needs (HESPER) Scale. They describe the development and psychometric properties of the scale. 9 We employed different sampling methods in the 3 sites according to what was appropriate and feasible. Phase 1 (2008): sampled general and psychosocial humanitarian experts across the world (24 men and 19 women), as well as 6 national aid workers in Sierra Leone. development of a draft scale through a 7 Alternative sampling plan Sampling plan yielding representative sample 4 If sampling was stratified, was a useful stratification variable selected? If a consecutive sample was used, was the time period long enough to address seasonal or temporal variation? Stratification process of item generation and item reduction, based first on a literature review and second on a survey with humanitarian experts. Phase 2 (2009): convenience sampling to recruit participants in the 3 pilot sites, with interviewers identifying and selecting participants. The following were interviewed: 40 Iraqis displaced following the 2003 invasion of Iraq (interviewed in Amman, Jordan in June 2009), 40 members of the local population in Gaza City (October 2009), and 42 members of the local population in Juba, Sudan (December 2009). All participants were at least 18 years old. pilot-testing of the draft scalein Jordan with displaced Iraqi people, in Gaza and Sudan with the general adult population, and in the United Kingdom with refugees from the Democratic Republic of the Congoto assess the scale's feasibility, intelligibility, and cultural applicability, and to establish the suitability of training materials. Phase 3 (2010): larger samples in 3 relevant humanitarian settings to assess its psychometric properties and to estimate the level of perceived needs in these settings (here we focus on the scale's psychometric properties only). In total, 269 Iraqi participants displaced following the 2003 invasion of Iraq were interviewed in Jordan (Amman, Zarqa, Irbid, and Madaba) in July 2010, 279 people living in displacement camps following the January 2010 earthquake were interviewed in Haiti (Champs de Mars and Bolosse camps in Port-au-Prince and Pinchinat camp in Jacmel) in September 2010, and 269 Bhutanese refugees were interviewed in Nepal (Beldangi-II camp in Jhapa district) in October and November 2010. field-testing of the revised draft scalein Jordan with displaced Iraqi people, in Haiti with people living in postearthquake displacement camps, and in Nepal with Bhutanese refugeesto assess its psychometric properties (i.e., validity and reliability). employed different sampling methods in the 3 sites according to what was appropriate and feasible. Phase 2 No. Because the convience sample doesn't usually yield a representative sample. ? N/A. Sampling was not stratified. Full-time, part-time and per diem nurses who worked at the selected units were given an opportunity to volunteer and participate in the study. Haiti, we purposively selected 3 displacement camps as study sites to fit in with the implementing agency's programs. Within camps, we 2 5 3 selected participants by using a 2-stage systematic random sampling method, the first stage being households and the second stage being individuals within households. Consecutive sampling If cluster sampling was utilized, what were the clusters and how was sampling done within clusters? Clusters Sampling method within clusters 5 How were people recruited into the sample? Does the method suggest potential biases? 3 Iraqi participants in Jordan were recruited through a multistage cluster sampling design, involving 30 clusters of city districts. The sample was geographically representative of Iraqis living in Jordan, with around 75% of the sample in Amman (23 clusters) and around 25% (7 clusters) in other governorates (4 in Zarqa, 2 in Irbid, and 1 in Madaba). 5 Jordan and Haiti, we employed random-walk methods to recruit households within clusters or camps; we then randomly selected individuals within chosen households by using a random-number Kish Participants were recruited based on voluntary participation. From the nurses who volunteered, one nurse would be selected at random for the observation 5 3 Potential biases 6 7 8 . I could not see a potential bias from this method because the nurses would have volunteered themselves if they wished to participate. Did some factor other than the sampling plan affect the representativeness of the sample? Yes, I think other factors should be considered that might affect the representativeness of the sample. (i.e. experience, education, etc.) Are possible sample biases or weaknesses identified by the researchers themselves? Yes, the researchers felt that because the study was only performed at this one institution, it may not be representative of other community hospitals. Are key characteristics of the sample described (e. g., mean age, percent female)? If yes, please provide that information. Male Total-305 (37.3) Jordan-116 (43.1) Haiti-50 (17.9) Nepal-139 (51.7) Female Total-512 (62.7) Jordan-153 (56.9) Haiti-229 (82.1) Nepal-130 (48.3) 3 5 5 5 9 10 Was the sample size justified on the basis of a power analysis or other rationale? What approach do the authors use to establish criterion validity? Age, y Total-37.09 13.5 Jordan-40.24 13.36 Haiti-34.22 12.31 Nepal-36.92 14.15 The sample size was justified on the basis of a power analysis. The authors calculated for test-retest reliability on the basis of previous psychometric testing of the different CANSAS versions. This showed the required minimum sample size for test-retest reliability to be 69 per site to give power (1 - ) of 0.8, using a P value of .05, a minimum acceptable level of test-retest reliability (intraclass correlation coefficient) of 0.6, and a predicted test-retest reliability (intraclass correlation coefficient) of 0.7. This sample size also allowed the detection of correlations for criterion (concurrent) validity of at least r=0.3 with power (1 - ) of 99%, or r=0.2 with power (1 - ) of 83%. They performed a calculation for interrater reliability on the basis of findings made during previous pilot-testing of the HESPER Scale. This showed the required minimum sample size for interrater reliability to be 39 per site in order to give power of 0.8, using a P value of .05, a minimum acceptable level of interrater reliability of 0.7, and a predicted interrater reliability of 0.8. 5 They established criterion (concurrent) validity of the HESPER Scale by comparing 15 of its 26 individual need items, as well as the total number of unmet needs, to similar questions of an established quality-of-life instrument, the World Health Organization Quality of Life-100 (WHOQOL100), (77 participants in Jordan, 79 in Haiti, and 269 in Nepal). For the remaining 11 HESPER items, there was no comparable external criterion available. the total WHOQOL-100 score as was predicted before data collection (i.e., Pearson's correlation was within 1 order-ofmagnitude step of the predicted value, where 0.1-0.3 represented a low correlation, 0.3-0.5 represented a medium correlation, and 0.5- 1.0 represented a high correlation) in all 3 settings (r=-0.629 in Jordan, -0.417 in Haiti, and -0.469 in Nepal), as well as with the WHOQOL-100 question \"How would you rate your quality of life?\" (r =-0.501 in Jordan, -0.302 in Haiti, and -0.286 in Nepal). 5 11 12 What approach do the authors use to establish interrater reliability? What approach do the authors use to establish testretest reliability? The authors use a second interviewer to measure the HESPER Scale's interrater reliability. The second interviewer acted as silent rater for 46 participants in Jordan, 44 in Haiti, and 42 in Nepal. Intraclass correlation coefficients (absolute agreement) for interrater reliability of total number of unmet needs were 0.998 in Jordan, 0.986 in Haiti, and 0.995 in Nepal; across sites it was 0.998. Percentage agreements for interrater reliability of need ratings of individual HESPER items ranged between 95.3% and 100%, and Cohen's ranged between 0.66 and 1.0 across the 3 field-testing sites To assess test-retest reliability of the scale, 70 and 73 participants in Jordan and Nepal, respectively, were interviewed a second time 1 week after the first interview by the same interviewer who had interviewed them before. They did not assess test-retest reliability in Haiti as it was considered too burdensome for local people in this intense humanitarian setting Retest interviews were conducted between 6 and 8 days following the first interview in Jordan, and between 5 and 8 days later in Nepal; the means were 6.9 days (SD=0.3) and 6.5 days (SD=0.8), respectively. Intraclass correlation coefficients (absolute agreement) for test- retest reliability of total number of unmet needs were 0.961 in Jordan and 0.773 in Nepal; across the 2 sites it was 0.907. Percentage agreements for test-retest reliability of need ratings of individual HESPER items ranged between 66.7% and 100%, and Cohen's ranged between 0.07 and 1.0 across the 2 sites. Test-retest 5 5 reliability was not measured in Haiti, as it was not considered appropriate in this setting. 13 Does the sample support inferences about external validity? To whom can the study results reasonably be generalized? Yes, I believe that the sample may have the potential to support inference about external validity. The study results can be generalized to other hospitals of similar capacity, with patients of similar comorbidities, and nurses of the same level of education and expertise. External validity Yes, I believe that the sample may have the potential to support inference about external validity. The study results can be generalized to other hospitals of similar capacity, with patients of similar comorbidities, and nurses of the same level of education and expertise. Generalizability 5 5 Source: Nursing Research: Generating and Assessing Evidence for Nursing Practice, 9th edition, p.289