{"id":126,"date":"2017-03-31T10:46:42","date_gmt":"2017-03-31T14:46:42","guid":{"rendered":"https:\/\/sites.bu.edu\/mahmaz\/?page_id=126"},"modified":"2018-04-10T16:31:45","modified_gmt":"2018-04-10T20:31:45","slug":"process-evaluation","status":"publish","type":"page","link":"https:\/\/sites.bu.edu\/mahmaz\/project\/process-evaluation\/","title":{"rendered":"Process Evaluation"},"content":{"rendered":"
\n
\n
\n
\n
\n
\n

Objectives and Evaluation Questions <\/span><\/h1>\n<\/div>\n<\/div>\n<\/div>\n

The overall aim of the Process Evaluation\u00a0is to understand key aspects of the implementation of the MAHMAZ interventions and how these components relate to the effectiveness of the overall MAHMAZ program. Specific objectives of this implementation evaluation are fivefold:<\/p>\n

Objective 1: To assess the degree to which each individual intervention was effectively implemented according to the project plan and to document adjustments made during the course of the project. <\/span><\/h3>\n

Evaluation questions: <\/span><\/h4>\n
    \n
  1. At intervention sites, what is the proportion of MS that meet each component of the MCM? <\/span><\/li>\n
  2. What are the barriers and facilitators to implementation of the MCM management models? <\/span><\/li>\n
  3. What proportion of sites (both intervention and control) have functioning governance and <\/span>management structures? <\/span><\/li>\n
  4. What proportion of intervention sites are operating according to the project SOPs? <\/span><\/li>\n
  5. At each of the interventions sites, to what extent are the proposed secondary interventions <\/span>(SDPG, IGA, skills training) being implemented, and how does implementation differ across <\/span>sites? <\/span><\/li>\n
  6. What are the barriers and facilitators to implementation of secondary interventions? <\/span><\/li>\n<\/ol>\n

    Objective 2: To assess the extent to which implementation of MCMSM was perceived as responsive to community standards of acceptability. <\/span><\/h3>\n

    Evaluation questions:<\/span><\/h4>\n<\/div>\n<\/div>\n
    \n
    \n

    <\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n

    \n
    \n
    \n
      \n
    1. To what degree are the intervention MS perceived as responsive to community standards and needs? <\/span><\/li>\n
    2. How does satisfaction with staying at the MS compare in intervention sites relative to control sites? <\/span><\/li>\n
    3. What are the essential features and characteristic of the MS as perceived by both women and communities? <\/span><\/li>\n
    4. What are continued barriers to accessing and utilizing MS after the intervention? <\/span><\/li>\n<\/ol>\n

      Objective 3: To assess the ways in which the package of interventions changed service utilization and maternal and neonatal outcomes at both BEmONC and CEmONC sites. <\/span><\/h3>\n

      Evaluation questions: <\/span><\/h4>\n
        \n
      1. What operational and financial systems are in place and functional at baseline at existent MS at CEmONC facilities? <\/span><\/li>\n
      2. How does the utilization of the MS at CEmONC facilities change over time with implementation of the MCMSM model at BEmONC facilities? <\/span><\/li>\n
      3. How do referral patterns, census and demographic of those utilizing BEmONC and CEmONC change over time? <\/span><\/li>\n
      4. How do maternal and neonatal outcomes change over time at both BEmONC and CEmONC facilities, across intervention and control sites? <\/span><\/li>\n
      5. How does the cost per outcome (maternal and neonatal outcomes) of the MS change over time? <\/span><\/li>\n<\/ol>\n

        Objective 4: To identify key features of both the entrepreneurial strategies and managements models developed to support the finances and operations of MS and to assess their individual and collective effectiveness. <\/span><\/h3>\n

        Evaluation questions: <\/span><\/h4>\n
          \n
        1. What is the contribution of each strategy in the overall <\/span>financial <\/span>sustainability of the MS, and how does this change over time? <\/span><\/li>\n
        2. What is the contribution of the management models in the overall <\/span>financial <\/span>sustainability of the MS, and how does this change over time? <\/span><\/li>\n
        3. What is the contribution of each strategy in the overall <\/span>operational <\/span>sustainability of the MS, and how does this change over time? <\/span><\/li>\n
        4. What is the contribution of the management models in the overall <\/span>operational <\/span>sustainability of the MS, and how does this change over time? <\/span><\/li>\n
        5. Which strategies and managements models are perceived as viable for long-term sustainability? What differences exist between communities? <\/span><\/li>\n
        6. What is the effect of each strategy\/management model on the utilization of the MS? <\/span><\/li>\n<\/ol>\n

          Objective 5: To generate a set of recommendations for GRZ to further adapt and\/or scale up of MS based on the summative findings from this implementation evaluation and the intervention effectiveness evaluation <\/span><\/h3>\n

          Evaluation questions: <\/span><\/h4>\n
            \n
          1. Are mother\u2019s shelters an effective solution to the distance problem in rural Zambia? <\/span><\/li>\n
          2. What are <\/span>the \u2018key ingredients\u2019 for successful implementation of the <\/span>MS? <\/span><\/li>\n
          3. Of the \u2018key ingredients\u2019 identified, which are primarily related to the changes in outcomes <\/span>of the target population? <\/span><\/li>\n
          4. What elements of the intervention are essential to sustainability of MS?<\/span><\/li>\n
          5. What is the annual cost of implementing a sustainable MS model?<\/li>\n<\/ol>\n<\/div>\n<\/div>\n
            \n
            \n
            \n

             <\/p>\n

            O<\/span>VERVIEW OF <\/span>S<\/span>TUDY <\/span>D<\/span>ESIGN AND <\/span>M<\/span>ETHODS<\/span><\/h1>\n

            In order to achieve the objectives, we will use a mixed-methods approach and collect longitudinal, cross-sectional data at both the MAHMAZ intervention BEmONC sites (n=10) and a set of matched control sites (n=10) that will not have any interventions implemented, before and during MAHMAZ project implementation. We will also collect data at 5 CEmONC sites, 2 of which will have an MS constructed and 3 of which will have no intervention. This approach has several strengths. First, it allows for triangulating findings from various data sources and methods. Second, the use of a longitudinal time-series study design with a control group allows us to differentiate the challenges or added burden posed by the core model implementation from those that are implementing the standard of care and allows for limiting threats to internal validity of history and maturation. Third, by systematically evaluating the implementation and outcome of each element of the intervention, we can <\/span>isolate \u2018key ingredients\u2019 for effective implementation <\/span>and relate it to both patient and facility- level outcomes. <\/span><\/p>\n

            Objective 1 <\/strong><\/span>will be achieved by analyzing routinely collected project data, including monthly <\/span>data from the Mothers\u2019 Shelter Experience Surveys (a sample of women (up to 6 per site per month) <\/span>from each RHC study site, both intervention and control, available on day of data collection who have stayed at an MS or other facility-affiliated lodging\/structure at least 3 consecutive nights), and semi- annual in-depth interviews (IDIs) with both governance committee members and management unit members at intervention sites only (see Annex J for interview guide). As part of the project, at each <\/span>site an MS Register, Others\u2019 Register (for <\/span>women utilizing the MS after 8 days postpartum or travelers, relatives of patients who are non-maternal cases) and MS Activity Log will be maintained. On a monthly basis, MAHMAZ Monitoring and Evaluation (M&E) study staff will extract data from these registers\/log using corresponding data extraction tools (Annexes G, H, and I). The project team will also administer the Core Model Checklist (Annex D) at BEmONC sites and CEmONC intervention sites. The quantitative survey and Core Model Checklist data will be triangulated with the IDIs at each intervention site. <\/span><\/p>\n

            Objective 2 <\/strong><\/span>will be achieved by analyzing data collected monthly from the <\/span>Mothers\u2019 Shelter <\/span>Experience Surveys (see above), semi-annual IDI with governance committee members and with management unit members, and annual focus group discussions (FGDs) among recently-delivered women (defined as women who have delivered in the past 12 months), men with children under the age of 1, community elders\/mothers-in-law of RDW and members of Safe Motherhood Action Groups (SMAGs) at each site (see Annex K for FGD Guide). Data will be analyzed against the community-defined standards of acceptability from the formative evaluation (ERES Protocol#2013- AUG-008\/BU IRB Protocol H-32470) and other standards that emerge through ongoing analysis of all qualitative data. Stratified by intervention and control site, data will be triangulated with quantitative data from objective one to understand acceptability by implementation fidelity of the core model, and findings from the impact evaluation (ERES protocol #2015-Dec-012\/BU IRB Protocol H-34526) in MAHMAZ sites only. <\/span><\/p>\n

            \u00a0Objective 3<\/strong> <\/span>will be achieved by using routinely collected health facility data at all 20 BEmONC sites (intervention and control) and at 5 CEmONC sites in the catchment areas beginning six months prior to implementation of the intervention. On a monthly basis, study staff will extract key variables about delivery and neonatal outcomes, referrals and postnatal care from the following BEmONC and CEmONC facility-based data: Delivery Register, Maternity Ward Admission Register, Postnatal Care Register, Facility Transfer Log, Ante-natal care register (where applicable) and other improvised registers as necessary. The data extraction tools can be seen in Annexes A-C. Secondly, using the Safe Motherhood Number (SMN), a unique code that a woman receives when visiting government facilities during her pregnancy, each woman will be tracked through <\/span>the \u201ccare cascade\u201d, meaning\u00a0<\/span>throughout <\/span>mothers\u2019 shelter stay, <\/span>delivery and post-natal visits, to understand both utilization of MS and health facility services and maternal and neonatal health outcomes. The SMN will also allow tracking during referral from BEmONC to CEmONC sites and to understand patterns of bypassing (i.e. women going directly to CEmONC facilities). We will also compare this information with data captured in Objective 1. Additionally, on a monthly basis we will assess the capacity of the health facility to manage obstetric complications using a health facility assessment tool (Annex I). We will conduct semi-annual IDI with health facility staff and health systems staff (Annex J) and extract MS and health facility financial records (as they relate to MS) to understand the cost-outcomes of the MS.<\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n

            \n
            \n
            \n

            Objective 4 <\/strong><\/span>will be achieved through use of routinely collected monitoring data from MS registers, as well as program data from MS activity logs, training attendance logs, production registers, in-kind donation logs, as well as MS and health facility financial records (as they relate to MS). For each entrepreneurial strategy we will systematically capture the costs of implementation, the change in revenue over time, and the proportion of MS operating costs that each strategy covers using MS costing tools. Additionally, program data will capture decisions made and changes in operating costs. These data will be analyzed to understand what mix of operating costs and revenue streams create a financially sustainable MS. We will also use data from the semi-annual IDIs with governance committee members and management unit members and FGDs among RDW, men with children under the age of 1, elders\/community members and SMAGs (see Objectives 1, 2 and 3). <\/span><\/p>\n

            Objective 5 <\/strong><\/span>will be achieved by synthesizing findings from this implementation effectiveness evaluation and the intervention effectiveness evaluation, guided by the CFIR and situating it all amongst current peer-reviewed literature. <\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

            Objectives and Evaluation Questions The overall aim of the Process Evaluation\u00a0is to understand key aspects of the implementation of the MAHMAZ interventions and how these components relate to the effectiveness of the overall MAHMAZ program. Specific objectives of this implementation evaluation are fivefold: Objective 1: To assess the degree to which each individual intervention was […]<\/p>\n","protected":false},"author":12998,"featured_media":0,"parent":50,"menu_order":4,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/pages\/126"}],"collection":[{"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/users\/12998"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/comments?post=126"}],"version-history":[{"count":5,"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/pages\/126\/revisions"}],"predecessor-version":[{"id":249,"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/pages\/126\/revisions\/249"}],"up":[{"embeddable":true,"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/pages\/50"}],"wp:attachment":[{"href":"https:\/\/sites.bu.edu\/mahmaz\/wp-json\/wp\/v2\/media?parent=126"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}