Note: The associated files are best viewed using Adobe Reader. To get the software, go to http://get.adobe.com/reader/

1. Language Access Program in OAPIA: Assessing LAS 2009 and Making Recommendations for LAS 2011. This report evaluates OAPIA's Language Access Survey (LAS) of 2009 and makes recommendations for the upcoming 2011 survey. This report found that the LAS successfully identified evaluation questions, targeted the information source, conducted the data collection, and implemented data quality assurance. The LAS, however, did have some weaknesses: it had a small sample size, did not filter non-residents of the District of Columbia from the sample, failed to collect important demographic information including respondents' educational background and language ability, and neglected crucial outcome information such as feedback on language services.

The author proposes the following recommendations to the 2011 LAS survey: (1) Increase the sample size to at least five hundred with at least one hundred for each of the five key ethnic groups including: Chinese Americans, Japanese Americans, Indian Americans, Korean Americans and Vietnamese Americans; (2) use a face-to-face format to probe for more information and collect anecdotes; (3) add survey questions to collect information on respondents' residency status, educational attainment, self-reported language ability, quality of language services provided by the District agencies, motivation to actively seek the language service, and one open question for any comments; (4) conduct a pretest; (5) assign a reference number to each survey to check data entries; and (6) disseminate the results of the survey.

Paper   Presentation

2. Recommended Performance Metrics for the Montgomery County Department of Economic Development. This study examines the existing performance measures of the Montgomery County Department of Economic Development and proposes new performance measures due to a restructuring of DED's mission and goals. It also proposes "comparison counties" to be used in benchmarking DED's performance. Performance measures were created for each program area in the new structure: Prospect Development, Problem Resolution, Technical Assistance, Agricultural Services and Workforce Services.

Comparison counties were determined through a similarity score matching system in which the criteria were structurally similar to Montgomery County's DED and services offered. The counties evaluated were the thirty-five that CountyStat has identified as demographically similar and are used to benchmark all other departments. The best matches scored above fifty percent on a system with eleven criteria and above fifty percent on a system where those eleven criteria were narrowed to the most important seven. They are: Howard County, MD; Anne Arundel County, MD; Fairfax County, VA; Westchester County, NY; Monmouth County, NJ; Bucks County, PA; and Oakland County, MN. The good matches scored above fifty percent on only one scale. They are: Douglas County, CO; Chester County, PA; Lake County, IL; and Hamilton County, IN.

Paper   Presentation

3. The Healthy Montgomery Qualitative Data Collection Process: Successes and Challenges in Implementation. This report analyzes the qualitative data collection process of the Healthy Montgomery community health improvement process. The author conducted an implementation evaluation, analyzing four community conversations using a series of criteria developed by the author and approved by the staff of the Montgomery County Department of Health and Human Services. The evaluation details developments in the design and methodology of the qualitative data collection process, successes and challenges in its implementation and stakeholder evaluations of Healthy Montgomery's progress.

The author proposes the following recommendations to be implemented in future iterations of Healthy Montgomery: (1) Establish more consistency in the methodology, (2) target underrepresented populations to a greater degree, and (3) explore additional methods of direct outreach.

Paper   Presentation

4. Performance Measures for the 2011 Montgomery County Community Service Week Survey Tool. This report develops performance measures for the Montgomery County Volunteer Center to capture the outputs and outcomes of volunteerism in the County. This survey tool will help quantify the contribution which the County's volunteers make and the results can be used to identify areas of improvement, to study volunteerism outcomes, and to apply for grants.

Performance measures will be collected through two surveys to be administered during the Community Service Week (October 16, 2011 to October 22, 2011). One of these surveys will be for the organizations which provide the volunteer activity and the other will be for the volunteers themselves. The complete surveys are included in Appendix A and Appendix B of the report.

Paper   Presentation

5. Performance Measures for the Silver Spring Civic Building and Veterans Plaza. This report develops performance measures for the Silver Spring Civic Building and Veterans Plaza to evaluate the Civic Building and Plaza's effectiveness in accomplishing their mission.

The program theory of the Civic Building and Plaza can be described as, if the space can effectively be activated by a variety of stakeholders including the county and the community, then the Civic Building and Plaza can improve community cohesion through community engagement, act as a catalyst for local economic development, and be a sustainable revenue generator. The recommended outputs are divided into three categories � operations and management, financial operations, and community involvement. Additionally, improvements in four outcome categories are the best way to evaluate the Civic Building and Plaza: "Access and Linkages"; "Comfort and Image"; "Uses and Activities", and "Sociability." All of these recommendations have been proposed within the context of a tight budget environment in which resources for data collection are limited. Finally, the report also recommends creating partnerships to collect additional outcome data.

Paper   Presentation

6. Lessons Learned for the Future: Money Follows the Person Demonstration. This report develops a template for states that are looking to implement a Money Follows the Person (MFP) initiative for their Medicaid programs. It surveys trends in existing MFP demonstrations to determine which program characteristics are likely to produce successful MFP initiatives, and to recommend guiding principles for incoming MFP states. By understanding the experiences of existing MFP demonstrations, new states can better optimize how their MFP inputs and activities can lead to positive program outcomes.

More generally, the research suggests nine guiding principles to new states implementing MFP:

• Conduct a comprehensive assessment of the state Medicaid long-term care system
• Invest early in infrastructure that facilitates rebalancing initiatives
• Build off existing supports and services
• Secure buy-in from key stakeholders
• Utilize contractors to help implement MFP initiatives
• Secure set-aside services within existing programs for MFP clients
• Manage early expectations
• Build flexibility in MFP initiatives to allow for continuous improvement
• Establish policies that reallocate resources from institutions to community-based care

Paper   Presentation

7. Assessing the Implementation of Quality Indicators in the Medicare Physician Group Practice Demonstration. This report is an assessment of the implementation of quality indicators in the Medicare Physician Group Practice demonstration. It focuses on two research questions: (1) How effectively did demonstration sites implement interventions to satisfy quality indicators?; and (2) How can the lessons learned from the PGP Demonstration be useful in implementing the Medicare Shared Savings Program (MSSP)?

Sites experienced barriers to implementation in a variety of forms--financial, geographical, administrative, and in data collection and reporting. As demonstrated in the quality performance results, sites are achieving their quality thresholds for the majority of measures. However, all sites are not earning shared saving payments. The report recommends that future quality measures need to be easily captured by electronic reporting systems and create meaning for providers and patients. Streamlining quality measures across Medicare and private payers will yield an effective national benchmark that can improve and coordinate future quality reporting. Physician buy-in is critical to the success of implementing outcomes-based quality measures and improving the coordination of care.

Paper   Presentation

8. Creating Greater Accountability and Organization in Applications and Reports for CSBG Grants. This report is a process evaluation of the funding opportunity announcements, applications, and semi-annual reports for nine grantees that received two Community Services Block Grant (CSBG) Training and Technical Assistance (T/TA) Grants to determine if there were inappropriate overlaps in funding or deviations from original plans without clear explanations. In examining the reporting documents, it became clear that the lack of organization and detail within the documents made it time-consuming and sometimes impossible to tell what activities were funded, how many community actions agencies benefitted, and the amount of overlap between the grants and between periods.

This report proposes a model set-up for applications and semi-annual reports that includes a categorization system that could help CSBG Program Specialists identify the types of activities their grants fund and enable them to identify problems more quickly. In addition, it provides some recommendations for the State Associations on the details they should include in their applications and reports.

Paper   Presentation

9. Electronic Health Record Incentive Program: Assessing Potential Cost Savings. This report attempts to quantify the potential cost savings as a result of the Electronic Health Records Incentive Program (EHR) between 2011-2019. The analysis looks only at practice-based physicians treating Medicare patients, even though incentive payments are available also for Medicaid providers and hospitals. In particular this paper attempts to answer the following questions: (1) What is the first year in which potential cost savings as a result of EHRs equal costs?; (2) Looking at the whole 2011-2019 timeframe, when will the cumulative savings equal the cumulative costs for the program?; and (3) For every dollar spent in incentive payments, how much savings will CMS see in return by 2019?

The report finds, using a 4.5 percent reduction in costs assumption, the first year in which savings equal the costs will be in 2014. Using a more conservative 3.0 percent savings assumption, savings will equal costs approximately three months later into 2014. Looking at cumulative costs and savings for the program, the break-even point between costs and savings occurs in 2016, assuming a 4.5 percent savings from EHRs. The more conservative 3.0 percent savings assumption gives us a break-even point closer to 2017. A cursory analysis of the savings assumptions indicated that a savings rate of 1.03 percent would be necessary in order to have savings equal costs at all by 2019.

Paper   Presentation

10. The Financing of Care for Dual Eligible Beneficiaries: An Overview of Care for Dual Eligibles and a Cost-Comparison Analysis of Two Service Delivery Models for the Centers for Medicare and Medicaid Services. This report examines the Medicaid costs for dual eligible beneficiaries enrolled in two of the service delivery models involving community-based care: HCBS waivers and the PACE program. The specific HCBS waiver examined is the HCBS-Disabled/Elderly (HCBS-DE) waiver commonly used by dual eligibles. The cost-comparison analysis sought to identify which service delivery model, HCBS-DE waivers or PACE, provided lower cost care for dual eligible beneficiaries 65 and older.

Using data from the Medicaid Statistical Information System (MSIS), this analysis compared the reported Medicaid data by state for (1) the number of beneficiaries, (2) total payment amount, and (3) payment per beneficiary. This study found that Medicaid costs were higher for dual eligibles 65 and above who were enrolled in a PACE program when compared to those using HCBS-DE waivers.

Paper   Presentation

11. Managing for Performance: Child Welfare Waiver Program and Cost-Effectiveness Analysis. This report seeks to evaluate demonstrations of waivers of certain provisions of Title IV-E of the Social Security Act related to foster care by focusing on the cost studies, typically the weakest portion of these evaluations. The three research questions are: (1) How can cost-effectiveness analysis improve child welfare waiver evaluations? (2) How can cost-effectiveness analysis be conducted in child welfare waiver evaluations? and (3) How can the performance management of the waiver program otherwise be improved?

Recommendations include that HHS require the use of cost-effectiveness analysis and experimental design. HHS should also encourage the use of similar program designs across States.

Paper   Presentation

12. Runaway and Homeless Youth Cost Study: Estimating the Impact of a Unified Definition of Homelessness Among U.S. Departments. This report addresses the question: "How much would it cost the Runaway and Homeless Youth (RHY) Program to alter its definition of "homeless" to the definition used by HUD or the definition used by Education?" Current program per youth expenditures and new eligibility estimates outlined by the alternate definitions are used to assess the program costs given the use of different homeless classifications.

The author analyzes these calculated costs under various program take-up rates, as well as diverse regional conditions. By illustrating the additional costs caused by accommodating a common vocabulary of homelessness and outlining the vast differences between departmental definitions, the report demonstrates the need for diverse program guidelines and illustrates their influence in shaping program goals and intentions.

Paper   Presentation

13. Statewide Longitudinal Data Systems in Education: Enabling Data-Based Decision Making in Florida, North Carolina, and Washington. This report is a process evaluation of the development of statewide longitudinal data systems in three states: Florida, North Carolina, and Washington. It finds that certain factors facilitate data system development and implementation. State-level political support encourages and aids the development of the states' data systems. Legislative mandates ensure the data systems moved from the development to the implementation stage. The availability of funding is a critical component; longitudinal data systems including data that encompasses all aspects of public education require funding for the system development, implementation, and ongoing staff, monitoring, and enhancements. Further, the data systems cannot be constructed in a day or even over the course of a year. The three states collect data from pre-existing data systems and used their past experiences to inform current decisions.

At the same time, some factors worked to hinder data system's usability. Inadequate staffing results in long wait times in receiving data requests. The analytical tools that could help relieve staff and allow stakeholders to access data themselves are not developed to their fullest extent. The data system stakeholders expect the systems to be developed as planned, including these enhanced capabilities. Managing multiple expectations and needs has proven difficult. Collaborating with the source data systems is a challenge, as well.

The author recommends the following to improve state's transition towards a data-based environment that is used to improve educational instruction: (1) Set priorities using the Statewide Longitudinal Data System Grant's Request for Application (RFA); (2) Mandate, rather than encourage, the Common Education Data Standards; (3) Encourage and support data system training at the state, district and school-levels; and (4) Provide guidance on the potential uses of longitudinal data.

Paper   Presentation

14. Identifying Potential Areas of Overlap in Federal Education Research. This report examines potential duplication within federal education research, looking at two major questions: 1) What types of education-related research do federal agencies and academies sponsor and conduct?; and 2) Do their research areas overlap?

A thorough study of these publications reveals no duplication in federal education research within the temporal and topical scope of the evaluation. Although agencies and academies support research on similar topics, studies within the same category make use of different types of methodology and include different subject matter. Despite this lack of duplication, agencies should endeavor to leverage resources between agencies interested in studying similar issues in education in order to maximize their efficiency and produce data that are relevant to multiple agencies and academies. Future research in this area might also include cost effectiveness research to increase government efficiency and identify which types of research are ultimately the most useful to policymakers and the public.

Paper   Presentation

15. Process Evaluation of Supplemental Educational Services. This paper documents a process evaluation of Supplemental Educational Services (SES) under No Child Left Behind. Using a case study method, 12 SES coordinators at the state and local level were interviewed. States and districts identified four primary challenges to SES implementation, including difficulty evaluating the effectiveness of SES providers, trouble removing providers from state approved lists, variation in provider rates, and the lack of alignment between tutoring services and school day curriculum.

Based on the findings of this evaluation, SES would benefit from a shift in focus from provider effectiveness to assessing provider quality. This would help improve the quality of tutoring services and make the provider removal process more objective and consistent. Districts would benefit from having greater control over the program's design; as the day-to-day operators of this program, LEAs should have a greater say in establishing provider rates, approving curriculum and removing providers from state approved lists.

Paper   Presentation

16. Health Reform: Implementing Health Insurance Exchanges. This report provides a timeline for state implementation of health insurance exchanges under the Patient Protection and Affordable Care Act (PPACA), detailing necessary state action to meet the 2014 deadline. This report serves only as a guideline for implementation, and states will need to consider their specific needs in determining how to approach mplementation. Considering the complexity of establishing exchanges, states should move forward with implementation immediately, remain flexible and ensure continued coordination with key stakeholders.

Paper   Presentation

17. Assessing the Impact of the Folger Shakespeare Library's Shakespeare Set Free Teaching Training Programs. This report uses a process evaluation to examine the Folger Shakespeare Library's Shakespeare Set Free program, a series of educational workshops offered by the Folger Shakespeare Library that teaches an alternative way of teaching and learning Shakespeare known as performance-based teaching. It aims to answer the following questions: (1) Is the program accomplishing what it expects to accomplish? and (2) Does the Shakespeare Set Free teacher training program change the way teachers teach and students learn Shakespeare?

A process evaluation was conducted in Providence, RI which included interviews with key stakeholders, observations of workshops in action, a review of the supplemental materials available to participants, and participant responses to a survey about experiences using the techniques in the classroom. The evaluation found that the Shakespeare Set Free program is working effectively towards achieving its desired objective.

To ameliorate the problem of lack of data, this report suggests conducting an impact evaluation, which would compare student test scores between a treatment group, whose teachers took the Shakespeare Set Free workshops, and a control group, whose teachers continued using the traditional methods.

Paper   Presentation