Developing a model process for collaborative evaluation of major programs

The Florida Cooperative Extension Service must increasingly respond to questions from stakeholders about the impact of its educational programs.  Given the need for relevant and credible impact information, county and state Extension faculty encounter the dilemma of having limited resources to obtain scientifically rigorous evaluation information.

Objective:  To develop a process to efficiently obtain credible, relevant evaluation information  about the impact of state major programs for use in accountability reports.

Approach: The Environmental Landscape Management (ELM) program was selected to pilot test a collaborative evaluation model.   A teamwork approach was used to coordinate the myriad of ELM activities by county faculty and state specialists.  The approach encouraged participants to contribute based on his/her expertise and responsibilities (e.g., state faculty provided leadership on the evaluation design, instrumentation, and data analysis and county faculty provided leadership in instrument review, data collection and data entry). I focused on developing the evaluation model and conducting educational programs to facilitate adoption.

• Developing the evaluation model

• I conducted the following activities: 1) Developed an evaluation plan for the ELM program with input from the design team and several county faculty.  The evaluation assesses both program impact and areas for improvement by using a non-equivalent control group design with pre- and post-tests; 2) Developed questionnaires to obtain information about landscaping practices.  The questionnaires were revised twice after obtaining input from county faculty.   Faculty in over 40 counties have received the latest version of the questionnaire and faculty from 21 counties are using the ELM evaluation materials; 3) Developed database templates for use by county Extension faculty to create data files for each of the pre-program and follow-up questionnaires; 4) Developed publications to support the planning and implementation of the evaluation and developed questionnaires for use with the general public and completed data collection of a sample for a control group; 5) Developed an Extension bulletin to model how evaluation results could be reported; and 6) During this last year, I have continued to analyzed data for county faculty who have collected pre-program and follow-up surveys.  The faculty use the results in their annual reports; this data is also aggregated for use in the state major program report.  For example, Figure 1 shows results from 1997 ELM programs. 

 
• Conducting educational programs to facilitate adoption by Extension faculty

• Efforts to increase awareness and adoption of the collaborative evaluation process within the Florida Extension service and in other states included:

 
• The ELM evaluation process was highlighted during design team orientation meetings held in summer, 1995.  Over 130 Extension faculty learned about the ELM model;

• Information about the evaluation methodology has been shared with evaluators in other states to use as a model through conference presentations and publication of proceedings of Extension Education Evaluation Topical Interest Group of the American Evaluation Association;
• The ELM collaborative process was highlighted in an invited presentation at the meeting on Education and Communication Applications in Natural Resource Management; and

• ELM bulletins have been distributed widely in the state and to other southern states.  Conversations with specialists in Maryland and New Jersey indicate that those states will adopt the ELM evaluation questionnaires.

Identifying, assembling, analyzing and disseminating data for program planning

In 1990, a long-range planning effort (Challenge ‘95) was begun in order to guide Extension programs during the 1992-1995 period.  One barrier to developing plans for Extension programs was that county faculty and subject-matter specialists made limited use of standardized sources of information to analyze customer needs and audience segments for Extension programs.  The focus of Extension programs can be sharpened to better meet customer needs by using appropriate data to develop situational statements and objectives in plans of work.  The need for relevant, current information continued with a second long-range planning effort, Florida 2000 and Beyond, started in 1995.

The objectives were 1) to identify, assemble, and analyze data relevant to program planning and provide standardized information to improve the quality of long-range plans of work and 2) to analyze and disseminate information for updating existing plans of work or creating new ones.

Approach: My activities for the long-range planning effort and  database development have focused on the collection of data relevant for the identification of target audiences and needs, the assembly of a microcomputer database, interpretation of selected components for programming purposes, and the dissemination of information to county and state Extension faculty as appropriate.  These activities are detailed below:

• The implementation of Challenge '95 required the assembly of information about trends and characteristics of Florida's population and farms for the state as a whole.  I co-authored the publication entitled Population and Agriculture in Florida: A New Look at Trends and Characteristics, which presented this information.  In addition, the bulletin combined the state-level information with tables of county-specific data.  A copy was provided to every county and state Extension faculty member.

• For the Florida 2000 & Beyond planning process, I obtained data from a variety of sources, including the U.S. Census of Population and Housing, U.S. Census of Agriculture, Florida Statistical Abstract, Florida Department of Education, and the Florida Center for Children and Youth.  Data were selected and analyzed, with some items being mapped or graphed.  Subsequently, I prepared an Extension bulletin, Population and Agriculture in Florida: An Update on Trends and Characteristics, which provided similar information to the bulletins used for Challenge ‘95.  A copy was provided to every Extension faculty member.

• Demographic and socio-economic data have been provided to state specialists, county faculty, and Extension customers for the development of situational statements in grant proposals, for background information for affirmative action reviews, and for planning new Extension programs.

Information in the Florida 2000 & Beyond bulletin was widely used for FY96-99 plans of work and even 2 years after printing, I received requests for copies.  John Woeste, former Dean for Extension, wrote, "It is an excellent and very timely document. Thanks for a job well done."
 

Conducting needs assessments

County and state Extension faculty often lack detailed information about the behaviors, practices, needs, aspirations, and characteristics of clientele groups which is necessary to implementing effective educational programs.  I work with these faculty to obtain needs assessment data for determining the educational content for specific audience segments.  Extension faculty can also learn how to design and conduct a methodologically sound needs assessment survey through this process.  For example, a survey on citrus growers cold protection practices was conducted in 1996 to assist with planning programs for this clientele group.

Situation:  In 1987, "Citrus Management in Florida" was designated as an Extension state major program (FL08).  To support effective educational activities in this area, four surveys, "Citrus Business Management," "Citrus Pest Management," "Citrus Cold Protection and Water Management," and "Citrus Young Tree Care" were conducted by PDEC and citrus faculty at the county and state levels.  Following the successful citrus management needs assessment surveys, the Citrus Root Weevil Needs Assessment Survey was conducted in 1992 and the Citrus Cold Protection Survey in 1996.

Objective:  To provide information to state and county Extension faculty for planning and prioritizing educational programs for the citrus industry in Florida.

Approach: A mail questionnaire was selected as the means to obtain information about citrus growers in Florida.  I worked with Dr. Jim Ferguson to develop a questionnaire to survey citrus growers about their knowledge and practices concerning cold protection, with input from other extension faculty; supervised data gathering, database assembly, and data analysis.  The results have been shared with Extension faculty and growers at state major program design team meetings and at the 1997 FACTS trade show.

The needs assessment surveys provided specific information about customer needs and helped Extension faculty in Florida developed better targeted programs.
 

Community Needs Assessment and Youth Leadership Development

Many small communities want to conduct a needs assessment to use in setting priorities for local development efforts yet lack resources and expertise to carry out a survey that is useful and relevant, scientifically rigorous, and cost effective. These communities also have youth needing opportunities to learn citizenship skills and how to serve their community.  With a colleague at the University of Kentucky, I undertook an effort to integrate these two program thrusts.  This work involved: developing a prototype program, refining the program and preparing a package of educational materials, and conducting educational programs to facilitate the adoption of the program.

•  Developing a prototype program

• My initial efforts focused on helping the Collier County Extension Director (CED) and a community organization in Immokalee, Florida to conduct a community needs assessment in 1992.  I also developed and implemented an evaluation of that effort to assess students' understanding of their community, feelings of empowerment as a community resource, and aspirations to be active in the community as adults.  As a result of the needs assessment, the following impact occurred: 1) The students completed 434 interviews; 2) The cooperating teacher reported observing increased self-esteem among some students, most students learned more about their community and some aspired to continue their involvement in community affairs after graduating; 3) Thirteen presentations to local organizations were made by a team of students,  the CED and the teacher; and 4) The CED organized a meeting of leaders identified in the survey and implemented a process of mobilizing the community to act on the results of the needs assessment.

• Refining the program and preparing a package of educational materials
• Following the successful completion of the project in Florida and Kentucky, I received funding by the Southern Rural Development Center (SRDC) to develop a handbook to help other communities conduct similar needs assessment projects involving high school students.  I developed substantial segments of the handbook, hired and supervised a graduate student who worked on the handbook, and reviewed and edited materials written by others.  The handbook was published by the SRDC. In addition, materials from the handbook have been pilot tested in Union County and Quincy, Florida, and in Shelby County, Kentucky.  In all cases, the projects were successfully completed.

• Conducting educational programs to facilitate the adoption of the program

• The results of this project have been communicated to national audiences through journal articles and conference presentations.  The SRDC has been distributing copies of the handbook regionally and nationally to several audiences, including rural development Extension faculty, state departments of education, and Americorps contacts.