21. Project Evaluation Plan

PRONETT Evaluation Plan

This report describes the initial outline for the evaluation plan for the Minerva project PRONETT. The approach being suggested addresses qualitative and quantitative data collection for both formative and summative purposes. This evaluation will:

• Provide evidence of PRONETT’s impact on the partner institutions involved
• Provide a valid and reliable statement of PRONETT’s impact to be given to the Minerva organization
• Provide timely formative data on the PRONETT initiative
• Support advocacy for the continuation of PRONETT after the three year grant has concluded

The success of PRONETT will be based on the ability and extent to which the portal produces the intended improvements in pre-service and in-service teacher education in ICT outlined in the initial proposal. The issues outlined in the initial proposal will be the driving force behind the questions the evaluation attempts to answer. A list of questions directly related to measure the success of PRONETT can be seen in Appendix A. Evaluation questions and instruments will be designed to reflect PRONETT’s success at achieving the aims and objectives. Each question will be designed to provide meaningful information about how well the program performed, or information leading to formative improvements and implementation procedures. Questions will relate to delivery, technical issues, outcomes, program services, program operations, and program design. Participating partners will be given the opportunity to add any other questions to the evaluation that are of interest to them. A timeline of evaluation plan development and instrument design and testing can be seen in Appendix B. Communication between partners will be facilitated by the use of PRONETT’s communication systems. This will also serve as a technical test for the tracking system encoded in PRONETT (Appendix E). A pilot study will take place which will also be testing ground for survey questionnaires and PRONETT functions.

Many similar have projects have failed because of the lack of appropriate implementation procedures and successful attention to the emotional and technical needs of the users. For formative purposes we the evaluators have identified several key phases that each partner must success through as they implement PRONETT at their respective institutions (Appendix F). The phases are based on information from the Concerns Based Adoption Model (CBAM) designed by researchers at the University of North Texas in 1970 Roger’s Innovation Decision Model and the 4 E’s described in Flexible Learning in a Digital World by Betty Collis and Jef Moonen. Because the PRONETT system should be directly responsive to the needs of the environmental and pedagogical contexts at each institution , the partners will only be asked to respond to questions regarding the implementation phases, but will have the freedom to provide training, resources, administrative mandates, and other logistics to the implementation at their discretion. They will record these events in their partner logs.

Following the compilation of the evaluation questions and agreement and sign off by stakeholders, the most appropriate instruments will be designed to answer these questions. A few ideas have been discussed at this point:

Data Collection Procedure: Dialogue Code System labeling levels of discussion at 5 levels

Instrumentation: Conversation is automatically logged by system and coded by students as a reflection process

Timing of Data Collection: System constantly logs conversations and students have opportunities to turn in reflection logs coded according to the code system. Sampling:

Data Collection Procedure: Code System to track conversation by user type Instrumentation: Tracked by PRONETT system functions with output

Timing of Data Collection: Continuous and available on demand

Sampling: This system will not be a major part of the evaluation because of the low return rate the evaluators feel they would have on this effort. It will only be optional if students and teachers choose to participate in this event.

Instrumentation 2: The code system will also be used to evaluate dialogue levels at the English University in Cardiff during the first two semesters of PRONETT’s implementation process as a formative evaluation procedure.

Sampling: Random sample of 30 individuals of each of the three user types over a period of nine months.

** For reliability, the same individual should code all conversations for the English participants.

Data Collection Procedure: Personal interview performed by students to students

Instrumentation: Questionnaire

Timing of Data Collection: At initial log on within the first week and at the end of the school year. Pilot study questionnaire will be different from later instruments.

Sampling:

Data Collection Procedure: Interview of teachers that have used Pronett

Instrumentation: Questionnaire

Timing of Data Collection: After contact with Pronett or after a project request has been fulfilled

Sampling:

(We need more ideas on this, should the system do this?)

Data Collection Procedure: Partner reports

Instrumentation: Observation by partners in the log (Appendix D) of the system function uses by all users will be a large part of the formative evaluation

Timing of Data Collection: Every three months

Sampling: 4 partners four times per year over a period of three years

Partners will be responsible for:

• Completing and returning the log sheet every three months for formative evaluation purposes. (Appendix D)
• Developing and maturing relationships with practicing teachers and logging the desires of those teachers in regard to needed ICT developments in the PRONETT system
• Developing faculty workshops which encourage the development of ICT related skills
• Completing the evaluation form at each PRONETT partner meeting (Appendix 4 in survival kit)

The system or project manager will be responsible for:

• Tracking conversation by user type as a backend log
• Collecting and merging log information of all partners
• Creating a half yearly newsletter to be available on the PRONETT site (this can be a student project and done as an ICT project for credit)
• Administering pre and post questionnaire to faculty, pre-service teachers, and practicing teachers
• Log private sessions for PRONETT users to be used for reflection purposes. This is a system function

The monthly newsletter will be sent to all partner institutions, Minerva administrators, and will be available electronically on the PRONETT site. Information for the newsletter will be provided via faculty logs. The newsletter will have the following components:

• Statement of PRONETT’s purpose (mission statement)
• Spotlight on some projects
• Spotlight on practicing teachers
• Stories of virtual mobility or actual
• Ideas for faculty workshops
• Student blurbs
• International trends in Education

The newsletter will serve several aims and objectives of PRONETT:

• Creation of an international community focused on education and ICT
• Reward System for good practice
• Development of ICT skills in web publishing
• Recent changes to the system and updates
• Areas of interest being developed

A quarterly formative evaluation report will be created and sent to Brussels for review.

• Statements of progress related to formative evaluation
• Evaluation timeline