21. Evaluation Plan (version 2.0)

PRONETT Evaluation Plan

This report describes the initial outline for the evaluation plan for the Minerva project PRONETT. The approach being suggested addresses qualitative and quantitative data collection for both formative and summative purposes. This evaluation will:

• Provide evidence of PRONETT’s impact on the partner institutions involved
• Provide a valid and reliable statement of PRONETT’s impact to be given to the Minerva organization
• Provide timely formative data on the PRONETT initiative
• Support advocacy for the continuation of PRONETT after the three year grant has concluded

The success of PRONETT will be based on the ability and extent to which the portal produces the intended improvements in pre-service and in-service teacher education in ICT outlined in the initial proposal. The issues outlined in the initial proposal will be the driving force behind the questions the evaluation attempts to answer. A list of questions directly related to measure the success of PRONETT can be seen in Appendix A. Evaluation questions and instruments will be designed to reflect PRONETT’s success at achieving the aims and objectives. Each question will be designed to provide meaningful information about how well the program performed, or information leading to formative improvements and implementation procedures. Questions will relate to delivery, technical issues, outcomes, program services, program operations, and program design. Participating partners will be given the opportunity to add any other questions to the evaluation that are of interest to them. A timeline of evaluation plan development and instrument design and testing can be seen in Appendix B. Communication between partners will be facilitated by the use of PRONETT’s communication systems. This will also serve as a technical test for the tracking system encoded in PRONETT (Appendix E). A pilot study will take place which will also be testing ground for survey questionnaires and PRONETT functions.

Many similar have projects have failed because of the lack of appropriate implementation procedures and successful attention to the emotional and technical needs of the users. For formative purposes we the evaluators have identified several key activities that each partner must success through as they implement PRONETT at their respective institutions (See the generic model Workpackage Implementation). The evaluation approach adopted in this project is based on information from the Concerns Based Adoption Model (CBAM) designed by researchers at the University of North Texas in 1970 Roger’s Innovation Decision Model and the 4 E’s described in Flexible Learning in a Digital World by Betty Collis and Jef Moonen. Because the PRONETT system should be directly responsive to the needs of the environmental and pedagogical contexts at each institution , the partners will only be asked to respond to questions regarding the implementation phases, but will have the freedom to provide training, resources, administrative mandates, and other logistics to the implementation at their discretion. They will record these events in their partner logs.

Following the compilation of the evaluation questions and agreement and sign off by stakeholders, the most appropriate instruments will be designed to answer these questions. A few ideas have been discussed at this point:

Data Collection Procedure: Dialogue Code System labeling levels of discussion at 5 levels

Instrumentation: Conversation is automatically logged by system and coded by students as a reflection process

Timing of Data Collection: System constantly logs conversations and students have opportunities to turn in reflection logs coded according to the code system. Sampling:

Data Collection Procedure: Code System to track conversation by user type Instrumentation: Tracked by PRONETT system functions with output

Timing of Data Collection: Continuous and available on demand

Sampling: This system will not be a major part of the evaluation because of the low return rate the evaluators feel they would have on this effort. It will only be optional if students and teachers choose to participate in this event.

Instrumentation 2: The code system will also be used to evaluate dialogue levels at the English University in Cardiff during the first two semesters of PRONETT’s implementation process as a formative evaluation procedure.

Sampling: Random sample of 30 individuals of each of the three user types over a period of nine months.

** For reliability, the same individual should code all conversations for the English participants.

Data Collection Procedure: Personal interview performed by students to students

Instrumentation: Questionnaire

Timing of Data Collection: At initial log on within the first week and at the end of the school year. Pilot study questionnaire will be different from later instruments.

Sampling:

Data Collection Procedure: Interview of teachers that have used Pronett

Instrumentation: Questionnaire

Timing of Data Collection: After contact with Pronett or after a project request has been fulfilled

Sampling:

(We need more ideas on this, should the system do this?)

Data Collection Procedure: Partner reports

Instrumentation: Observation by partners in the log (Appendix D) of the system function uses by all users will be a large part of the formative evaluation

Timing of Data Collection: Every three months

Sampling: 4 partners four times per year over a period of three years

Partners will be responsible for:

• Completing and returning the log sheet every three months for formative evaluation purposes. (Appendix D)
• Developing and maturing relationships with practicing teachers and logging the desires of those teachers in regard to needed ICT developments in the PRONETT system
• Developing faculty workshops which encourage the development of ICT related skills
• Completing the evaluation form at each PRONETT partner meeting (Appendix 4 in survival kit)

The system or project manager will be responsible for:

• Tracking conversation by user type as a backend log
• Collecting and merging log information of all partners
• Creating a half yearly newsletter to be available on the PRONETT site (this can be a student project and done as an ICT project for credit)
• Administering pre and post questionnaire to faculty, pre-service teachers, and practicing teachers
• Log private sessions for PRONETT users to be used for reflection purposes. This is a system function

The monthly newsletter will be sent to all partner institutions, Minerva administrators, and will be available electronically on the PRONETT site. Information for the newsletter will be provided via faculty logs. The newsletter will have the following components:

• Statement of PRONETT’s purpose (mission statement)
• Spotlight on some projects
• Spotlight on practicing teachers
• Stories of virtual mobility or actual
• Ideas for faculty workshops
• Student blurbs
• International trends in Education

The newsletter will serve several aims and objectives of PRONETT:

• Creation of an international community focused on education and ICT
• Reward System for good practice
• Development of ICT skills in web publishing
• Recent changes to the system and updates
• Areas of interest being developed

A quarterly formative evaluation report will be created and sent to Brussels for review.

• Statements of progress related to formative evaluation
• Evaluation timeline

Appendix A

Evaluation questions related to the issues in AIMS AND OBJECTIVES:

1. Facilitate electronic communication between pre-service teachers and teachers to enhance professional development through the use of ICTs

• Did teachers and pre-service teacher communicate?
• How much
• Which mediums were used and how often for each medium

• Was professional development enhanced?
• Are uses in ICT duplicated/repeated by students and faculty How much and percentage by which type of user
• Which resources were used the most to promote professional development (checklist of ICT types)
• What do in –service teachers find beneficial in regard to ICTs. Formative evaluation question

1. Offer a network approach…….

• Was a network approached offered and technically available on a consistent basis?
• How much was the network used
• Which type of user used it the most
• Which partner/country used it the most
• Where was communication and collaboration most lacking
• Types of collaboration (coded system for discussion)
• Was teaching and learning discussed in the environment (part of the code)

1. Develop a regional and cross national networked learning community of teachers, teacher trainers, and pre-service teachers offering a variety of web-based resources and tele-learning tools for collaboration inquiry into and facilitation of the discourse on learning to teach in a networked classroom?

• How many nation/regions were represented
• How much communication between nations by %
• How much communication between nations by user type
• What was the extent to the variety of web-based resources offered?
• What web-based resource was most often used
• Was there a change in type of used resource over time?
• Were web-based resources and tele-learning tools used for collaborative inquiry teaching in a networked environment (code)
• Were web-based resources and telelearning tools used for facilitation of discourse about teaching in a networked environment (code)
• Is there a difference in type of discourse by country
• Is there a difference in type of discourse by user type

1. Develop and implement a "Dynamic" and "practice –based" curriculum for ICT in teacher training

• Was the curriculum Dynamic (innovation section can define this)
• Was the curriculum practice based
• Were the partner school curriculums incorporation ICT into the curriculums
• What % by whom nation and user type

1. Develop models of online support, assessment, and reflective practice

• Were models of online support created
• Types of "reflection" by nation or school
• Types of "reflection" by nation or school and by user type

1. Develop "useful" "cross-curriculum" resources

• What curriculums were supported by the resulting developments of student work
• What is the percentage of these per curriculum area
• How much were developments used by those other than the author

1. Cooperation between regional and cross national training provision together with the potential of placement opportunities with other PRONETT partner institutions

• Was mobility increased in these areas?

Evaluation questions related to other expectations and statements not listed in the AIMS AND OBJECTIVES:

1. Did the PRONETT environment promote constructivist development?
2. Did users experience a positive first experience so as to become subsequent users in a national and European initiative?
3. Can trainees and trainers and teachers articulate ICT training and teacher requirements
4. Are various e-rooms available for various levels of skill and application?
5. Was a virtual gallery available
6. Were catalogues evaluated, tried, and adapted for new uses
7. Were items regularly placed in a catalog/database by partners
8. Were discussions scheduled in a roundtable arena
9. Was the system easy to use (languages, translation, interfaces, manual, technical matters)
10. Was a newsletter sent out quarterly?
11. Were evaluation instruments available for the resources created by students
12. Was the expected number of people affected n=
13. Did PRONETT increase the quantity and quality ICT based products in schools

• how do practicing teachers rate the materials
• Do scores on rubrics improve over time per student

1. Did PRONETT help integrate the methodology of ICT into continuos professional development programs

• How many faculty workshops were done at each partner institution?
• Was there follow-up to check for level 4 impact?

Appendix B

Evaluation Plan Deadlines and Procedures

Appendix C

Code Systems for PRONETT evaluation

In reviewing the proposal to Minerva and the Aims and Objectives outlined within, it is clear that there is need to track dialogue by type, distinguishing various levels of dialogue, will be of importance in order to judge the success of PRONETT. PRONETT had as its objective the facilitation of communication which enhances professional development, reflection, assessment, constructivist ideas, and both discourse on teaching and learning related to ICT and teaching in a networked environment. The following system has been suggested. The PRONETT system will be operationalized to log conversations for post reflection and coding. Not only will this serve the aims of the evaluation, but will provide students with the opportunity to return to dialogue and reflect on the contents. It will be the responsibility of the students to reflect on the dialogue and label each conversation according to the system below.

Dialog Code System

1 = Uncodable Response such as typos or non translatable communication

2 = Chat unrelated to the aims or objectives of PRONETT

3 = Technical discussions, help, or comments

4 = Shallow responses to work i.e.: "that’s good", "I liked it"

5 = Discussion about teaching and learning, ICT, teaching in a web-based environment or any discussion related to PRONETT’s aims

Question: Students will code this

When will it be coded?

Where will the information be stored?

The development of a user code would allow the PRONETT system to track discussions by user and user type. This system would allow the partner groups to see trends in discussion currents and by type, country and subject interests. The system would allow a number of reports to be printed out providing invaluable information to the evaluation effort. After filling out a brief questionnaire, users would be provided a unique code to track their discussions. Discussions would remain confidential because only the code is tied to the name. Names would not appear on printed reports.

User Code System Example

Ton Koenraad: Faculty (F), Netherlands (NL), Number (1), Interest IT = FNL1IT

Student from Wales: Student (S), Wales (W), Number (1), Interest MA = SW1MA

A system tracking conversation between users of PRONETT could track how many times FNL1IT and SW1MA used PRONETT for communication with each other. One could ask for a report on student communication only in Spain by running a report just on SSXX the XX being any number in the SS (Students of Spain) group.

Partner Log Name____ User Code _____ Date ____

To be turned in every 3 months

The PRONETT system has been designed to provide you with the information on the first section of questions.

1. Did you register any new students in the PRONETT System? Yes __ No __

How many ___

2. Did you register any new practicing teachers in the PRONETT system?

Yes __ No __

3. How many ___
4. Did they indicate and special desires for ICT developments?
5. Please list any special indications made by teachers.
6. Did any of the students in your region/country adopt these special indications?

Yes ___ No ___

7. How many special indications were fulfilled by pre-service teachers for practicing teachers _____?

Email

Chat

???

9. What functions of the system was used the least by your students for communication?

10. Did your students develop any ICT projects in the past three months?

11. How many were added to the PRONETT system?

12. How many courses are incorporating the PRONETT system into their curriculum? ___

13. Please describe how this is being done.

14. How were these evaluated for their appropriateness for the PRONETT database?

15. How did the PRONETT system facilitate the development of these materials?

16. The system is designed to log private sessions for students post reflection and logging. Please have students total their conversations by type and then provide this report with a total by type for all students.

Level 1 = ____

Level 2 = ____

Level 3 = ____

Level 4 = ____

Level 5 = ____

17. Were faculty offered workshops? Yes __ No __

18. Please check the events covered in your workshops?

____Models of online support

____ Assessment and Evaluation of ICT materials

____ WebQuests

and list of various hardware and software needed for online communication

19. Please check any areas you feel had technical problems

___ Chat

___ Email

___ Database links

20. Was the management in your region supportive of PRONETT and your responsibilities to the system?

Yes ___ No___

If no, please explain.

21. Were the materials available on the PRONETT site appropriate for your needs regarding workshops, management support etc? Yes ___ No___.

If no, please explain.

22. Please list three interesting relationships, outcomes, or projects that PRONETT has facilities for use in our newsletter.

23. Please describe what transnational activities your region was involved in as facilitated by PRONETT in the following areas:

a. Sharing information about your country and education systems:

24. Was PRONETT successful in overcoming language barriers? Yes ___ No___

Please expand on this question.

25. Please provide information about the ability of PRONETT’s functions to meet the needs of users in your region.

26. If a newsletter has been published in the last 3 months, how do you feel about its representation of all aspects of PRONETT and its international representation?

27. Please list any improvements that could be made to the newsletter.

28. Do you feel there is equitable commitment by all partners?

Yes ___ No ___

29. Do you feel you had ample opportunities to participate in the development of PRONETT? Yes ___ No ___

Please provide any further information on this topic with considerations to finances, planning, meeting location and times, evaluation procedures and discussions of data, and communication opportunities.

30. Please supply other comments not addressed in this log.

Appendix E

System Functionality Speculations

User Code:

The user code should identify each PRONETT user with an individual code at the initial login moment. The system should provide a brief questionnaire with pre-designated answer choices so as to provide the system with a consistent tracking system for report generation. This code should identify the following characteristics of the individual:

Nationality _ _ ES, NL, UK, BE

Status _ F, S, T

Subject of interest _ _ MA, SC, IT, CH, PH, ED, MFL

Number _ _ _ _ Each user should have a unique number ascending form the first person to sign on to the system. Each user type (teacher, student, faculty) should have a set of numbers, each starting with 1. There will be a teacher 1, a faculty 1, and a student 1.

The PRONETT will automatically log and save all conversations between all users so that users can access past dialogue and print this information. This information should be available for purging at a pre-designated point every semester so as to save system space. Users should receive emails of this purge date 5 days before the purge activates.

The tracking of special interests indicated by practicing teachers should be a function of the PRONETT system. PRONETT should have an area where teachers can log their desires for ICT developments to be developed by PRONETT students and used in their classrooms. The system should provide a check box for students to utilize once a project is selected and another check box to designate the projects completion. The system should provide the number of suggested projects by teachers, chosen projects by students, and the number of projects completed by students. Each of these data should be operation in an excel spreadsheet and should be shown by country.

The system should provide the following information to partners, which should be easily downloaded and functional in an Excel spreadsheet in an effort to provide formative evaluation for PRONETT development.

• Number of new students in the system by country
• Number of new teachers registered in the system by country
• Number of special interest indications provided by teachers and by subject
• Number of special interest indications fulfilled by students and by subject
• System functions most used in PRONETT system by user type
• System functions least used in PRONETT system by user type
• Number of new items added to PRONETT database by country and by date