2.3 An evaluation of the data collection and exhibition pilots.
Data Collection Pilot
What went well
• the log sheets produced a reasonable amount of data
What did not work so well
• It was very hard to get the log-sheets returned
• people were unsure of what I wanted in the 'distractions' section
• asking people to look at 3 sites was too much and demanded too great a time commitment from people
2.32 The Pilot Exhibition - lessons learned.
The Presentation
What went well
• I would use the interactive white board/ laptop combination again. It was very clear and easy for people to see.
• JAlbum provided an excellent alternative to using specific presentation software. The images were large and clear and worked very well
• The visual representation of the research findings.
• Participants made no objections to my interpretations of the findings even though they were given both public and private opportunities to do so.
• my audience appeared to be engaged and interested.
• talking directly to the audience without a script but using the slides as a prompt worked well. I was comfortable doing this and on listening again to the audio recording I sound quite confident and relaxed. It gave me the flexibility to engage with the audience in a natural way and it is clear from the recording that I did not take too formal a tone or place myself in the role of 'expert'.
What did not go as well
• using an on-line presentation was a risk and right at the end the connection did fail. I had a version of the presentation on a CD already loaded on the laptop in case this happened. The connection at work can be unreliable but I wanted to experiment with using an on-line version. This is partly because I may use a on-line tools such as web notes for my main exhibition and so I need to try it out.
• There were too many slides that were just text, especially towards the end.
They added nothing to the presentation except to act as prompts for me.
• There was too much detail for the time I allowed.
• Introducing my ideas for first actions at the end of the presentation. This distracted people from the research as such. It introduced them to a range of new concepts that they had not heard of before and did so in a rather rushed way. It seems likely that this may be one reason that the focus group did not work very well as people were still engaged with trying to understand and process the new information they'd been given. This might explain why the focus group was dominated by people asking questions about blogs and wikis rather than the research as such. I had pointed them at this future action and it was hard to bring them back to discussing the current situation.
2.33 The Display Boards
What went well
• the layout was clear and uncluttered
• the choice of images from Flickr - colourful and attractive
• using the structure of 'what, so what, now what' gave 3 clear sections
What did not go so well
• putting the boards up in the staff room. They quickly became part of the surroundings and were ignored. The only place in the staff room where notices can be placed is a 'dead spot'. The new notice board area is away from the seating and people have to make an effort to get up and look there.
• printer problems meant that I was limited in my choice of images. I probably need to buy a new printer before the main exhibition.
2.34 Structured Focus Group
What went well
• there was no negative feed back
• the TAs were able to attend which they would not have been able to do after school.
What did not work so well
• finding out what people thought about the research
• getting people to focus on the research rather than answers to factual questions
• I wasn't able to control the group. This was partly because the two male participants dominated the group. These are both people who are in positions of authority in the school and are used to being in control. I didn't feel able to assert authority to keep the group focused or move on without answering their questions.
• holding the focus group straight after the pilot was a bad idea. People had no time to assimilate the information.
• holding the focus group at lunchtime also caused problems as people were keen to get back to the staff room to relax before lessons resumed.
2.35 Feedback Sheets
The lack of response to my request for feed back led to considerable depression on my part. It was easy at this point to dismiss my pilot exhibition and study as having had little value.
Looking at this lack of response from a greater distance I can start to see some value in it. As all my participants were present at the exhibition it therefore follows that they did not disagree with the findings of my study.
In retrospect the use of 'evaluation ' sheets was not a good choice of data collection method. As I noted in a posting to my learning set:
They immeadiately related what I was asking for to the dreaded evaluation forms we are given at the end of every course - a boring task to be avoided or got over as quickly as possible with as little response as possible.
(Posting to AR2)
2.36 Informal Feed Back
I wrote about my worries about the lack of response in my Meta-Learning Journal and received this reply from Richard Millwood of Ultralab:
So what's left as an evaluation? Well your own view is no bad thing. Also, have you seen any change in subsequent days, in conversations or actions on the part of your colleagues? It may be that you need to keep your eyes peeled for signs for some time to come! I do wish you all the best in this - impact is often a slow burn, but I know it wasn't a wreck.
This proved to be good advice as can be seen from the learning journal quotes in the description of the pilot. There seems to have been some sort of shift in the way my colleagues see me. I will continue to monitor this in my learning journal.
(See Appendix1 near end for further examples)
An LF on Tuesday, December 6, 2005 at 10:27 pm +0000 wrote:
Yes, and observations recorded in a learning journal as a source of data for subsequent reflection / analysis are very valuable.
Yes - I must say though it would have be better to have some actual genuine feedback. I'm not trying to say we should try to do without.
I would stress though that it is important to go through the process of collecting the data If you mean recording it in the learning journal I agree. It's too easy to let the day slip past without recording incidents, keeps happening to me.
(From a conversation in FirstClass Cohort 1)
[Forward to Significant Learning]
[Back to index]
Data Collection Pilot
What went well
• the log sheets produced a reasonable amount of data
What did not work so well
• It was very hard to get the log-sheets returned
• people were unsure of what I wanted in the 'distractions' section
• asking people to look at 3 sites was too much and demanded too great a time commitment from people
2.32 The Pilot Exhibition - lessons learned.
The Presentation
What went well
• I would use the interactive white board/ laptop combination again. It was very clear and easy for people to see.
• JAlbum provided an excellent alternative to using specific presentation software. The images were large and clear and worked very well
• The visual representation of the research findings.
• Participants made no objections to my interpretations of the findings even though they were given both public and private opportunities to do so.
• my audience appeared to be engaged and interested.
• talking directly to the audience without a script but using the slides as a prompt worked well. I was comfortable doing this and on listening again to the audio recording I sound quite confident and relaxed. It gave me the flexibility to engage with the audience in a natural way and it is clear from the recording that I did not take too formal a tone or place myself in the role of 'expert'.
What did not go as well
• using an on-line presentation was a risk and right at the end the connection did fail. I had a version of the presentation on a CD already loaded on the laptop in case this happened. The connection at work can be unreliable but I wanted to experiment with using an on-line version. This is partly because I may use a on-line tools such as web notes for my main exhibition and so I need to try it out.
• There were too many slides that were just text, especially towards the end.
They added nothing to the presentation except to act as prompts for me.
• There was too much detail for the time I allowed.
• Introducing my ideas for first actions at the end of the presentation. This distracted people from the research as such. It introduced them to a range of new concepts that they had not heard of before and did so in a rather rushed way. It seems likely that this may be one reason that the focus group did not work very well as people were still engaged with trying to understand and process the new information they'd been given. This might explain why the focus group was dominated by people asking questions about blogs and wikis rather than the research as such. I had pointed them at this future action and it was hard to bring them back to discussing the current situation.
2.33 The Display Boards
What went well
• the layout was clear and uncluttered
• the choice of images from Flickr - colourful and attractive
• using the structure of 'what, so what, now what' gave 3 clear sections
What did not go so well
• putting the boards up in the staff room. They quickly became part of the surroundings and were ignored. The only place in the staff room where notices can be placed is a 'dead spot'. The new notice board area is away from the seating and people have to make an effort to get up and look there.
• printer problems meant that I was limited in my choice of images. I probably need to buy a new printer before the main exhibition.
2.34 Structured Focus Group
What went well
• there was no negative feed back
• the TAs were able to attend which they would not have been able to do after school.
What did not work so well
• finding out what people thought about the research
• getting people to focus on the research rather than answers to factual questions
• I wasn't able to control the group. This was partly because the two male participants dominated the group. These are both people who are in positions of authority in the school and are used to being in control. I didn't feel able to assert authority to keep the group focused or move on without answering their questions.
• holding the focus group straight after the pilot was a bad idea. People had no time to assimilate the information.
• holding the focus group at lunchtime also caused problems as people were keen to get back to the staff room to relax before lessons resumed.
2.35 Feedback Sheets
The lack of response to my request for feed back led to considerable depression on my part. It was easy at this point to dismiss my pilot exhibition and study as having had little value.
Looking at this lack of response from a greater distance I can start to see some value in it. As all my participants were present at the exhibition it therefore follows that they did not disagree with the findings of my study.
In retrospect the use of 'evaluation ' sheets was not a good choice of data collection method. As I noted in a posting to my learning set:
They immeadiately related what I was asking for to the dreaded evaluation forms we are given at the end of every course - a boring task to be avoided or got over as quickly as possible with as little response as possible.
(Posting to AR2)
2.36 Informal Feed Back
I wrote about my worries about the lack of response in my Meta-Learning Journal and received this reply from Richard Millwood of Ultralab:
So what's left as an evaluation? Well your own view is no bad thing. Also, have you seen any change in subsequent days, in conversations or actions on the part of your colleagues? It may be that you need to keep your eyes peeled for signs for some time to come! I do wish you all the best in this - impact is often a slow burn, but I know it wasn't a wreck.
This proved to be good advice as can be seen from the learning journal quotes in the description of the pilot. There seems to have been some sort of shift in the way my colleagues see me. I will continue to monitor this in my learning journal.
(See Appendix1 near end for further examples)
An LF on Tuesday, December 6, 2005 at 10:27 pm +0000 wrote:
Yes, and observations recorded in a learning journal as a source of data for subsequent reflection / analysis are very valuable.
Yes - I must say though it would have be better to have some actual genuine feedback. I'm not trying to say we should try to do without.
I would stress though that it is important to go through the process of collecting the data If you mean recording it in the learning journal I agree. It's too easy to let the day slip past without recording incidents, keeps happening to me.
(From a conversation in FirstClass Cohort 1)
[Forward to Significant Learning]
[Back to index]