Home

View the SET Feedback Survey Report

1. still cannot extract a report from the system The only option can see to click leads me to an empty excel spreadsheet with unreadable headings hope this is a defect in me as a user but The link to the user manual does not function It should Names of topics which actually are hyperlinked do not look like hyperlinks and do not change the cursor to indicate that they are hyperlinks unlike every other piece of software use am so surprised that cannot control the questionnaire or even see the questionnaires that relate to me unless am topic co ordinator and thus cannot choose optional questions or set up reminders or control the date the survey goes out to match when my teaching in a topic ends think this should be changed Students complain to me about the volume of emails they get about SET This clearly does not make them feel more like complying with requests to fill the form in The response rate for my classes so far this year is so low that will have to write off SET as a useable form of feedback for me personally and it will also be unuseable for any other purpose such as teaching awards promotion etc will have to return to paper forms handed out in class time in future in order to avoid this outcome There is clearly not enough incentive for students to respond and this jeopardises the usefulness and representativeness of what feedback has been generated Best to have the system where we choose the quesitons and print
2. SET Feedback Survey Results 2009 Planning Services Unit January 2010 SET Feedback Survey Results 2009 Introduction The following tables summarise staff and student responses to the SET Feedback Survey conducted online from October to December 2009 The SET Feedback Survey had two versions a staff survey 168 responses and a student survey 36 responses An email was sent to all academic staff to invite their feedback online No emails were sent to students about the SET Feedback survey Instead students were invited through a notice on the Student Portal around exam time A copy of the staff and student survey questions can be found at Appendix C and D respectively Summary Staff SET Feedback e 67 of the 129 staff responses had broad satisfaction with the overall service and support from Planning Services Unit Table 1 1 e There were 84 comments about best aspects of support or services Staff tended to comment on the promptness and availability of Planning Services Unit s support as well as the helpfulness of the training sessions provided Appendix A Part 1 e 41 staff had suggestions for service or support improvements Suggestions included having an on line training tool with no assumed previous knowledge or help documentation in addition to more training sessions Appendix A Part 2 e 52 of respondents had broad satisfaction with the SET system overall a further 32 rated the SET system average overall Table 1 2
3. e 92 staff responded to the question What are the best aspects of the new Flinders SET system Staff mentioned less paperwork automation greater access ie the ability to now survey external students and ease of use Appendix A Part 3 e Almost half 47 comments out of 97 staff responses of all comments to the question Do you have any suggestions for improvement to the new SET system related to a low response rate for the online SET Some suggestions to increase response rates by staff were to offer incentives to answer to allow both paper and online surveys to run simultaneously and to make it compulsory to answer the SET possibly not releasing topic results until the SET was completed Some staff asked for tips on how to increase response rates while others were curious to know the overall response rate to online SET Appendix A Part 4 e 77 staff members responded with thoughts on the SET instruments questions 16 staff members 21 said they would like the ability to customise questions as well as remove questions that do not apply It was suggested that external students intensives postgraduate computer lab and industry placement topics do not always suit the generic questions currently asked In addition it was suggested staff should have the ability to ask their own questions particularly to get feedback on new methods of teaching things they are trialing or questions specific to their department Appendix A P
4. Advice on how to use it through individual tuition Never had to use them gt the system must be pretty good Always very helpful Provides feedback in a short space of time Very friendly Talitha Bennett has been very helpful when have had questions regarding creating SETs absolutely excellent one on one help over the phone to walk me through my mistakes amp assist me to fix them can speak to someone or email the presence of people in the department who have more experience training Part 2 Staff responses to the question Please suggest any service or support improvements 1 2 3 To not just roll out a system and expect people to know what to do No support If support is necessary the system is incompetent As a teacher who has specialised in Educational theories and methods MEd Dip Ed PhD think that the reliance on students to be the best judges of what should happen in tutorials to be somewhat dubious Many students do not bother to do required readings or make an effort to contribute but they still expect to be entertained during the tutes I m not sure teachers abilities and achievements are best reflected by simple surveys Look at online trigger prompts Make sure that the trainer is experienced in teaching adults and change management When introducing change please be aware of the time period in the middle of the semester was the worst time to bring about this change The compu
5. it is OK Not always sure what is meant by some of the language used Avoiding the paper war Nice to get all student comments electronically Ability to target surveys to particular classes breakdown results etc funky interface but a little distracting Ease of use Faster more efficient Diverse range of questions That it involves less paper work if any than before That we don t have to remember to do it ourselves The automation and using Admin staff to do it has been a great idea Ability to organise everything online The fact you can still elect to do paper versions and get valid responses Given we work in an academic setting we should have concerns for validity At least when you administer SETS in class you know that the students that are sitting in the lecture theatre and or tutorial room are in a position to evaluate e g lectures tutorials WHen you administer them electronically there is a high chance someone who has never attended a lecture including electonically and or tut may provide a response by virtue of the fact they re enrolled in it In addition by administering them in class you get a better response rate They don t waste paper easy access to past and present SETs fairly easy to use like the fact that the SETs are now automatic wasn t very good in remembering to get the SETs ready for classes Making them automatic is an advantage Online is probably
6. knowledgeable in the area Students are not really capable of judging this It s the University s responsibility to assign a knowledgeable person to teach the topic The questions should be focusing on improvement rather than making judgment teach a postgarduate online course that has an optional intensive find the SET very undergraduate internal focused and therefore many of the questions are not relevant to my context find have the mandatory questions frustrating and would prefer to see a set of questions to choose from for diffferent domains wanting to seek feedback and evaluation The Set is difficult in one off situations such as use for hte intensive More flexibility would enable us to use the online SET rather than hardcopy forms that are not put through the system don t think they are as accurate as the old system of getting students to do at the time of a tutorial Had much better response rates than this last electronic choice system Can skew the results as the only ones completing them are either motivated and enjoyed the subject or wanted to grip about it Fortunately mine were skewed the right way but one negative comment impacted so much on the overall scores as the number completing them was pretty poor low Not sure how this could be addressed but am sure there are ideas out there from other uni s that now use electronic choice systems they are fine as is as with any likert scale the lack of behavioural
7. The survey instrument is baised towards teaching in a face to face setting Students who take a topic via distance have expressed frustration with their inability to answer questions that make no sense in their context am happy wioth the SETs as insturments of revaluation as they stand When the systme is a 8seful as the SETs themselves all will be well Open question for students to put input about the tutor topic coordinator or topic itself would be useful The questions are not very useful They don t ask students very specific questions and encourage vague impressions rather than carefully thought out answers As far as can tell students are not good at answering them I usually get very good ratings When get good ratings get good ratings on all the questions even though I know I have sometimes been badly organised and have put in too little time writing comments on work When get bad ratings tend to get bad ratings on all of them even though know I have been well organised etc In one survey last year got a bad result for availability for consultation however was there every week in my consultation hours Only one student in a class of more than 40 ever came to see me He saw me repeatedly even though he didn t need to You need to ask some much more specific questions if you want to get reliable results eg did you attempt to seee this lecturer during consultation hours How many times Was s he available Did you attemp
8. anchors means that responses subjective choice of questions seems better most useful info for me is always from the free text comments SETs for external topics tutors would be greatly appreciated Questions seem ok The instrument is fine However online data entry does not work due to extremely low particpation rates providing unreliable and invalid results see above this is the biggest problem and it skewes the result The main issue is the drop have perceived in students actually undertaking it had an EXTREMELY LOW response rate using this system will return to using the paper version next time It seemed very similar to the previous versions apart from the FLO options and used some of these It was a real plus when the evaluation moved the How would you rate this lecturer to the end of the form so they responded to this after having completed more specific questions am happy with the format apart from having to set up for both internal and externals don t think it is a good idea to ask first year students if know my topic how would they know that Perhaps it would be more accurate to frame the question as their perception or my ability to share knowledge We need to be able to create questions appropriate to the department of medicine if we are to use the program and SET approach Poor response rates poses serious questions as to how serious the university is about the SETs Off loading res
9. good although see some disadvantages Easy access The whole process happens automatically without the need for form filling where do access it Automatic sending ease of use The detailed results and comments that are sent to staff none prefer paper sets response rate much lower than paper The on line should make it easier but it will take a lot more work to convince students to go back to their computers to complete the surveys rather than just handing them out in class It s easy All are very good There is some feedback It is automatic An update is sent weekly You can view the SET at a later date and the information is stored Planning Services Unit Page 8 of 23 92 SET Feedback Survey Results 2009 given i generate a paper SET i don t see much advantage in this new system a major disadvantge is that as topic coordinator i now have to make these individually for all teaching team members whereas before staff did their own That the surveys are set up for us which saves academics time fabulous mechanism for receiving feedback Great list of possible questions to draw from The fact that it is set up automatically the fact that it can be carried out online however this is also a great disadvantage with students taught on the University of Adelaide campus as they rarely access their Flinders University account It reaches all of the enrolled students Part 4 St
10. hand them in when staff passed out and collected a SET for a different teacher The online version means that it would be harder to tell who filled out the SET as you won t see people s writing which would give clues as to who wrote it Although if people don t fill in the comments sections then that wouldn t make a difference On the other hand the online version is easier to ignore it would be nice to know if the surveys are anonymous but other than that found the survey system easy to navigate and answer There are too many emails sent from the SET system to our Flinders email and it becomes like spam It s ok to send reminders but not one email for every survey we have to do One email saying we have for example 5 surveys to complete would be sufficient The SET seems quite comprehensive and concise but would like to know if the feedback is actually acted upon have heard of and participated in some topics with very high failure rates and yet the lecturers are still teaching am distance ed so couldn t complete in class Disciplines need to agree on systems for students i e reference systems active passive voice third person etc rather than each teacher having their own way of wanting things library services and IT services are very relevant for distance students and need to be evaluated as part of the overall teaching experience Credit transfer processes systems need to be clearer and as agreed in each discipline also pa
11. it out Electronically some stduents do the survey many don t and one does not know who actually is doing the survey the stduent could open it an dleave it to others to enter the details The system as is now is totally useless Sorry do not mean to offend but it is the truth Unless it s manadatory to complete it gives very skewed responses Students who might give the most useful feedback may not complete response rate is poor Manual SET was therefore more useful Planning Services Unit Page 13 of 23 85 86 87 88 89 90 91 92 93 94 95 more user friendly and intuitive 97 SET Feedback Survey Results 2009 prefer the old system There seems to be protocols e g if you re not topic coordinators you can t generate your own so if topic coordinators don t remember to include you you don t get SET Much more peer review The disappointing aspects are the complexity in setting up the various surveys for individual staff and the very poor response rate from students get rid of it The SET Teaching and Topic should have different names The SETeaching results must be accessible to supervisors and HoS currently they aren t unless the staff member ticks the box Completing on line has reduced student response rates Not having a consolidated set of results for a School means having to search the whole database The trouble is that fewer students seem to use it response rates are much lower than the pa
12. not fit the teachign that i do initially did not find the qualitative remarks by students as the pdf file icon was small and not noticeable found out after asking someone Improvement of student participation so the performance is a true representation of the tutor s performance with their class otherwise it is worthless needs to be compulsory or predictable low return rate provides innacurate percentages no release of grades until SETs are completed lack of incentive for students to complete having to log in makes it less likely that they will provide feedback potential distortion of results Allowing casual staff the ability to tweak their forms Automatic pre population of support staff as viewers for certain topics would be useful though feel am a little lazy for trying to avoid doing this myself From my end can t see any postive aspects Only a few students did the survey and haven t been able to check thir results because can t find out how to do it Better training for pleaple like me is essential Online version does not seem to have any advantage over the paper one other than saving paper A better response rate would have been achieved by still using the paper version or allowing for some combination of the two Planning Services Unit Page 11 of 23 43 44 45 46 47 48 49 50 How to improve the response rate 52 53 54 55 haven t received last semesters online set results
13. of course not very useful therefore hold on to my belief that paper surveys are going to be more effective suggest some survey questions that ask the students to reflect on their contributions to the learning process and questions that ask about their motivations expectations and reasons for undertaking higher education Also how much time do they expect to spend on preparation for tutorials Make the questions for teaching staff more explicitly different that those for topic evaluation Perhaps even different layout to enhance the difference The group of generic questions is sometimes limiting why shoud academics topic recive a score for cultural considerations whn stduent are interpreting this as having theory or practice on a particular culture we may know what it means but stduetns do not The SETs are used by academics and also be the univeristy as evidence of qulaity teaching this naturally could be debated With the SETs on line most of the PTI lecturers received very low numbers eg m out of 25 only 3 8 actually evaluated the lecturer In other cases 140 rather than the 25 evaluated the lecturer There are only a few questions for Planning Services Unit Page 14 of 23 SET Feedback Survey Results 2009 clinical SETs and although sending them to one of the staff they stated that we can only put in suggestion for possible groups of questions or questions from the DEAN Woudl strongly recommned that if the service cannot gu
14. question addressing something that has been trialled in the topic still don t understand who the comments write in response go to and how the process continues after the students have finished contributing This part of the process could be improved In my mind the biggest issue is how do we increase student participation don t think this is related to the system as such but i have noticed a significant reduction in the percentage of students who now return SETS a much higher complete paper based SETs than computer based SETS and at times the reply rate is not high enough to gain and useful feedback was listed as topic co ordinator in topics am not co ordinating and not listed as co ordinator for topics am co ordinating The source information used by the system clearly needs to be better updated and the need for this to be done may need to be streamlined The link provided in the email sent to students in one of my topics did not function then lost the chance for feedback from those students at the point expected them to feel mopst motivated to respond ie immediately after clases ended the questionnaire became avaiable and they received their first email about it with my recent encouragement to complete This was quickly fixed but too late to get those students back into the pool of respondents had surveys for a staff member in a different faculty appear on my page on the portal suddenly and for no reason could understand
15. responsive to the issues experienced namely e versions of SETS were organised without my consent moreover had already completed paper based versions Talitha responds to emails quickly Staff in the Planning Services Unit have always been extremely helpful and given an excellent level of services All staff from this area have given good advice and sought to deliver data that is fit for purpose Staff have often met additional demands fo data upon request and have complied with requests for information at very short notice Reports have at all times been received on time or before the due date Where any variation to scope of work timelines chan not be easily managed these have been satisfactorily communicated Easy and rapid generation of standard SETs Staff contacted me when had set my SET survey up incorrectly and then fixed the problem for me Clear information about which surveys are running Quick responses to questions concerns Happy to help and knew what they were doing Office staff supporting the School appeared to have been trained well and were able to deal with all my queries have no idea about SETs and would love to have the opportunity to do sets in rural sites for the 10 clinical educators who supervise and me Reminder emails relatively easy to use no need for a lot of support Support training and services have been v good do need administrate the survey by the staff
16. staff together in the questions about materials tutorials assessment and feedback If we take these matters seriously as part of teaching we should make individual lecturers accountable for their individual work That requires giving students the opportunity where applicable to evaluate identified parts of the topic separately Student demographic data A few years ago a graduate entry student with a background in psychology wrote to me and said she was mystified that the student evaluations did not require students to give their age She thought that it would make the data more meaningful if we could disaggregate younger and older studentsa responses am inclined to think that anything that enables us to get some insight as to who is having relatively positive and negative and strong and weak reactions to our work would be a help In addition to other kinds of demographic data such as gender and enrolment pattern there are a number of other details of individual students that could be useful in different contexts for example J the grade the student expects to get in the topic 1 the proportion of classes the student attended the proportion of classes prior to which the student completed the preparation tasks ff whether the student has completed prerequisites or had them waived ff whether the student ever consulted a member of staff in his or her consultation times and if so which member s and how many times i whethe
17. the survey please contact Talitha Bennett Evaluations Analyst Planning Services Unit on 08 8201 3727 or by email to talitha bennett flinders edu au Thank you for contributing Please note that feedback is not anonymous but will be kept confidential Service How would you rate aspects of the SET service and support from the Planning Services Unit Very Poor Poor Average Good Very Good Don t Know 1 Communication and information dissemination 2 Training opportunities 3 Support in using the system 4 Overall service and support OOOO OOOO OOOO OOOO OOOO OOOO Service and support 5 Please comment on best aspects of support or services 6 Please suggest any service or support improvements System Usability How would you rate the following aspects of the SET system itself Very Poor Poor Average Good Very Good Don t Know 7 Interface and ease of use O O O O O O 8 Functionality and flexibility O O O O O O 9 Overall rating O O O O O O 10 What are the best aspects of the new Flinders SET system 11 Do you have any suggestions for improvement to the new SET system The Instruments Questions 12 Building on the work in 2007 by Mandy Price and the Working Group we are interested in your thoughts on the instruments questions Planning Services Unit Page 22 of 23 SET Feedback Survey Results 2009 Appendix D Student SET Feedback Survey Questions Welcome to the
18. were invited through a notice on the Student Portal around exam time e 85 of respondents had broadly agreed with the online SET being an easy to use survey system Table 2 1 e 86 preferred to respond to the SET survey online as opposed to the paper survey in class Table 2 2 e There were 22 students who commented on any aspect of the SET survey Some students mentioned they liked the ability to complete the SET in their own time and with somewhat more anonymity ie lecturers were unable to recognise handwriting It was unclear to some students as to whether the online system was anonymous or not as they needed to log in with their FAN One student suggested having a reminder to complete the SET by SMS It was also suggested that students should be able to give feedback on different teachers of the same topic separately rather than giving an aggregated response for multiple teachers of the same topic Appendix B Table 2 1 Student SET Feedback Ease of Use Rating Number of Respondents and Very Poor Average Good Very Total poor good responses i The online Student Evaluation 2 0 3 12 17 34 2 29 of Teaching survey system is 5 9 0 8 8 35 3 50 0 100 0 5 9 85 3 easy to use Table 2 2 Students preferred method of answering the SET ii would prefer to answer SET surveys a on paper in class 5 b on line via Internet in my own time 31 c no preference 0 Total Responses 100 0 Pl
19. ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate Tutors in Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open blank Online Error No Planning Services Unit Page 10 of 23 18 19 20 Having to creat new SETS for evey tutor demonstrator is hopeless it needs a much better system 22 23 24 25 26 27 28 29 30 31 32 Incentives to improve response rate 34 35 36 37 38 39 40 41 42 SET Feedback Survey Results 2009 Survey Alternative This survey cannot be deleted because it is already open blank Online Error No Survey Alternative This survey cannot be deleted because it is already open Don t allow paper versions guess my only concern is that the number of respondents is lower than when SETs were collected manually and this affects the representativeness of the SETs Alerting Head of AUO regularly to the number of SETs completed available in the reporting function When you have 20 or more SET to organise for your cohort it seems a tedious process to repeat the same information in preparing for a SET i e a form c which requires me to complete the same questioins and fill in the blanks for each class rather thanhaving a template that then modify by simply including the list of names to be eval
20. ARCH1001 Online Closed This survey cannot be deleted because it is already open Evaluate Mark Staniforth as a Lecturer in Topic ARCH1001 Online Closed This survey cannot be deleted because it is already open Michael Diplock as Coordinator in topic ARCH1001 Online Closed This survey cannot be deleted because it is already open Evaluate Heather Burke as a Lecturer in Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate Michael Diplock as a Lecturer in Topic ARCH1001 Online Closed This survey cannot be deleted because it is already open Evaluate Tutors in Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist No Survey Report Viewer This survey cannot be deleted because it is already open Evaluate Tutors in Topic ARCH1001 Online In progress Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate Tutors in Topic ARCH1001 Online In progress Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate Tutors in Topic ARCH1001 Online In progress Multiple Surveys With Same Description Exist No Survey Report Viewer This survey cannot be deleted because it is already open Evaluate Demonstrators in Topic ARCH1001 Online In progress This survey cannot be deleted because it is already open Evaluate Tutors in Topic
21. RCH1001 Ethics Component Online Closed This survey cannot be deleted because it is already open Evaluate the Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate the Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate the Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate Heather Burke as a Lecturer in Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate Lecturers in Topic ARCH1001 Paper In progress No Survey Report Viewer This survey cannot be deleted because it is already open Evaluate Katy Hasenohr as a Lecturer in Topic ARCH1001 Paper Pending This survey cannot be deleted because it is already open Evaluate Amy Burke as a Lecturer in Topic ARCH1001 Paper Pending This survey cannot be deleted because it is already open Evaluate Michael diplock as a lecturer in topic ARCH1001 Paper In progress This survey cannot be deleted because it is already open Evaluate Heather Burke as a Lecturer in Topic ARCH1001 Paper In progress Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate Heather Bu as a Lecturer in Topic
22. SET Feedback Survey for Students October 2009 The survey will take about 5 minutes If there are any questions about completing the survey please contact Talitha Bennett Evaluations Analyst Planning Services Unit on 08 8201 3727 or by email to talitha bennett flinders edu au Thank you for contributing Please note that feedback is not anonymous but will be kept confidential Student Evaluation of Teaching system How would you rate aspects of the SET service and support from the Planning Services Unit Strongly Disagree Undecided Agree Strongly NA Disagree Agree Haven t used i The online Student Evaluation of Teaching survey system is easy to use O O O O O O ii would prefer to answer SET surveys O a on paperin class O b on line via Internet in my own time O c no preference iii Please comment on ANY aspects of the Student Evaluation of Teaching survey process paper surveys online surveys amount of surveying the questions anything at all to do with SETs Planning Services Unit Page 23 of 23
23. Survey Results 2009 Appendix B Comments by Students Student responses to the question Please comment on ANY aspects of the Student Evaluation of Teaching survey process 1 2 3 10 11 12 13 14 15 16 17 18 Some questions find irrelevant and have had to do surveys of lecturers who have never lectured me before in the topic think the SETs are good as they can help the uni help the future students maybe more space for general comments as sometimes what you want to say doesn t always relate to the more generic questions often asked Several of my friends forgot to do the online ones until it was too late Would it be possible to do an sms reminder or another email before the time is up Is quality of feedback diminished because of the apathy of students in not responding Are fewer negitive responses easier to write off than a whole class trend have chosen not to complete any student evaluations this semester because not all of my lecturers have allowed electronic submission The underlying feeling in the student body from speaking to other students is that poor teaching staff do paper surveys in class themselves and remove poor evaluations and to intimidate those students who may be inclined to give a bad review There was also a suggestion from a senior staff member that this is done to identify through hand writting which students have provided poor assessment reviews In addition havin
24. aff responses to the question Do you have any suggestions for improvement to the new SET system 1 PON 10 11 12 13 write to enquire if the online SET system has been changed in light of the clear failure to obtain adequate response rates As you should know by now by switching from the old peer administered system you have effectively halved response rates thus impacting on the validity of the results This is actually a very great disadvantage to early career academics as SETS are always in the mix when academic teaching positions are available at any University They are also one of the few rewards especially in the current climate where part time teachers are underpaid Response rates under 50 decrease the value of a student evaluation of teaching However there are mechanisms you can instigate to return the response rates to their previous typical figures of at least 70 Simply make the SET part of the course requirement for satisfactory completion of any topic University wide in the same manner that you might have a library skills test as mandatory task for all undergraduates etcetera If all students know that SETS are part of completing the course then they will complete them and just like the compulsory voting system they can say whatever they like as long as they turn up to the polling booth This would seem a reasonable and far more thorough means of evaluating our teaching and highly cost effective It i
25. and therefore the correct answer would be a not applicablea This is reason enough Planning Services Unit Page 15 of 23 10 11 12 13 14 15 16 17 18 These issues are already covered above 20 21 22 23 SET Feedback Survey Results 2009 why such a question has no place on the compulsory list In addition experience shows that students do not necessarily answer a not applicablea but give an evaluation which suggests that they use the question as a cipher for expressing a view about some other matter We can only speculate what that matter is in any given case which is further reason to see the responses to the question as meaningless Compulsory topic question 1 There should be no question about whether topics are a relevanta This word is often misused in this way but there is no excuse for the University to fall into the trap of using it as if it were interchangeable with a of significancea A topic cannot be a relevanta in the abstract it must be relevant to something Unless we know what a somethinga students have in mind when they answer this question the results are meaningless Either that or we assume that the student interpreted a relevanta as meaning a of significancea in which case they are not frankly always in a position to make a judgment At the very least this question should be made non compulsory don t know what an
26. anning Services Unit Page 3 of 23 SET Feedback Survey Results 2009 Appendix A Comments by Staff Part 1 Staff responses to the question Please comment on best aspects of support or services No one knowing what was going on Staff in the faculaty did not have training Training should never be necessary The system MUST be constructed so as to be entirely transparent Availability of access Clear instructions It wasn t that hard so needed little training find that the system is not very intuitive or easy to access Prompt answer to emails Ultimately this is a simple system and the on line suppot promtps could be much better Don t know Didn t have time to use them Timely and knowledgeable general feeling of desire to help There were none Being available at the end of the phone in case of difficulty is great The system is very easy to use and even though we didn t have proper access at the training session found the session to be really easy to follow and haven t really needed any help since Queries were answered with good speed and informative Individual support for me as Head of AOU to understand the new SETs and its reports Training sessions were good very good support in using the system SIMPLICITY if this system eventually works well students ability to fill these out at their leisure will be a great advantage Email notifications and training My queries have b
27. arantee that only the students from the lecturers class see teh request for a SET evaluation that this go back to a pen and paper system Think a meeage needs to be sent to the uni HODs etc that 2009 SETs may not be vaid or relaiabel and therefore shoudl not be used in an evaluation as the service cannot be certain of wheher the apporpriate group of stduents ealuated the appropraite person 8 Many of the questions are just plain stupid Stupid in the sense that they are pointless and will not elicit any useful information For example a student s ego will largely determine how they answer the following questions understood the concepts presented in this topic This topic helped me develop my thinking skills And then there is the question of I understood the assessment requirements of the topic This is nothing more than an invitation to students to say whether they LIKED the assessment criteria And could go on But why bother 9 a Teachinga and a topicsa Student evaluations of teaching currently distinguish unjustifiably between a teachinga and a topicsa Many of the activities that are evaluated as part of a topicsa are actually integral to teaching for example choosing materials setting the pace assessing and providing feedback The distinction as built into the student evaluation structure has a number of unfortunate consequences First it perpetuates a a chalk and talka view o
28. arly the two are connected The manner in which SETS are now conducted means that in the main only students who have some point to make will bother to complete the SETS Almost certainly this point is more likely to be some sort of criticism of the topic lecturer than a positive comment Unless you can work out some way to ensure a meaningful and representative sample then would suggest that you dump the whole new SET process In the past have administered the SETs in class and had better than 95 response Under this system it was about 25 It needs to improve dramatically given that staff need these documents for tenure and promotion applications taught across three topics in Semester 1 2009 but only one of those got evaluated was given the impression that these would be set up online but my students in two topics were not sent a questionnaire For this reason have almost no SETS results and this will look bad for me on my performance review did review my questionnaires and believed that had done everything required for my topics to be evaluated Very disappointed You might improve your response rates by incentivising students e g offering to enter them into a prize draw if they complete a questionnaire Curtin Uni does this successfully had no feedback at all on research degree supervisions Again it seems that my students were not contacted Students do not complete it so its hopeless please get rid of it and return to th
29. art 5 Table 1 1 Staff rating of SET service and support from the Planning Services Unit Number of Respondents and Broad Broad Very Poor Average Good Very Total dissatisfaction satisfaction poor good responses very poor good poor very good 1 Communication and 6 10 31 39 54 140 6 93 information dissemination 4 3 7 1 22 1 27 9 38 6 100 0 11 4 66 4 iti 1 1 2 Training opportunities 4 7 25 35 39 110 1 74 3 6 6 4 22 7 31 8 35 5 100 0 10 0 67 3 3 Support in using the system 4 12 29 27 52 124 16 79 3 2 9 7 23 4 21 8 41 9 28 2 12 9 63 7 4 Overall service and support 4 14 25 34 52 129 1 86 3 1 10 9 19 4 26 4 40 3 22 5 14 0 66 7 Planning Services Unit Page 2 of 23 SET Feedback Survey Results 2009 Table 1 2 Staff rating of the SET System itself Number of Respondents and Broad Broad Very Poor Average Good Very Total dissatisfaction satisfaction poor good responses very poor good 7 Interface and ease of use 10 45 38 39 148 26 77 6 8 26 4 100 0 17 6 52 0 8 Functionality and flexibility 13 41 42 31 146 32 73 9 Overall rating 11 46 44 32 146 24 76 7 5 21 9 100 0 Student SET Feedback e There were 36 students who responded to at least one question of the Student SET Feedback Survey The survey was open to all students however no emails were sent to students about the SET Feedback survey Instead students
30. cturing only etc Make it far more user friendly and am NOT computer illiterate Each PTI to do their own SET Have a proviso that allows duplication of SETs ie the same for Adelaide as Renmark do not think it makes sense for students who do not attend class who miss for example the last four classes of a term and who are clearly failing to be able to assess a topic Return to handing out the questionnaires during a lecture and giving the students 5 minutes to fill them in virtually none bothered to do the on line surveys in my topics The present system will mostly get responses only from those irate enought to login so the results are intrinsically biassed The old system provided the opportuntiy to survey a larger proportion of the students in the class The new system where student select to do results in a lot lower sample number Please attempt to remedy this some how Perhaps the ability for departmental administrators who organise these SETs in part for others to be able to change topic coordinators etc on the survey schedule if need be Also the ability to delete a survey setup after say a paper SET has been generated in error Would save asking Talitha to do this Maintain the option of paper based versions However it should not take 5 months or thereabouts to get an evaluation Why is it not possible to enter additions to the free form discussions This precludes one of the most useful aspects of the exercise to customise a
31. e previous system that works We need meaningfull feeback Make it compulsory for students to give feedback Planning Services Unit Page 9 of 23 SET Feedback Survey Results 2009 14 Although the on line administering process may well be easier although the person typing the comments in might not agree it reduces even further the number of students who respond and although didn t see it in my own results may well lead to only students with polarised views answering That is those who are aggrieved and those who are particularly pleased And in fact more likely to be the aggrieved remember watching a student who I had given an incredible amount of time and support to during the semester walk away from spending moments filling out a paper copy placed in front of their nose what chance them filling in on line forms for multiple topics lecturers and tutors 15 the fact that in previous years i have had almost 80 response rates for all of my classes Now i have 20 response rates due to the fact that it cannot be enforced it can only be encouraged am very much into multimedia technology and learning but like high response rates for feedback otherwise it becomes meaningless 16 Less students fill this out online than if given to them or sent to them by mail 17 don t like the automatic addition of a form for every general staff member who has access to the FLO site Even though these aren t activated in order to find the
32. een dealt with very quickly and in great detail Ease and speed of return of SETS to staff went to the training held at FMC There were only 2 of us there which thnk was because it was held during semester break but the trainng was helpful Prompt response to request for SET service Knowledgeable staff Have not needed to use the services or support Support for using the system was given lots of assistance did not use them worked it out myself havent had to interact with staff from the area however training was very clear and well presented ls automated and saves doing survey in a lecture It s good to receive feedback It was presented clearly No problems for me plenty of reminders was good easy access to reports Haven t yet used PSU support used to spend 2 days each semester preparing printing collating and posting paper copies of SETs to post to external students can now do it electronically in 2 hours Talitha Bennett is great Whenever have had to ask her for help she has been quick and friendly NA one of our department s admin staff set up our online SETs for us immediacy Promptness in replying My questions were dealt with quickly quick response over the phone Was unsure how to access reports Called Tahlia who got back to me immediately by email with instructions on how to use the SET Automatic completion of the forms Enthusiatic people
33. ernal students topics with low enrolments eg theses remain a problem regarding anonymous feedback The system generates set surveys for partial topics eg the first 6 units of a 24 unit Masters thesis which don t make any sense to survey Summary of students comments to be incuded in SET results don t believe that we are getting a full representation of student comments and evaluations have a lower level of return than did with paper SETS Students must be inundated with reviews and likely are ignoring most of them still believe that the supervisor should be notified of his staff SETs One of the questions is about working independently Of course that is an important learning outcome However think that working in a groups is also important and mirrors an aspect of life and employment would like to see a questions about group work in the standard questions More clinical teaching and learning questions It will be good if we can have the option of entering customised questions rather than clicking one of the options in the extended set of questions Need to 1 To be able to transfer sets of questions from one topic to another topic It is tedious to have to add the additional questions to many topics 2 Separate SETS for different supervisors Supervisor s names are not listed on enrolment So for example show up as the supervisor for a large number of students who are enrolled in topics coordinate but where am not the indi
34. ery low level of reply rate Need flexbility to implement department appropriate questions allow small sections to use paper our return of surveys was high for many years and now numbers are so low that scores are no longer significant ss comments have also reduced and students written feedback is invaluable and helpful There needs something to be done to encourage first year students to participate We had only 18 percent responses in the SETs in first semester YES Students get confused wi th the rating system and have been known to do it back to front with the numbering Isthere any way to alert them to look carefully as to whether it is 1 7 OR 7 1 compulsory SETs ie grades are not released until SETS are completed to improve response rate Because of extremely low response rates the survey results are of no practical value Make the responses compulsory for students or go back to the paper system The SET system is open too long in first semester it was open way past end of semester poor student use already on holidays etc Print version still required for administration in hte classroom as email response is varied and mostly poor Yes Get rid of it A voluntary web system will only get comment from students that are highly motivated either for or against the topic Better follow up found it difficult to find the right combination of class type and questions for my particular purpose am teaching a large lecture clas
35. eted or not I ve tried a couple of times to learn the system but gave up after failing entirely to understand even the simplest tasks The answer is NOT training A simple task such as setting SETs should be completely transparent and require very little time without training could write pages and pages on this issue but have other things to do would rather do sets entirely by hand than use the current system In terms of each particular SET topic it is very good My feedback relates to my use of the overall system find reporting not always completely intuitive but you get used to it it is good to get some feed back about aspects of teaching Greener The best is also the worse that is given to students on line There aren t any The new SET system has made SETS even more meaningless than they already were and that s saying something less paperwork to handle Do not have to do SETS in classes Not having to photocopy the forms are the best aspect well the only aspect that impinges really as staff don t really have to do too much Staff are able to maintain their records electronically Its extensive reporting functionality for Head of AOU Online is great its online so external students can send in ther evaluations easy to use clear interface Ease of use and good to get back comments with minimum of effort the electronic system has great potential yet to be realised Doing it online al
36. ey are able to recognise handwriting or match specific comments to experiences or events so I m curious to know whether the online surveys are realise that the system tracks whether an individual has filled out the appropriate SETs but can the instructors actually access that tracking information prefer online surveys as can do it in my own time have privacy and don t feel pressured by time constraints or other peoples observations A few people in my course had not received notification and so have not filled in any feedback this year until a paper version was presented last week The current Evaluation specifically refers to the lecturer but not to the tutor who is sometimes a different person It might be useful to include a question about tutorials SETs have not been carried out for every topic in the past which was disappointing believe that surveying should be carried out on all teaching staff so that students have the opportunity to evaluate staff and there are some excellent teaching staff which students have not had the opportunity to comment on Paper surveys would be best as this would guarantee students that all constructive feedback positive negative remain anonymous Planning Services Unit Page 21 of 23 SET Feedback Survey Results 2009 Appendix C Staff SET Feedback Survey Questions Welcome to the SET Feedback Survey September 2009 The survey will take about 10 minutes If there are any questions about completing
37. f what teaching is and therefore a passive model for learning That is the 4 teachinga questions seem to focus on what the staff member a deliversa to students in the classroom rather than incorporating those activities that go on before and after the classroom that are arguably even more important to the studenta s learning experience Elevating classroom activities in this way does not sit well with the Universitya s goal of producing independent learners Second it makes it impossible to disaggregate some things that need disaggregating if we are to have meaningful evaluations of teaching activities such as choice of materials and assessment For example in large enrolment topics different studentsa work would normally be marked by different markers who would not necessarily be their tutors If we see the provision of feedback on assessment as integral to good teaching practice every individual marker should receive an individual evaluation Yet the current practice is to lump all the markers together in a 2 topica question about adequacy of feedback Similarly many topics have two or more lecturers each of whom is responsible for a discrete part of the topic The responsible staff member would normally select the readings for that part of the topic set tutorial questions brief tutors set any assessment arising and brief markers perhaps even draw up a feedback sheet Yet once again we lump all academic
38. g just completed Defamation Law am amazed this topic does not allow anonymous SETS especially given we have studied cases of students being sued for defamation for giving poor reviews of lecturers will continue to refuse to do SETS until a uniform policy is adhered to know of many other students would have agreed to adopt this policy found it misleading when asked to complete an evaluation when both the lecturer and the tutor were cited as tutors At the time thought had to do both evaluations but now realize the lecturer was also a tutor as a few days later then was notified to complete evaluations for the topic and the lecturer Surely it would be better if they could all be sent so they appear on the same day or better clarification given While don t mind doing evaluations on line hear complaints from younger students including my daughter that they find they have time to do the evaluations in class but don t seem to make time to do them on line and therefore quite often do not fill them in despite having some worthwhile comments about the lecturer tutor topic to make and that should be made Online surveys are easy still get to evaluate even if you miss the last lecture It might be helpful on the paper surveys to put an appropriate place to return them received some at a start of a lecture and didn t get to filling it out until a few days later wasn t sure where was supposed to drop it off luckily an opportunity arose to
39. i can t get that excited about it really online reports more timely The electronic option is a time saver and a much greener option If the purpose of SET is reliability and validity then the move to an online data input has been a spectacular failure convenience Planning Services Unit Page 7 of 23 SET Feedback Survey Results 2009 set up for me Can t see any That it is available electoinically and can get better Easy to navigate around takes a little while to get used to but generally fairly transparent No paper involved Ease of use doesn t waste class time quick and saves paper don t have to run off SETs or take in class time for them to be completed Quicker less paperwork stotage of outdated SETs Full online service and opportunity for students to provide evaluation feedback on teaching great to be able to access online and share with Head of school It is online and automatic for topics which reduces the administrative burden on teaching staff NOTE haven t set them up myself Less time in class for students to fill out as they get really board towards end of semester needing to fill them all out in the same week The fact that its on line like the documentation of the reports and how things are grouped together Access available for external students Functionality and flexibility Interface and ease of use Once you get the language sorted
40. ing understand that the students unlike when the sets were handed out in class do not respond to the degree that they did when the sets were in hard copy form and handed out to each student How can this without a 100 participation rate of the students being taught by a particualar tutor represent the tutor s performance Allowing casual staff the ability to tweak their own surveys would be even better particularly as we have a number of casual staff annually coordinating specific topics More exhaustive and personalised training There needs to be some way to improve student responses to online SETs My impression is that students are participating less due to less of a forced response methodology than the old way There needs to be particular emphasis training on how we get good number to participate I was sent SET questionnaires for topics and could not easily delete any had to seek help to deal with it Provide user help documenatation as promised There were problems with the system during the training Maybe provide an on line step by step training tool not just a test site wondered how students were notified by uni email guess My return rates were very low In future will advise students on FLO as well Is there a standard way in which we can do this so that we are not biasing the outcomes Some sort of standard statement Need flexibility so faculties and departments can use their own questions wou
41. instrument is or what Mandy Price and the working group did in 2007 The questions were the same as in the old system do not know about the 2007 work referred to above have deep reservations about the validity of the content of the SET process usually get scores of an average of about 6 5 so what What does that mean Are these students really learning anything more or better under my tutelage How does that help me with my teaching The system encourages popularity not improved learning at all have never had my SETs mentioned to me by a supervisor in any way they seem to be largely a waste of time Questions are not the issue it is the low response rates despite much encouragement Some questions are not so useful Inevitably those who do well provide positive feedback and those who do less well provide negative feedback The median rather than the mean responses should be used for consideration in the promotion Question 10 is entirely subjective e g realistic appreciation of time and effort to complete work academics should not be penalised for using the penalties they should give under the student policies if people hand in work late There should be a question about academic s communication skills given the importance of this to teaching These are all fine have no problems with them Seems some question about topics that only have a very small number of students They don t really need to have SET s done automatically
42. k questions but couldn t find them anywhere or they were only available in the topic version but wanted to ask it for the teacher SET or vice versa Unfortunately can t remember now what they were It took me many months to realise there were comments to download It wasn t obvious Can t they be included at the end of the bar graphs in the report would prefer students to complete the SET on the last day of the course There is a tendency for students not to complete the forms after classes have ceased As academics we should be able to see the responses that are being input immediately the survey becomes live Make it more obvious as to how to access student comments It should be done automatically There needs to be a better way to administer SETs with very large student cohorts e g 500 across about 20 tutorials this needs to be set up automatically to ensure that it is being done the same way across all topics improving the validity of the results students need to be sent individual emails advising them of EACH topic AND tutorial they need to evaluate and EASY access to a link for them to complete the survey At the moment it is too confusing and have students commenting on other tutors in my survey etc so can t be sure really what the results are indicating suppose could just say that all the low results are confused ones and all the high marks are clearly about me do not like the fixed question structure as most do
43. ld have appreciated an organised training seminar instead of the trainers not having an idea of how academics were to use it id like a document with SETS in pdf form so that can save these on my computer i dont recall the training covering this and i d like to have this information im not sure if the system can do this a word document would be great too prefer evaluations where all students are handed SETs so that responses cover a broader range of people not only those who complain or take the initiative to complete a SET Sometimes being handed SETs makes students stop and think about responding whereas electroinic SETs can be forgotten if optional Apalling return rate even worse than face to face There is a need to emphaisie the importance to students to fill them in and to chase them up Really disappointed online help or guidelines would be useful I m still not convinced we will get sufficient numbers to evaluate this way would be interested to know the percentages Offer good online tutorials on the system one s that ussume NO previous knowledge It would be good to have someone talk me though what i had done worn so that could learn to do it myself next time would like the user manual to be available It is not fair to support staff on SET to have to field ignorant questiions because they are my onoly source of information having missed whatever training happened in s1 when I was on leave don t know how to acces
44. lows for easy and timely access of my choosing For the academics the best aspects must be that the surveys are on line and they do not have to rely on paper or having to do the surveys during class time For the administrative staff that had nothing to do with surveys before they became on line the new SET systems means more work because we are expected to make it part of our workload to assist academis that do not attend any training It can be accessed by staff quickly There is no particularly good aspect It is slightly more diffiuclt to use than the previous system Still it is not particularly bad either It is an easier way to set up the SET and don t need to organise someone to administer it have the potential of getting feedback from students who didn t attend the last lecture The free text comments are all collated in the feedback summary I think got better more metered responses from students Printed comments by the students more legible document and better presentation for submission as data e g for promotion awards etc Ease of use Quick ease of setting them up Love that it automatically is set out which save us doing mail outs That it is automated Save us from printing hard copies and administering the survey Good to get feedback Itis much better than paper based surveys Its now online that can access them and print off etc and they are all stored in the one convenient place
45. make it a compulsory aspect of their topics scroll down in the set form is confusing as there is also next page button underneath Some caracters are too small Perahaps some more user friendly language think it is disappointing that SETS are done after the students final assessment This could skew their response It seems that on line participation is lower Survey completion in our topics has been very low There needs to be some incentive for students to complete surveys Planning Services Unit Page 12 of 23 70 71 72 73 74 75 76 TT 78 79 80 81 82 83 84 SET Feedback Survey Results 2009 Currently only the topic coordinators can create surveys for their topic and often many other tutors or lecturers are involved It would be nice for teaching assistants to also be able to create their own surveys The new system has resulted in a dramatic reduction in student response rates in my experience do not believe the electronic system where students can ignore requests to respond is the way to go yes it is not at all clear just how to proceed For example logically if only have 6 9 students in a topic will select teacher of a small group but this is for tutors Conversely if am only lecturing in a topic some of the standard eval _ as a lecturer just don t apply Might be better to rename them for Tutors for Lecturers lectures and tutes for le
46. n the spot not in their own time on the internet if I m going to get any sort of varied or useful feedback Furthermore having a SET with only 4 responses out of 20 is not going to be useful for any of my future applications getting a very positive SET result from your students doesn t have much impact if only 4 of them have bothered to reply Generally when distributing paper SETs in the class you are bound to have some absent students so the numbers you receive won t be 100 accurate Nevertheless getting responses from 15 18 students in class is enormously more useful than having 4 20 students reply online In short am interested in hearing the students opinions and in reading their evaluations of my teaching methods the online SET form has not allowed me to get an accurate representation of what my class thinks and will not be able to use these online SETs to help me with any future teaching positions so as it stands the current SET is not doing the students or the teachers any service Subjects with multiple instructors don t always allow allow for providing feedback about all of them online surveys which is realy frustrating especially when one instructor is great and the other an oxygen thief never knew that SETs were optional for part time tutors shouldn t they be a compulsory thing Surely know that the paper surveys are NOT completely anonymous they can t be if they are handed to the instructor involved and th
47. per system The questions are generic and are manipulatable There is no capacity for individualized questions relavent to the defferent faculties and schools allow all staff to do their own The most important aspect is that response rates are very low so some way of encouraging students to use it is important Our response rates have gone from around 70 paper to 30 online We need a way of making sure the student only evaluates their tutor lecturer when multiple people teach in a topic Finally found the usability difficult when accessing the results the number of clicks to simply print off the surveys is painful We should just be able to print or save everything in one go Some of the questions possibly need revisiting re what students think the question is asking eg is the material culturally inclusive This should be rated highly in one of our Topics that addresses cultire and nutrition Some of the language used is counterintuitive large classes means small classes etc Make completion of the SETS a compusory requirement for completion of topics The response rate is alarmingly low Part 5 Staff responses to the question Building on the work in 2007 by Mandy Price and the Working Group we are interested in your thoughts on the instruments questions 1 Only 33 out of 80 students ie lt 50 completed the online SET When the SETS are done in person the response rate has never been less than 90 in any of my topic
48. ponsibility for the SETs to the teachers or academics raises issues around what is it exactly that admin staff of the university do to support teaching i think the questions are quite basic and not being an anonymous tool as it requires a sign in deters staff or students from disclosing honest and critical feedback on teaching learning and praxis There are a couple that seem pointless according to some of my students also feel some of the questions are a long bow re whether e g lecturers of an External topic have supported students ability to work independently There are other questions too that seem problematic As a generic set of questions they are fine Students dont appear to be completing their surveys a very low turnout online for my topics at least would like to see a comparison of completion rates between the good old paper version and this online one overall Are students reminded regualrly to complete their surveys Planning Services Unit Page 17 of 23 48 Pls see answer 11 50 51 52 53 54 55 56 57 58 59 60 61 62 use a computer lab and the ability to write my own questions would be very useful The question about 64 65 66 67 68 69 70 SET Feedback Survey Results 2009 Prior to making comment it would be good to have an example to review questions that are more suited to individually supervised topic theses etc are needed also for ext
49. r the student had any penalties imposed on assessment for lateness or over length f whether the student is undertaking an overload enrolment Identifying students On other occasions when have suggested that this would be desirable have met with the response that the provision of this kind of information could compromise confidentiality do not think this is necessarily the case in all cases in relation to all kinds of information and it would be possible to draw up a policy or a set of guidelines that indicated when it would and would not be appropriate to ask particular questions certainly believe that the kinds of questions involved in the dot points above would not tend to identify individuals However they do give an interesting and useful indication as to the studenta s level of engagement in the topic and the kinds of experience that might have shaped his or her views In any event think that the case for requiring students to provide information that theoretically could be used to identify them is much stronger than the theoretical possibility of some detriment to the integrity of the process in isolated instances Compulsory teaching question 9 A question that is desperately in need of change is the compulsory teaching question about whether the staff member respects studentsa views and opinions There are many teaching settings where a teacher has no opportunity to find out what studentsa views and opinions are
50. rt of the overall experience 2hr lecture too long try the workshop style encourage students participation Allows for feedback to be given when appropriate Feedback is still obtainable if the last lecture is missed if it s availible online Survey questions are very standard SETs are pointless and just lower standards A gist of the info gathered from the survey should be shared with the participating student Further outcomes or action taken should be reported to the participating student This type of ifeedback will motivate students to participate more constructively presently since there is no feedback and in my view students undertake these exercises half heartedly am a tutor in the English department so haven t filled out the SET form myself However wanted to fill in this survey because I ve found that the online survey is utterly useless as a tool for teachers for example have 20 students in one of my classes and only 4 of them have completed the online SET Planning Services Unit Page 20 of 23 19 20 21 22 SET Feedback Survey Results 2009 The feedback get from these students is not going to be helpful for future teaching because basically the 4 students that have bothered to fill out the SET form are the ambitious keen HD students who only have good things to say about the course need to be able to distribute the paper SETS to a class full of students who will be required to complete the form o
51. s If the Uni wants to continue with online SETs then it will need to realise that they are not capturing most students feedback The questions are fine However the response rate from the students was really poor and makes a mockery of the system The paper system works because students have to do it there and then otherwise it is one more thing for them to do The title above this question is a prime example of what makes this system incomprehensible What Instruments What questions You are writing as if the audience can read your mind How can we tell what you mean by an instrument presume the questions are the ones on the Sets and you are asking if we like the choices but this is not at all clear Really scrap the whole thing Get new people start over My next option on this survey is Save amp Exit If do this will this be sent automatically or not Maybe when press the button will find out but it would be nice to know ahead of time Another example of the kind of deficiency that pervades this system Sorry to be so negative but this system has really irritated me know this is not a question you ask but will answer it anyway While paper surveys are still possible you strongly encourage the online version did do the online version in first semester as a trial but found as expected a much lower response rate which makes the survey a lot less valid On one topic with 13 students enrolled received 1 one response which is
52. s to be taken into account Hence they are valuable in that sense and play a role Knowledge that would like to Know is to get some idea about which students fill in the survey In the past it was those who attended a lecture on a given day in some senses those who were the most conscientious Now it is hard to gauge who is filling out the survey I d like to get a sense of who is filling out the survey So questions about whether they regularly attend classes in person or watchi listen to recordings on FLO of course self report of such information may not be accurate assuming some self presentation biases may be evident Maybe even something that indicates FLO usage for all know this may not be permissible due to some privacy issues This partly revolves around the online nature of delivery know there are some students who rarely attend look on FLO from time to time but do not really put in a conscientious effort Having some indication whether or not these people are filling in the survey would be useful to know Questions are Ok As could have been foreseen the student response rate is down would be good to see how the electronic administration of the SETs has affected overall response rate across all topics at the uni Out of campus teaching needs SETs too Students from other campuses do not seem to reply How many people bother to fill in electronic surveys Unable to use without help from trained admin staff The question
53. s easy for any convenor to add one sentence to their course outline Not expecting everyone to know what to do On line tut Start over Get someone involved who understands how to make the basic tasks completely transparent The system should knock out child topics so that academics dont have to bother with them Academics should be given a choice of whether SETS are sent out or not within the functionality Not making continuous changes that require a new learning curve The system should include questions about the learning environment are classes overcrowded What teaching aids technologies are available to assist teachers have found the equipment to be outdated and not well maintaqined Find a way to encourage students to actually do it Much lower response rate for all my topics with this version and not being able to talk through differences between topic and lecturer tutor evaluations resulted in lower scores for some questions but reading the comments showed that the responses were off track If this cannot be connected to Student systems and therefore specific groups classes of students in a year level targeted than we need to go back to the pen and paper system Dump it Or is some other way solve the following problems Having looked at my SETS for the last semester immediately noticed two things 1 It is the worse response rate have ever had a Flinders 2 It is the worse set of results have ever had a Flinders Cle
54. s services or support Just know should do these improve the rate of replying from the students Ensure that SETeching are available to supervisorss and HoS automatically It would be good to have a way to generate SETs using the portal if you want to evaluate your teaching and you are not a topic coordinator wasn t aware of any training opportunities maybe i missed them student participation with electronic surveys is very poor do you have any suggestions for increasing this Planning Services Unit Page 6 of 23 SET Feedback Survey Results 2009 Part 3 Staff responses to the question What are the best aspects of the new cele SET System i w ONDA 21 22 23 It was fairly easy to set up once you knew what to do This system is the most horrible of its kind imaginable The directions are incomprehensible the words used to describe option are illiterate cannot tell when something has been done correctly or at all It takes much too long because it is incomprehensible Organising sets should be a completely trivial task Among all the task we have to perform managing sets must come extremely low on the list in terms of time allotted The whole point of the system is to make this important task nearly trivial Instead the new system requires infinite time because it is incomprehensible cannot understand any of the instructions cannot understand the options cannot tell when a task has been compl
55. s that has 7 separate tute groups give all the lectures but none of the tutes wanted to get feedback on the topic design overall lectures assessment and tutes and particularly on my lecture delivery Second bit of feedback I m hearing from mates at other unis that the return rate is less than 10 35 returns from a class of 500 enrolled If this is the case will we be returning to the former system so that sample sizes are relevant My comments would be that the response rate was lower as we expected at least for the 1st year and my results and the results of some of my colleagues were poorer possibly as a result of the fact that students had more anonymity and they could fill in the forms at any time i e perhaps when they were feeling particularly stressed and hating their lecturers for working them so hard am actually having a hard time coming to terms with the new SET system This year semester 1 for the 1st time in my casual career at FU had 4 people out of about 25 or so even respond to SET As far as providing feedback on my teaching this was pretty useless If can t see that students are responding to SET surveys then imagine they are not taking time out in their own time that is to be bothered with the survey had much better responses and therefore useful feedback on my teaching when we handed out paper surveys I m not sure what would improve response rates given this new system perhaps topic convenors could
56. s will always be bland and of limited use as they are the only questions that there will be agreement on Questions need to be specific to student groups distance students have no idea how to answer questions that relate to on campus issues when they are distance students who never visit the Planning Services Unit Page 18 of 23 SET Feedback Survey Results 2009 campus A NA response just doesn t seem to work Students must be advised that using obscenities on response sheets isn t appropriate 71 They seem to cover everything pretty well now 72 Very good No further comment 73 The questions need to be able to be created for specific purposes 74 It would be good to have a way to generate your own questions For example used votapedia in my lectures thus would of liked a question about whether the use of this system improved student s understanding of the lecture content The other big problem is that the answer rate is not as high as the paper based SETs During a lecture all students are targeted With an electronic system only students who feel passionate about the lecturer topic or who had a negative experience tend to reply wonder whether you get the average student 75 support maintaing the same questions to allow trend analysis over time 76 Some issues with lack of participation with electronic SETs Good that the paper oiption is still available 77 These are fine Planning Services Unit Page 19 of 23 SET Feedback
57. t to get back your written work Did you read the comments on your written work Were they helpful This would make it more difficult for lazy students to produce answers that have no basis in fact should say that as far as can tell most of my colleagues think the surveys are useless we would be better off having a distinguished professor in the field sitting in on lectures and looking at our materials and reviwing our marking Doing this every few years would eliminate the need for SETs It would be good to have the option to write our own questions or ask the unit to include certain questions Last semester wanted to ask questions but couldn t find them anywhere or they were only available in the topic version but wanted to ask it for the teacher SET or vice versa Unfortunately can t remember now what they were The scale of Undecided ie 4 7 is ambiguous It is possible that students check this box instead up not applicable thus lowering the score for that question ie the the student reads the question can t make up their minds if they agree or disagree because the question isn t really relevant so they select undecided When construct student surveys use the phrase adequate to mean 50 Even in this particular survey am filling out you have used the therm average to mean 50 am satisfied with the instruments and questions As mentioned earlier it is the timing of the SET completions by students
58. ter system meant that students didn t fill out any of my SETs The feedback is therefore useless More regular communication by email Planning Services Unit Page 5 of 23 17 23 24 25 26 27 28 29 30 31 32 Everything seems to be fine and so don t have any suggestions at the moment 34 SET Feedback Survey Results 2009 if the reply of the email would be quicker it woud be great Basic traing should apply for academic although there are some admin people to do the job the system was very difficult to use and unfriendly Creating parallel SETs for tutors was nigh on impossible It was faster and more efficient to use the old electronic paper based system Training for academics should be made compulsory as part of their administrative load of being a topic coordinator The online SETS encourage many students to not respond causing a polarisation of feedback Email prompt that SETs are available for viewing Cannot comment as above feel will need to be shown again next year so more sessions would be invaluable Could not find student comments to read last semester As someone using this system for distance teaching despite several requests with assistance receieved no individual assistance but lots of generic spam messages In the end gave up and don t know whether my students used the system or not As far as am aware there is no service provided specific to distance teach
59. that is a major problem Planning Services Unit Page 16 of 23 24 25 26 27 28 29 30 31 41 42 43 44 45 46 47 SET Feedback Survey Results 2009 think there are always loaded questions in such surveys As long as the overall quality question remains that is good In previous years have set some of my own questions to evaluate students use of various support materials provided Did not investigate if it was paossible this year as i used the standard forms Would like to be able to set some questions of my own Don t know see above Lack of communication over my inabililty to use the software left me unable to see whether there were questions relevant to the specific needs of distance students If this is truly lacking then would be happy to assist in trialling some questions for next year as this is a real problem for those of us who teach mostly in distance mode The new methodology means that the sample of respondents may be quite different to the sample under the old methodology This may mean that users of the SETs should be warned about making any simple comparisons of results under the old system with what might seem to be directly comparable results under the new system Compared to the past system find no problems nor any evident room for improvement at this stage Some questions are not making very good sense such as asking students is the teacher
60. training in humanities was brief but relevant a little confusing at times You re a phonecall away emailing contactable prompt replies and assistance have not had to use them yet so cant really comment availability ie always someone availble training sessions quick response to queries That it offers the students the opportuntiy to fill in the SETs on line Planning Services Unit have always been amazingly prompt and effecient in their response to any queries Planning Services Unit Page 4 of 23 SET Feedback Survey Results 2009 workshops are useful Convenience Each time phoned was assisted straight away Haven t really needed to ask for support have always been able to get through to somone if need help The promptness of the service Also important is the friendly staff and attitude of those in the Unit Thank you The best are the emails informing me about the status of the SET Human support good On line support lousy Staff are friendly and helpful in attending to my queries It appeared to be just as easy to use as previously Have received assistance in tracking down lost SETs It s difficult to single out anything special since in my opinion everything was good have only interacted with the person in our School who has set up the SETs so it is difficult for me to make a comment Reasonably quick responses to help requests found Talitha very helpful and
61. uate The questions need to be more flexible or even the questions that are mandated their need to be a little flexibility in the wording ie Question 7 Likert This person stimulated my interest in learning in this topic topic is too generic would like to see it modified so that could say case study or pathophysiolog which is more specific to the topic NURS3102 etc also there needs to be asystem so that students are looked out of other tutor evaluations had 321 return rate for one of my SETs there needs to be ways of setting it up so that you allow or disallow students survey instruments MUST be modified questions in each SET can t be changed some of questions are not suiable for the particular topic It will reduce the number of students who conduct the SET If it s too hard to fix suggest to have a comment box like the following on to allow students to write their own thoughts in make the generation of the SETs more user friendly and give staff the opportunity to identify who should be able to access the SET generators in a particular topic The system relies on Student Two and data is not always up to date so automatic surveys are often generated with errors in the information such as topic coordinator tutors in the teaching team etc The menu system is confusing and poorly designed It would be good to have the option to write our own questions or ask the unit to include certain questions Last semester wanted to as
62. very small number of correct forms there is an enormous amount of content to wade through In addition do not have authority as the Topic Co ordinator to delete any of them Here for example is the list of SET forms for my topic in S1 Since the topic is finished and the topic and teacher SETs have been done can you find out how to delete all of these other SETs from the system please can t do it all the delete buttons are greyed out All of them say they cannot be deleted but it seems ridiculous that so many have been created for no reason Plus cannot see the tutors sets that originally set up for Bob Mike and Michael do you think that someone else has deleted them Is part of the problem too many people having access to the SET system The list is Evaluate the Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate only part of the topic ARCH1001 Online In progress This survey cannot be deleted because it is already open blank Online Error No Survey Alternative This survey cannot be deleted because it is already open Evaluate the Topic ARCH1001 Online In progress Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate the Topic ARCH1001 Online Closed Multiple Surveys With Same Description Exist This survey cannot be deleted because it is already open Evaluate the Topic A
63. vidual supervisor of each student Industry placement topics where there are no lectures and no tutorials need a specific SET form In other words there is no topic content aside from the actual industry placement Tacking on some questions to a standard SET makes no sense have not used Sets since 2008 so cannot comment I am currently on secondment from teaching wil Ireturn in 2011 At the moment I don t have them am not sure that students use the electronic SET system think it is best to submit handwritten SETs to assure that they have been completed Otherwise we cannot measure student satisfaction in a reliable manner See above culturally inclusive teaching materials confuses the students particulalry liked the 1 page aggregated question rating for summarising overall performance satisfaction generally have doubts that the questions are useful in informing practice The questions focus on very general aspects of the topics teaching They also assess perceptions rather than whether or not students have learnt on the other hand don t think the latter is easily assessed and can t give any constructive advice on how to do this as part of a topic teaching evaluation All of the ideas have are only applicable within the teaching process and relate to specific aspects of what the students learn Nevertheless the SETs do at least inform one about how the students perceive their learning experience and this need
64. yet so can t comment 57 58 59 60 61 62 63 64 SET Feedback Survey Results 2009 Abandon online data entry how are we to get more participants The SETS don t carry a lot of weight when only a third or fewer of students participate not set up for both teaching and topic had an EXTREMELY LOW response rate using this system will return to using the paper version next time The problem with on line SETs is that the students do not fill in the questionnaire At least when they were handed out in the classroom all of those in attendance did so Lat semester had 6 out of 20 replies Sine the University relies on SET scores for a variety of measures of staff performance find the new measure of teaching to be poor for the sample size is likely to be small All of the undergrad topics in my area have internal and external offerings Having to set up both is a bit of a pain Can there be an option to combine internal and external evaluations if the lecturer requests this We use the same FLO framework and teach the group as a learning community 1 now it appears it is too late to run an official feedback for semester II 2008 2 mostly only very pleased students or very angry students feel the pressure or desire to fill in the questionnaire the old system was forcing the entire class to set time aside for this exercise now it is considered a waste of time unless there is a strong message to leave see the v

View the SET Feedback Survey Report

Contents

Download Pdf Manuals

Related Search

Related Contents