HERDSA 1996: Csete, Jones and Kwan - student feedback on teaching and staff appraisal

Student feedback on teaching and staff appraisal

Josie Csete, John Jones and Kam-por Kwan
Educational Development Unit
Hong Kong Polytechnic University

This paper describes the evaluation of a set of procedures designed to gather student feedback on teaching, which were introduced at Hong Kong Polytechnic University (PolyU) in 1995. The procedures were mandatory for all teaching staff, and involved a standardised Student Feedback Questionnaire (SFQ). Results from the SFQ are to be used in a Staff Appraisal system, designed to be "both developmental and judgmental" which was introduced at the same time. Feedback from staff was gathered via a questionnaire, and through individual and focus group interviews. Numerical results from the questionnaire suggested that staff were (surprisingly) positive toward the SFQ procedures. However, the qualitative interview data suggested that staff had some serious reservations; as a result some changes were made. The paper describes some of the issues that emerged from the staff feedback, and draws an analogy with student evaluation of teaching and the problems of relying exclusively upon numerical questionnaire data from students to rate teaching performance.

Introduction

Hong Kong Polytechnic University (PolyU) is the largest of the seven Universities in Hong Kong, with a permanent teaching staff of more than 1,000, and a headcount of approximately 20,000 students (13,000+ Full-time Equivalent). About half of the students are studying for degree and post-graduate qualifications, with the remainder enrolled mainly in Higher Diploma and Diploma programmes. There are 26 academic departments organised into six Faculties: Applied Science and Textiles; Business and Information Systems; Communication; Construction and Land Use; Engineering; and Health and Social Studies. PolyU achieved university status at the beginning of 1995.

In 1994 a decision was taken by senior management that a comprehensive and systematic system of individual staff appraisal was to be introduced at PolyU. This was to be carried out annually, and was to be "both developmental and judgmental". The appraisal centres around a formalised interview between staff member and appraiser, based upon an Interview Preparation document that is completed by the staff member prior to the interview. This document is designed to detail the achievements of the staff member over the previous year, and also to focus upon developmental plans. This system was introduced for the first time at the start of the 1995/96 academic year.

A further decision was taken by senior management that student feedback on teaching, using a standardised instrument and procedures, was to be part of the appraisal process. For many years student feedback on courses has been generated at PolyU. The course reporting system requires that student feedback should be taken into account in the course report. It has been the responsibility of departments and course teams to collect that feedback, in ways that they deem appropriate. A variety of instruments and procedures were used for that purpose. However, the requirement of the new Staff Appraisal system was for student feedback on the individual staff member - rather than the course. To accomplish this task, a Working Party was established in 1994. The major elements of the instrument and system that emerged were as follows.

A standardised, optical mark read questionnaire was designed to gather student feedback on teaching: this was entitled the Student Feedback Questionnaire, or SFQ. The SFQ consisted of three main Sections, as follows [1].
1. Three open-ended questions which asked students to comment on the "best" and the "worst" aspects of the staff members teaching, and for suggestions about how the teaching could be improved.
2. 18 standardised closed-response questions, divided into six Scales: Learning Outcomes; Interaction; Individual Help; Organisation and presentation; Motivation; Feedback. These questions were of the conventional type, and asked students to indicate their level of agreement, on a five-point scale, to statements such as:
  "I feel that I have learnt a lot from this staff member" [Learning Outcomes]
  "The staff member ensured that class discussion took place" [Interaction]
  etc.
3. A Section for the inclusion of optional closed-response questions that were to be added by the staff member, or department, concerned.
Normally, each staff member would gather feedback from students in two classes, in each academic year.
A "pilot study" should be carried out, in order to test the usefulness of the SFQ instrument and the administrative procedures associated with it. The evaluation should be used to generate data that could be used to modify the instrument and procedures.

(In addition there were requirements relating to the administration and processing of the SFQ aimed at maintaining confidentiality and consistenct)

These recommendations and requirements were adhered to by the Educational Development Unit (EDU) in the design of procedures for the administration and analysis of the SFQ, and for the reporting on the results obtained.

It is recommendation 3. above that forms the focus for the rest of this paper. As it turned out, the "pilot study" became a fairly full-blown administration of the SFQ across campus, and in all more than 1,200 class sets of data were produced from 20 of the 26 academic departments on campus. A full Report on the evaluation has been produced. (Csete et. al. 1995).

Procedures for evaluating the SFQ and its implementation

The evaluation study focused on four "domains", as outlined in Fig 1, below. For this paper, it is only the feedback from Academic staff - and the three associated "domains" indicated in Fig. 1 - that will be discussed. Feedback was gathered from academic staff in three ways.

A questionnaire survey was conducted of all academic staff in seven pilot departments, which were representative of the six Faculties on campus [2]. The decision was taken to sample departments, rather than survey all staff, so that efforts could realistically be made to follow up staff and ensure as good a response rate as possible. The questionnaire was distributed to all academic staff members in the seven pilot departments (N=286), and after two follow-up procedures, 212 completed questionnaires were received: a response rate of about 74%. The questionnaire itself was designed in the light of issues that emerged from a focus group interview (see below), and from concerns that had been expressed at Staff Appraisal training Workshops. It was divided into 5 parts to gather information about the following matters:
1. General Information on the staff member (5 closed items)
2. Guidelines for administering the SFQ (9 closed items)
3. The SFQ format, and the particular items (30 closed items)
4. The reporting and use of the SFQ results (7 closed items)
5. General comments (One open-ended question)
In addition to the closed items, there was room for staff to write open-ended comments in each section.

Figure 1: Domains, sources and methods of the evaluation

Source Domain

Administration guidelines and procedures The questionnaire instrument Reporting and use of the student feedback results Inputting and processing of data

Dept. administrative staff Focus group interview of dept. AA/EO from 7 pilot depts. -- Focus group interview of dept. AA/EO from 7 pilot depts. --

Academic staff Focus group interview of staff from BRE
Staff survey questionnaires to all staff members in the 7 pilot depts.
Individual interview of selected staff members from 6 pilot depts.
--

Students Focus group interview of students from 3 pilot depts. --

Empirical student feedback data -- Statistical analysis of data, including reliability test and covariance structure analysis of scales -- --

EDU staff involved -- -- -- Document analysis of EDU report

ITS staff involved -- -- -- Telephone interview of officer-in-charge

A focus-group interview was held with staff from one of the pilot departments. The interview was semi-structured and focused on the three "domains" indicated in Fig.1. As described in 1, above, this interview was useful in the design of the survey questionnaire.
Individual interviews with twenty-five staff from across the seven pilot departments were conducted. These staff were nominated by their respective heads of departments. Each interview (which typically lasted about an hour) was tape-recorded, transcribed, and translated into English where the staff member had wished the interview to be conducted in Cantonese.

Essentially then, there were three "kinds" of data sets generated.

The quantitative data that was generated from the "ticked boxes" of the survey questionnaire.
The qualitative data from the written open-ended comments on the survey form.
The qualitative data provided by the records from the individual and focus-group interviews.

In the next section, some of this data is described, together with selected issues which arose when it came to making recommendations based upon it.

Issues arising out of the evaluation data

It is worth emphasising that the evaluation was carried out on a "pilot" instrument and set of implementation procedures, within a set of pre-conditions that was non-negotiable. These "non-negotiables" were:

All staff were to collect student feedback on their teaching;
The feedback was to be collected via the same standardised instrument for all staff
The feedback was to be collected each year;
The results of the feedback were to be used in a Staff Appraisal exercise that was both developmental and judgmental;
The collection of data, analysis and reporting was to be done independent of the staff member concerned.

Given the "imposed" , and potentially threatening, nature of the SFQ exercise it was predicted by the authors that staff reaction would be critical of the procedures - even hostile. However, the quantitative data from the survey form generated a very surprising picture. Every one of the closed items suggested that most staff were happy with the status quo: implementation guidelines, the SFQ instrument itself, reporting procedures - everything!

In order to give the flavour of the data, the questionnaire Section on "Implementation Guidelines" will be described in a little detail. Staff members were asked to tick one of two boxes with respect to each of the items below. These boxes were: "Keep as it is" and "Changes required". (Staff then had the opportunity to write an open-ended comment about any of the items). The items, together with the proportion of staff [N = 212] ticking "Keep it as it is", is as below.

"The SFQ focuses on collecting feedback on individual staff members rather than the subject {i.e. paper} or course" [61.3%]
"Two classes are to be sampled for each staff member" [81%]
"Individual staff members have the right to choose the classes to be sampled" [87.3%]
"The evaluation is to be conducted each year" [76.1%]
"The SFQ is to be administered toward the end of a semester" [85.8%]
"Confidentiality in collecting, processing and reporting of data is necessary" [93.7%]
"The assembling, distribution and collection of SFQ forms are to be handled by the departmental administrative staff" [88.7%]
"The same standardised questionnaire is to be used for all staff members in all departments" [69%]

Thus, if the survey had been solely conducted via a quantitative, "tick a box" instrument only then the conclusion would have to have been that staff are happy with the implementation guidelines. However, some serious - and educationally valid - reservations were expressed by many of those who made written comment: those same reservations were expressed in some of the interviews.

For example, the first question - about the focus on individual staff - attracted 70 open-ended comments on the questionnaires that were returned. The fact that a third of all respondents took time to write comments is an indications of the need to provide opportunity for feedback beyond "ticking a box". The great majority of the comments can be grouped into two categories, as follows.

There is a need to focus on course, subject, teaching teams, and the overall structure of the programme of study, in addition to the individual teacher. Many departments and staff have been carrying out formative evaluations, designed to monitor the quality of courses and subjects for some time. This has been very valuable. In many ways the SFQ, with its judgmental focus on the individual teacher which is designed to inform the staff appraisal system, subverts this.
Teachers do not have total control of syllabuses or course and subject arrangements, though student reactions to these matters will inevitably colour their judgments of the quality of the teaching. Student feedback has to be interpreted in context.

Actual written comments that capture the essence of these two points are as follows.

"It is more important to focus on the quality of the subject or the course rather than individual staff members if it is for developmental purposes. Indication of individual staff members performances can be incorporated into a questionnaire based on evaluation of a subject and course".
"We need to know about course content, design and balance, about the assignments and their content and timing. We can change these things easily. Personality and interactive style are harder to change".
"The SFQ should aim to collect student feedback on both individual staff and the nature of the subject or course, and even the planning /timing of the subject, because sometimes it could be the problem of administration policy or the planning of the programme".

Again in the questionnaire Section on "Implementation Guidelines", the final question about the use of a standardised questionnaire for staff in all departments generated 53 open-ended responses, most of them suggesting that any valid student feedback questionnaire needs to be tailored to the culture of particular departments, and styles of teaching. Typical comments are the following.

"Individual departments got [sic] their own 'style' of teaching because of the course nature. Questionnaire should be specially designed to match with the department".
"Extra questions to allow for different style of delivery must be allowed"
"The aims and nature of a course may be so different to the others. It is not fair if the SFQ cannot reflect this to those courses which have different requirements in teaching methods or models"

Thus while a simple majority of staff are prepared to accept the status quo as being fair and reasonable, there are significant numbers (including some of those accepting the status quo) who are nevertheless uneasy about some aspects, and advance cogent arguments as to why the debate should continue. Consider, as additional examples, two further items from the questionnaire: this time from the Section on "Reporting and Use of SFQ Results".

The statement was: "University or departmental averages are not provided".

56.5% of respondents thought that this should continue to be the case. However, there were some significant counter-arguments, the main thrust of which are captured by the two following comments on the survey forms.

"Departmental averages may lend yardstick incentives for individual improvement"
"If averages were provided, this would allow staff to assess their own performance more clearly, i.e. give greater feedback".

Another questionnaire statement was: "How would you rate the usefulness of the SFQ results for judgmental (i.e. appraisal) purposes?"

Altogether, 61.3% of the staff responding ticked "Quite useful" or "Very useful". At the same time, there were 83 open-ended comments, almost all of them advising caution in the use of the SFQ data for judgmental purposes: the comments can be grouped into three themes. The first of these suggests that teaching (and student learning) is a complex activity, and that at best the SFQ feedback can only represent a simplistic and generalised indicator of that complexity. Associated comments related to the necessity to provide a context in which to interpret the data. The second theme related to the - unproved - validity and reliability of the SFQ instrument itself, and the caution that not too much emphasis should be placed upon the data that it generated. The third theme concerned the desirability of using SFQ results only for developmental purposes.

Typical open-ended responses are the following.

"Students' opinions are very subjective and depend upon the context and timing of the evaluation. I do not think too much weighting should be put on this form of appraisal"
"This all depends on how you view appraisal. Some see it as essentially developmental, some judgmental, others as both. As it stands there is little in this questionnaire that can help with development and much that can feel judgment despite the fact that the grounds on which such judgment is made are open to abuse".
"My main concern about using the SFQ for appraisal is that appraisers may use it rather crudely without considering the context of the exercise...."

As a result of the feedback which has been received, modifications have been made to both the SFQ form itself, and the procedures for administration and reporting. Although there is not the space to document all of these here, it is worth noting that in this academic year more "context" data (class size, level, etc.) will be given with each report, and "norms" for various settings will also be provided. These decisions were taken on the basis of the qualitative data received - even though a straight majority vote would have resulted in the status quo being maintained.

Conclusions and discussion

The first point to be made is that staff at PolyU are generally not strongly opposed to the SFQ procedures. This is perhaps a little surprising in that the process was "dumped on them from above" - and has a strong judgmental element in tandem with a developmental aspect. The conventional view is that if staff are to accept appraisal systems then conditions apply (see e.g. Lonsdale, 1990): these include (a) the separation of judgmental and developmental aspects; and (b) "ownership" by the staff involved. Neither of these elements was present in this case, in spite of the generally positive response of the staff to the whole exercise. However, there are still issues to be monitored and addressed in an ongoing way. It might be that the response of staff is a case of "procedural compliance" (Bryman et.al., 1994): i.e. an acceptance of the "ritual" of the procedures, without a buying into the rationale and ethos in which it is based. And again, one needs to take heed of Hutchings' (1993) argument that an over-emphasis on individual appraisal, at the expense of the team, is "a recipe for mediocrity".

A final issue to be addressed relates to the mix of qualitative and quantitative data that informed the evaluation and the decisions which have been taken as a result. If we had simply taken the numerical data generated from the closed responses on the questionnaire, then no changes would have been made on the basis that "staff as a whole are happy". However, if we adopt the value that as many staff as possible should feel as comfortable as possible with the SFQ procedures then we need to look beyond a simple majority vote, and to seek for modifications that:

preserve the essence of the status quo (with which most staff are satisfied), and
address as many as possible of the valid concerns expressed by the minority of staff.

This position means that qualitative and quantitative data must be taken into account in assessing the status quo, and planning modifications.

It is interesting to relate this principle to the rating of teaching, through the use of student feedback. There is a plethora of data to suggest that such rating is both valid and reliable, providing that appropriate instruments and procedures are used, and that interpretation of the data is carried out sensibly. (See e.g. Cashin, 1990). However, if the feedback is to be optimally useful, for developmental as well as judgmental purposes, then properly reflective open-ended comment is also required. There is evidence that staff find open-ended comments more useful than the bare quantitative results. (Baxter, 1991).

There is a tendency for overmuch weight to be given to numerical data. It may be reasonable for staff individually to wish to see "where they stand" in relation to their colleagues: this is necessary for rational decisions to be taken in relation to developmental efforts. It is unreasonable for a Head of Department to use this same information as the sole measure of the quality of a staff member's teaching. However, in the absence of other systematic data, this is precisely what most are doing: and sometimes making decisions based upon it which significantly affect staff members. "Tick a box" student ratings are a useful set of data which can assist in the making of developmental and judgmental decisions. However, they need to be supplemented by other sources of data, and developments in teaching portfolios, or dossiers, offer some promise in this area (Centra, 1994). The danger though is that because more time is involved in decision makers absorbing these essentially qualitative sources of data, the quick-fix, tick-a-box, mean score will be with us for some time yet.

Endnotes

Copies of the questionnaire are available from the authors.
Copies of the questionnaire are available from the authors.

References

Bryman, Alan, Haslam, Cheryl and Webb, Adrian (1994). Performance appraisal in UK universities: A case of procedural compliance? Assessment and Evaluation in Higher Education, 19(3), 175-187.

Cashin, William, E. (1990). Student ratings of teaching: A summary of the research. Management Newsletter, 4(1), 2-7.

Centra, John A. (1994). The use of the teaching portfolio and student evaluations for summative evaluation. Journal of Higher Education, 65(5), 555-570.

Csete, Josie, Jones, John and Kwan, Kam-por (1995). Report on the Evaluation of the Implementation of the Student Feedback Questionnaire (SFQ). Educational Development Unit, HK Polytechnic University, Hong Kong.

Hutchings, Ian (1993). Appraisal procedures: A recipe for mediocrity? The New Academic, 2(3), 1-3.

Lonsdale, Alan (1990). Achieving institutional excellence through empowering staff: An approach to performance management in higher education. In Moses, Ingrid (Ed), Higher Education in the Late Twentieth Century: Reflections on a Changing System. HERDSA, Sydney, 91-107.

Authors: Josie Csete, John Jones and Kam-por Kwan
Educational Development Unit, Hong Kong Polytechnic University
Hung Hom, Hong Kong
Fax: (852) 2334 1569

Please cite as: Csete, J., Jones, J. and Kwan, K. (1996). Student feedback on teaching and staff appraisal. Different Approaches: Theory and Practice in Higher Education. Proceedings HERDSA Conference 1996. Perth, Western Australia, 8-12 July. http://www.herdsa.org.au/confs/1996/csete.html

[ HERDSA ] [ Proceedings Contents ]
This URL: http://www.herdsa.org.au/confs/1996/csete.html
Created 27 June 2001. Last revision: 14 June 2002.
© Higher Education Research and Development Society of Australasia Inc

Source	Domain
Source	Administration guidelines and procedures	The questionnaire instrument	Reporting and use of the student feedback results	Inputting and processing of data
Dept. administrative staff	Focus group interview of dept. AA/EO from 7 pilot depts.	--	Focus group interview of dept. AA/EO from 7 pilot depts.	--
Academic staff	Focus group interview of staff from BRE Staff survey questionnaires to all staff members in the 7 pilot depts. Individual interview of selected staff members from 6 pilot depts.			--
Students	Focus group interview of students from 3 pilot depts.			--
Empirical student feedback data	--	Statistical analysis of data, including reliability test and covariance structure analysis of scales	--	--
EDU staff involved	--	--	--	Document analysis of EDU report
ITS staff involved	--	--	--	Telephone interview of officer-in-charge