Measuring quality across different years within a particular degree program
Raj Gururajan
Department of Computer Science, Edith Cowan University
Trish Sherwood
Department of Rural and Community Studies, Edith Cowan University
This paper presents a model that enables quality to be measured across different years in a particular degree program. Resource, Curriculum and Teaching factors that influence the quality of the learning process are identified and quantified. Resource factors identified and measured are library, computing, laboratory, photocopy facilities, counselling services and financial assistance. Teaching factors identified and measured include personality of teacher, his/her ability to answer questions, to explain difficult concepts, interest in subject matter, attitude towards students and organisation. Curriculum factors identified and measured include processes for learning such as group discussions, class question and answer time allocations, individual discussions, tutorials, class handouts and audio-visual teaching. The implementation of the model is illustrated through a survey of students enrolled in six different degree programs. The factors which most significantly affect the outcomes are measured. The scores are standardised. The resultant measurement clearly illustrates comparative strengths and weaknesses of the degree program in the different years of its implementation.
1.0 Introduction
Postsecondary students are customers in class room teaching. Those who seek to improve class room teaching in higher education must examine features of classroom teaching as a Total Quality Management system dealing with both students and the university system. Fram and Camp (1995:69) emphasise in their article on "Finding and Implementing Best Practices in Higher Education" the urgent need for tertiary institutions to begin developing ways of assessing best practice within the tertiary classroom. This paper develops a new model to assess and develop recommendations for best practice. This model considers the process of learning as the output, with inputs fed from three major areas of Teaching, Curriculum and Resources.
Various factors have been identified that influence the quality of learning using this model.(Gururajan & Sherwood, 1995 : 394; Gururajan et al, 1995). The factors and their strengths usually vary from department to department. These factors are ranked according to their importance with respect to those departments. The factors identified and assessed under the category of teaching fall into three subcategories. The first sub-category refers to teaching qualities of the lecturer as a person resource. These include the following: lecturer's ability to answer questions relating to the unit content, lecturer's personality, lecturer's ability to make the course interesting, lecturer's ability to elucidate difficult concepts, lecturer's attitude towards the unit, and lecturer's organisation of materials. The second sub-category refer to learning processes that are used in teaching. These include group discussions, class question and answer opportunities, opportunities for individual discussions with lecturer, tutorials, class handouts, and use of audio-visual materials. The last sub-category within the teaching factors refer to situational mediums that are provided to facilitate learning. These include lectures, tutorials, assignments, library setting, talking with lecturer outside classrooms and talking with colleagues outside classrooms. Factors in the Curriculum category included unit objectives, assessment level, fit of unit content with assessments, library resources to support units, content of curriculum, detailed information in curriculum, organisation of information, compatible textbook, fit of unit content with overall course, and relevance of information to student's needs. Factors included in the resource category are library resources, computing resources, laboratory resources, photocopy resources, counselling services, and financial assistance.
2.0 Methodology
A university campus wide survey was conducted [1]. About 800 survey forms were distributed to students in all the three years of tertiary education in Computing, Technology, English, Psychology and Business departments by the respective lecturers on the request of the authors, with approximately 70 % return. The survey form was designed in such a way that the information tracked through the survey form is two dimensional. On one dimension students ranked their expectations, while the second dimension represents students' assessment of actual performance of each of the identified factors within the category. This model focuses on the students as clients of the service, and whom Helms and Key (1994:98) identify as playing the key role in TQM implementation in the University setting. The comparative median scores for each of the above categories of factors are represented graphically throughout this paper. The survey results are analysed to find out the course median scores on quality factors. This model fulfils Hart and Morrison's (1992:25) criterion of good quality management models in that it is interdisciplinary and applicable to a wide variety of degree programs.
3.0 Discussion
In this section the factors in each category are defined and the survey findings are presented. The two dimensions, expectations and performance, are discussed for different courses.
Teaching
This included the most extensive listing of factors. Six of these factors relate directly to teaching abilities of the lecturer. The nature of these factors are discussed below:
- Lecturer's ability to answer questions relating to the unit content.
This factor is considered one of the important factors in the area of teaching. This factor directly results from the depth of knowledge of the lecturer.
- Lecturer's personality
Lecturer's personality plays an important role in professional courses. Some lecturer's have more sophisticated interpersonal skills, are more liked by students and/or are perceived as more professionally competent. This is a difficult factor to measure because it is really a composite of many complex sub-factors which comprise personality. Essentially student's tended to rank lecturer's personality on whether or not the lecturer was liked by students and demonstrated competence with subject matter.
- Lecturer's ability to make the course interesting
This refers to the lecturer's skills in making the unit material presentable in a way that is attention holding. This is particularly important where unit teaching blocks are three of four hours in duration, and established attention span averages at 45 minutes.
- Lecturer's ability to elucidate difficult concepts
Students rely on the lecturer to act as a teacher and elucidate difficult concepts.
- Lecturer's attitude towards the unit
Attitude is a major key in holding the students throughout the course. A positive attitude towards the subject and the learning process, provides an environment that encourages positive responses form the students.
- Lecturer's organisation of materials
This refers to the ability of the lecturer to present the material in a cogent systematic manner.
The first question on teaching asked how important the above six factors were in determining the success in a unit. The campus average is shown as the zero scale line, running horizontal. The various scores that were compiled for different departments are discussed below. The first paragraph discusses student ranking within particular courses, the second paragraph provides a comparison of different expectations across courses.
Student Expectations
Students of the second year ranked highest in the overall median score. They were 0.38 above the campus average for the factor lecturer's personality. The students of the third year ranked the lowest median score (-0.48) for the same factor.
Second year students had the highest overall expectation in all the factors of this category. Third year students had the lowest expectation for all the categories. First year students were oscillating above and below the campus average.
Student's Ratings of performance
While the previous two paragraphs deal with the students expectations, the following paragraphs depict the facts on how well the lecturers completed the tasks of teaching factors.
The first year students rated their lecturers above the campus average in almost all the factors. The only factor that scored well below campus average was the lecturer's ability to elucidate difficult concepts. Second year students rated their lecturers well below the campus average in the attitude factor and organisation of lecturer's material factors. These two factors were rated well above the median score in their expectation. While the personality of the lecturer was rated well below the campus average in terms of expectation, third year students rated this factor well above the campus average in terms of performance.
The survey findings in relation to these teaching characteristics of lecturers that affect the quality outcomes of learning, are graphically represented below.
Figure 1: Teaching Factors - Success Factors
The process factors in the teaching category which affect quality outcomes in learning were defined as following:
- Group Discussions
These were exchanges of information conducted by the lecturer in the classroom setting which involved all students and gave all students the opportunity to participate.
- Class Question and Answer Opportunities
This refers to opportunities throughout the teaching period to engage in dialogue with the lecturer.
- Individual discussions with lecturer
This refers to the opportunity to have one-to one exchanges with the lecturer outside of the formal classroom setting.
- Tutorials
This refers to allocated times for individual students to present a prepared paper to the other students . A group discussion monitored by a tutor follows.
- Class Handouts
This refers to written material that is handed to students that further supplements or elucidates what has been taught in the classroom.
- Use of audio visuals
This includes videos, overheads, films, slides, computer visuals and any other media which provide sound or visual images that complement the materials that are being presented in the lecture format.
Student Expectations
Second year students had the highest expectations on the median score (0.55) well above the campus average. Third year students had the lowest expectations on median score (-0.56).
Third year students had the highest expectation for the factor group discussions with a score (0.16). First year students had the highest expectations for the factor class question & answers. Second year students had the highest expectation for the factors individual discussions, class handouts and audio visuals.
Student Ratings of performance
The following paragraphs summarise the students' ratings on how well the teaching factors (discussed in the preceding two paragraphs) were completed by the lecturers.
Third year students rated the lecturers highest on the overall median average with score (0.89). Third year students rated their lecturers highest in all the factors except tutorials. First year students rated the performance of tutorials the highest among this group of factors.
The above results are summarised in the graph below.
Figure 2: Teaching Factors - Success Processes
The final subcategory of factors in the teaching category are those which define the range of information exchange formats. These are outlined below:
- Lectures
These are formal forums of one hour or two hours of duration during which the lecturer presents information to the students.
- Tutorials
In these learning forums, most often students research and present information on a specific topic to other students in a formal learning setting monitored by a tutor.
- Assignments
These vary significantly in format, some being purely reading exercises, others including a significant practical component. All however, provide a formal contract to research or record a particular facet of information.
- Library
Libraries provide a learning environment with both formal access to information and the opportunities for informal sharing of information with other students.
- Talking with lecturer outside classrooms
This is the informal situational learning that occurs when individuals or groups of students seek out lecturers to elucidate particular aspects of their learning. Here, information is tailored to meet individual needs.
- Talking with colleagues outside the classroom
This refers to informal discussions among groups of students that involve the exchange of information on course related matters. It includes information about content, as well as ideas on how to access information or use processes within the university system.
Student ratings
The critical information acquired from various sources to assure quality, has been rated by the students. The findings are as follows.
Third year students rated the factor assignments the highest median, well above the campus average (0.41) and the lowest score was rated by the same set of students for the factor tutorials (-0.61).
The factor lectures was rated highest by third year students while the factor tutorials was rated highest by the first year students. Second year students rated talking with the lecturers and colleagues the highest. The factor class handouts was rated the highest by third year students with in the list of factors.
The results are summarised in the following graph.
Figure 3: Teaching Factors - Critical Information Sources
Curriculum
The four following factors were identified as key aspects affecting quality in curriculum.
- Unit Objectives
All units have performance objectives which identify the learning outcomes of the unit in terms of students behaviours. The clarity of these objectives is a particularly important issue in directing and structuring the teaching and learning processes.
- Level of Assessment
This factor refers to the range of formats used to evaluate students' learning in a particular unit.
- Fit of Unit Content with unit objectives
This referred directly to student's perception of the cogency between what the units stated learning goals were and the actual information provided in the teaching process.
- Library Resources to Support Unit
The relationship between the unit information resource needs, and their availability in the campus library is the critical concern here.
- Content of Curriculum
The relevance of the content of curriculum to students' perception of their needs in preparing for the workplace is assessed here.
In determining how well the lecturers have completed the curriculum factors, the following was found:
Second year students rated the factor library support for the unit the highest with a score well above the median average (0.50). The same factor was rated the lowest by the first year students with a score well below the campus average (-0.37).
First year students scored the factors unit objectives, level of assessments, and fit of unit content with unit objectives most favourably. These factors were rated least favourably by the second year students. The factor content of curriculum was rated favourably by the second year students.
The above points are summarised in the following graph.
Figure 4: Curriculum - Level of Completion
Curriculum factors centering around the quality of available information included the following:
- Detailed Information
This attempts to establish the depth of the information available on required subjects.
- Well Organised Information
This refers to the cogency and accessibility of the required information.
- Compatible Textbook
Here the fit between the textbook and the required information is ascertained.
- Fit of unit content with overall course
The cogency of relationship between the information provided in any one unit and the overall information goals of the course are assessed here.
- Information relevance to students' needs
Here students satisfaction with the information in terms of their learning needs is assessed.
Students are asked to highlight the importance of the above factors in determining the success in their unit of study. The findings are:
Student Expectations
Lecturers teaching the first year students scored the highest in expectations with a median score of 0.29. The lowest score was realised by the third year lecturers with a median score of -0.47.
Students from the first year rated their expectation for the curriculum factors highest and well above the campus average. Both second year and third year students had rated their expectation well below the campus median.
Students Rating of performance
Students teaching the first and second years scored on the median score, well above the average for this group of factors. The two exceptions being the factors compatible text book and fit of content. The first year students and the second year students rated these factors below the average respectively.
Students from the third year rated the performance of their lecturers in terms of curriculum below the campus average for all the six factors.
The above points are summarised in the following graph:
Figure 5: Curriculum - Aspects of Success
Resources
Six key factors were assessed in the resource category.
- Library Resources
These include the range of information resources including books, journals, CD-ROM and Internet.
- Computing Resources
These refer to hardware, software and maintenance and service facilities available in the students computer suites.
- Laboratory Resources
Included here are the services for research and experimentation. These vary from faculty to faculty.
- Photocopy Resources
This includes cost, maintenance and accessibility of the service.
- Counselling Services
Included here are the services that students can access when they are experience personal difficulties that cannot be resolved within the confines of their consultation with their lecturers.
- Financial Assistance
This refers to accessibility of finances, financial advice and employment to students experiencing economic hardship.
Students ratings
These factors were rated by the students and the findings were as follows:
Second year students had rated the computing facilities the highest on the median score (0.45) . Third year students had rated the computing facilities the lowest below the median score of -0.48.
First year students rated the library, lab, photocopy and counselling facilities very favourably. Second year students had rated computing facilities favourably.
Third year students had rated all the facilities below campus average. Computing and lab facilities had been rated well below the campus average with scores -0.48 and -0.42 respectively.
The above points are summarised in the following graph:
Figure 6: Resources
4.0 Interpretation
Comparative Statement on Quality Factors between the five departments
This paper, When looking on a comparative basis across the five courses at the fit between expectations and performance, notes the following trends.
The greatest positive fit was found in the third year rating in terms of teaching processes. The list of factors identified were group discussions, class question and answer sessions, individual discussions, tutorials, class handouts and use of audio visuals. While the students rated their lecturers below the campus average in their expectations, the lecturers scored well above the campus average meeting well over the expectations.
The greatest mismatch was found in the first year ratings for the teaching factors. This group consists of factors answering questions, personality, interest generated in the unit, elucidating difficult concepts, attitude and organisation of materials. While the lecturers have scored a positive score in expectations, the performance was rated well below the campus average. Where the performance was above the campus average, the scores was not that noticeable.
5.0 Recommendations
On the basis of the above analysis based on the comparative median scores, the following recommendations to improve quality can be made:
| 5.1 | That the lecturers teaching in the third year units improve their overall teaching processes to reflect the expectations of students. |
| 5.2 | That appropriate audio visuals be used for the first year teaching units so that the effect of knowledge delivery be realised by first year students. |
| 5.3 | That more tutorial sessions be conducted for the third year students to train them in project presentation, management of projects and other concepts that relate to project management. |
| 5.4 | That the third year project details be informed to students well in advance so that necessary planning be done by students. |
6.0 References
Fram, E. and Camp, R. ( 1995). Finding and Implementing Best Practices in Higher Education. Quality Progress, Feb. 1995, pp 69-77
Gururajan, R. and Sherwood, P. (1995). A quality metric for the instructional process. Proceedings of Pan Pacific Conference on Information Systems, Singapore, June 29 - July 2, 1995, pp 394-397.
Gururajan, R., Sherwood, P., and Newby, R. (1995). A total quality management paradigm for small businesses in training and education: A new definition. Proceedings of INTENT 95, Bunbury, Western Australia, June 26 - June 28, 1995.
Hart, C. and Morrison, E. (1992). Students aren't learning quality principles in business schools. Quality Progress, Jan. 1992, pp 25-27
Helm, S. and Key, C. (1994). Are students more than customers in the classroom? Quality Progress, Sept 1994, pp 97-99.
Rubach, L (1994). Transforming US education with the quality movement. Quality Progress, Sept, 1994. pp 93-95.
Endnote
- The Bunbury Campus (of Edith Cowan University) Principal's funding is acknowledged for this purpose.
Authors: Raj Gururajan, Department of Computer Science, Edith Cowan University, Mt Lawley WA 6050. Email: r.gururajan@cowan.edu.au
Trish Sherwood, Department of Rural and Community Studies, Edith Cowan University, Bunbury WA 6230. Email: p.sherwood@cowan.edu.au
Please cite as: Gururajan, R. and Sherwood, T. (1996). Measuring quality across different years within a particular degree program. Different Approaches: Theory and Practice in Higher Education. Proceedings HERDSA Conference 1996. Perth, Western Australia, 8-12 July.
http://www.herdsa.org.au/confs/1996/gururajan.html |
[ HERDSA ]
[ Proceedings Contents ]
This URL: http://www.herdsa.org.au/confs/1996/gururajan.html
Created 22 Apr 2002. Last revision: 24 May 2002.
© Higher Education Research and Development Society of Australasia Inc
