Analysing Quality Audits in Higher Education

Authors

Abstract

In many countries and many cultures, the issue of quality management is firmly on the agenda for higher education institutions. Whether a result of a growing climate of increasing accountability or an expansion in the size and diversity of student populations (Oldfield and Baron, 1998), both quality assurance and quality enhancement are now considered essential components of any quality management programme (Brookes and Downie, 2002). Despite the abundance of research on the subject, there is no universal consensus on how best to measure quality in higher education. In the UK, quality assessment procedures are both internally and externally driven using different mechanisms and quality criteria. As such the assessed ‘quality’ of any given programme of study may therefore be variable across different audits. In addition, the potential value of the different assessments for quality enhancement purposes may also vary. Given the importance of quality assurance on the higher education agenda, this paper undertakes a critical evaluation of the different methods used to assess the quality of provision in the UK through a case study of one department. A comparison of internally versus externally driven quality audits is undertaken at three different levels; module, programme, and subject level. By examining the methods and criteria used to assess the quality of provision and the way in which the quality assessment findings are disseminated, the paper considers the potential value of these different audits in enhancing the quality of higher education provision and identifies action which could be taken by institutions and departments as a result. The authors adopt a secondary approach, drawing on relevant quality management literature, the quality audits conducted within one department as well as those by the Subject Centre for Hospitality, Leisure, Sport, and Tourism of the Higher Education Academy, and the most recent Quality Assurance Agency (QAA) audits within the relevant area, i.e in Unit 25 of their classification of subjects. The findings suggest that while externally driven quality audits enable comparative benchmarking between individual programmes, they may not be the most appropriate for enhancing the quality of higher education provision. The potential for quality enhancement is determined by the manner in which the evaluation is conducted and subsequent change implemented.

Nina Becket and Maureen Brookes

Introduction

In many countries and many cultures, the issue of quality management has been firmly on the agenda of higher education institutions for quite some time. Higher education for the masses and a growing climate of increasing accountability (Oldfield and Baron, 1998; Eriksen, 1995) are frequently cited as the rationale for a greater emphasis on quality. Accordingly there has been a good deal of research into the subject of quality in higher education, with well-recognised contributions from the UK, Australia, Norway, and the USA, amongst others. Within the UK there are particular environmental forces imposing the need for effective quality management, these include:

  • a growing climate of increasing accountability an expansion in the size of student populations
  • an increasingly diverse student population as a result of widening participation initiatives and targeting international markets
  • diminishing resources by which to deliver programmes of study
  • greater expectations of students as paying customers
  • more flexible educational provision at both undergraduate and postgraduate level
  • an increase in collaborative provision between institutions

Some of these forces demand that institutions have quality assurance procedures that are both rigorous and transparent to external stakeholders. Others however, clearly require that quality enhancement initiatives are firmly embedded into any quality management programme, and that continual efforts are made to enhance the quality of provision. As Avdjieva and Wilson (2002) suggest, Higher Education Institutions (HEIs) as organisations of learning are now required to become learning organisations (p. 372), where internal stakeholders also interpret and assess the quality of higher education provision. As a result, many institutions have worked towards the adoption of total quality management (TQM) practices in order to achieve these goals. However, critics of this approach suggest that a wholesale adoption of TQM without adaptation to reflect the particular characteristics of higher education is unacceptable (see for instance Yorke, 1994). It has even been purported that the practice of TQM in higher education is deteriorating into managerialism because of the disparity between TQM techniques and educational processes, as well as the lack of shared vision within institutions or educational fields (Srikanthan and Dalyrymple, 2003).

Despite the progress that has been made through research and debate, there is still no universal consensus on how best to manage quality within higher education. This paper therefore seeks to evaluate current approaches used to assess quality within higher education in the UK. It begins with a brief review of relevant literature on quality management in higher education to develop a framework for analysis. Six key criteria for analysis are identified, which are then applied to the undergraduate provision of one case study department whose practices are arguably indicative of many other departmental quality management practices.

Assessing quality in higher education

Defining quality in higher education has proved to be a challenging task. Cheng and Tam (1997) suggest that ‘education quality is a rather vague and controversial concept ‘(p. 23) and Pounder (1999) argues that quality is a ‘notoriously ambiguous term’ (p. 156). At the broadest level, education quality can be viewed as a set of elements that constitute the input, process and output of the education system, and provides services that completely satisfy both internal and external strategic constituencies by meeting their explicit and implicit expectations (Cheng and Tam, 1997, p. 23). If higher education is viewed as a system, then any quality management programme must therefore assess inputs, process and outputs.

Cheng and Tam (1997) also identify both internal and external stakeholders in the quality management process. Current students and front line staff are internal constituents whereas employers, government bodies, institutional management, prospective students, and professional bodies are external. These stakeholders might have disparate definitions of quality as well as different preferences for how quality is assessed. While Hughes (1988) suggests quality indicators may differ for internal and external stakeholders, Cheng and Tam (1997) further argue that expectations of the different constituencies may be not only different but contradictory (p. 23). According to Cullen, Joyce, Hassall, and Broadbent (2003) the challenge is to produce a performance evaluation framework that permits the equal expression of legitimate voices, though they may conflict or compete in some ways.

Harvey and Knight (1996) suggest that quality can be broken down into five different but related dimensions: quality as exceptional (e.g. high standards), quality as consistency (e.g. zero defects), quality as fitness for purpose (fitting customer specifications), quality as value for money, and quality as transformative (an ongoing process that includes empowerment and enhancement of the customer satisfaction). While the authors advise that quality as transformative incorporates the other dimensions to some extent, it can also be argued that different stakeholders are likely to prioritise the importance of these different dimensions of quality according to their motivations and interest (Owlia and Aspinwall, 1996). In some quality initiatives therefore, there may be a preference for the monitoring of some quality dimensions at the expense of others.

What is apparent from this discussion is that the concept of quality in higher education is complex and dependent upon different stakeholder perspectives. While Pounder (1999) suggests that we should be looking for another more appropriate concept than quality to benchmark performance in higher education, in reality quality remains the key focus within UK higher education provision and continues to be assessed in a number of different ways by different stakeholders.

Traditionally external stakeholders have been associated with quality assurance procedures. Quality assurance refers to the ‘planned and systematic actions [deemed] as necessary to provide adequate confidence that a product or service will satisfy given requirements for quality’ (Borahan and Ziarati, 2002, p. 914). For higher education institutions this requires them to demonstrate responsible actions in their professional practices, be accountable for public funds received and demonstrate the results they achieve with the available resources (Jackson, 1998, p. 46). Elton (1992) refers to these as the quality ‘A’s: Accountability, Audit, and Assessment’ and suggests these are concerned with the control of quality and the people who control quality. The particular mechanisms for assurance are usually imposed by external bodies, such as university management and most commonly include accreditation, external examiners and quality audits (McKay and Kember, 1999). As a control tool therefore, the focus is predominantly on the extent to which the procedures and conditions that are perceived to result in appropriate levels of quality are followed within institutions or programmes and are effective in meeting their purpose (Jackson, 1996, p. 39). Prioritised dimensions of quality (as above, Harvey and Knight, 1996) therefore include quality as consistency, quality as fitness for purpose, and quality as value for money. As these external stakeholders also assure that a minimum threshold of quality is maintained, quality as exceptional is also a relevant dimension. To this extent, external quality assurance mechanisms have a pertinent role to play in quality assessment practices for a number of stakeholders. However, the processes adopted are by nature summative providing only a snapshot of ‘quality’ and therefore McKay and Kember (1999) argue that quality control measures in isolation may have limited impact if not accompanied by appropriate educational development initiatives (p. 26). In addition, the legitimacy of these approaches for quality enhancement purposes has been questioned (see for instance Roffe, 1998) as the use of external change agents does not imbed quality into the culture of programmes. Furthermore, there may be a tendency towards ‘gaming the system’ (Bradach, 1998) where what gets measured gets done or fixed in the short term in order to meet assessment targets.

Internal stakeholders, on the other hand, are more likely to be concerned with quality as transformative (as above, Harvey and Knight, 1996) where it is an ongoing process that includes empowerment and enhancement of customer satisfaction. The emphasis for these stakeholders is therefore not only on quality assurance, but additionally on quality enhancement which aims for an overall increase in the actual quality of teaching and learning often through more innovative practices (McKay and Kember, 1999). Elton (1992) suggests that this approach focuses on the quality ‘E’s: Empowerment, Enthusiasm, Expertise, and Excellence’. McKay and Kember (1999) report that initiatives for quality enhancement tend to be less clearly defined and are often more diverse than quality assurance initiatives. Mechanisms adopted by internal stakeholders are likely to include self-evaluation practices and student surveys. As students are viewed as an integral part of the learning process (Wiklund et al., 2003) this approach tends to be more formative in nature and therefore more likely to lead to continual quality improvement efforts. The involvement of internal stakeholders is also more likely to result in a culture of quality being embedded within programmes. However, these measures are representative of a comparatively limited number of stakeholders and where self-evaluation practices are employed there can be a tendency to paint a rosy picture especially when linked to appraisal or tenure decisions.

The actual measurement of quality is also approached differently by various stakeholders. While some prefer to utilise quantitative data to produce quantitative ratings, others prefer to adopt a qualitative approach. There are, of course, pros and cons to both approaches and benefits to be gained from either. While quantitative ratings facilitate performance comparability especially on a longitudinal basis, they generally fail to provide any clear explanation as to why certain ratings are given. As such they may be more suitable for quality assurance initiatives. Qualitative data, on the other hand, often provides richer data (Powell, Hunt, and Irving, 1997), which can more readily inform decision making for quality enhancement purposes. However, it may prove less beneficial when benchmarking performance. A quality management programme that utilises a mixture of both types of data would seem most appropriate for both quality assurance and enhancement purposes (Brookes, 2003).

Colling and Harvey (1995) argue that external scrutiny should adopt an approach that ensures accountability, enhances quality, is practical, is efficient, is effective and offers a degree of autonomy (p. 30). They also suggest that quality enhancement requires that external scrutiny is linked into a process of continual quality improvement (p. 30). For this to be achieved, external scrutiny must therefore be reviewed and acted upon within internal quality enhancement processes. O’Neil (1997) purports that no amount of external exhortation can compensate for local ownership and individual responsibility in a change process (p. 14). Jackson (1997) argues that the demands of trying to fulfill both internal and external expectations for accountability are not always consistent with expectations of development, improvement and innovation.

There have been various attempts to utilise the quality dimensions of Gronroos, Garvin and Parasuramman (see for instance Owlia and Aspinwall, 1996) and the balanced scorecard (Cullen et al., 2003) to develop quality assessment models for higher education. However, the tool most frequently drawn upon (see for example Motwani and Kumar, 1997; Eriksen, 1995), is that of Total Quality Management (TQM), defined as:

‘A management approach of an organization, centred on quality, based on the participation of all its members and aiming at long run success through customer satisfaction and benefits to all members of the organization and to society.’ (ISO 8402 in Wiklund et al., 2003, p. 99)

The rationale for adoption is that TQM has the potential to encompass the quality perspectives of both external and internal stakeholders in an integrated manner, and thereby enable a comprehensive approach to quality management that will assure quality as well as facilitate change and innovation. However, there have been a number of limitations identified in the wholesale adoption of TQM to higher education. Roffe (1998) suggests that while there are a small number of quality indicators in industry, these are more numerous and complex in higher education and therefore more difficult to assess. Similarly Yorke (1994) advises that accountability relationships are more complicated, and Roffe (1998) indicates that while the accountability emphasis of TQM in industry is on a team, this tends to lie with individuals in higher education. Harvey (1995) further argues that the emphasis of quality in industry lies predominantly with the customer whereas in higher education there is a continued debate regarding who the customer actually is. As a result of this debate, Hewitt and Clayton (1999) recommend that a model of educational quality that is different from, but capable of being related to, commercial models is beginning to emerge (p. 838), however, it is not yet complete.

This brief literature review suggests there is as yet, no definitive model to evaluate quality within higher education, however there do appear to be certain key constituents of an analytical quality framework to assess current quality management practice within higher education. For the purposes of this analysis these have been identified as:

  • the degree to which inputs, processes and outputs are assessed
  • the degree to which different stakeholder perspectives are taken into account
  • the degree to which the different quality dimensions are considered
  • the extent of formative versus summative assessment inherent in quality assessment practices
  • the extent of quantitative versus qualitative assessment inherent in quality assessment practices
  • the balance of quality assurance versus quality enhancement practices, and the extent to which these processes are interlinked.

The following section applies these factors to one case study within the Department of Hospitality, Leisure, and Tourism Management (HLTM) at Oxford Brookes University. The procedures used within this department are reflective (for the most part) of those applied in other programmes within the UK, so are likely to be an exemplar of current practice.

Case study

This section reports on the methods used to assess the quality of provision within the undergraduate programmes in the Department of HLTM at Oxford Brookes. It presents an overview of the various evaluation tools employed, and examines the quality management processes employed at the module level and the programme level, and contrasts these methods with the national level assessment of QAA Unit 25 Hospitality, Leisure, Recreation, Sport, and Tourism provision.

Module evaluation

At the modular level, the most nominal level of the evaluation, students review their experience within each unit of study against a number of specified criteria. For the majority of HEIs, such feedback is sought for each run of the module and results are likely to be disseminated through relevant committee meetings. The extent to which module evaluations are discussed and valued as a quality enhancement tool will depend upon the manner in which they are reported and discussed amongst those involved in programme review and development. The degree to which staff reflection feeds into module evaluation is subject to a number of factors, which may be personally or institutionally driven. In the latter, this may depend upon the institutional purpose of the evaluation. If it is to assess the teaching capabilities of the module leader or a member of a teaching team for tenure or salary purposes, then it is highly unlikely that staff perceptions are taken into account. In HLTM the process of module evaluation is driven by staff members themselves and thus used for course development, and potentially for personal development purposes.

Within HLTM, evaluation of individual modules is undertaken every time the module is run. Instruments used to obtain student feedback are designed by module leaders and may vary for individual modules. Generally students are surveyed on the extent to which they believe they have met the stated learning outcomes for that module, the module design, organisation, and content, teaching methods employed, learning resources provided, and the appropriateness of the assessment tasks and weighting. Students are also asked about their attendance in order to provide additional insight to the feedback given. Typically a Likert-type scale is used to assess student opinion of achievement of learning outcomes for benchmarking performance on a longitudinal basis, and a qualitative approach is used to solicit further feedback for the other criteria.

Externally, quality assurance at the modular level involves the appointed External Examiner (see below). Within Oxford Brookes their role is to provide feedback on modules at two stages, firstly on the proposed assessment tasks in relation to the learning outcomes and the marking criteria to be used. In this way, module leaders receive formative feedback prior to the actual assessment taking place. Additionally, at the end of the module, external examiners provide feedback on student performance generally, in relation to expected norms, and in relation to the specific learning outcomes, assessment tasks, grades awarded, quality of tutor feedback, and the marking consistency to the published criteria and across teaching teams.

Both the student and external examiner feedback is then used in the compilation of a standard Module Evaluation (MEV) report, completed by the module leader. Student feedback is clearly articulated under the headings identified above in conjunction with evaluative comments by the module teaching team. The MEV also includes data on student performance on the module by grade category. The module leader is asked to reflect on the overall evaluation of the module and to propose actions for the next run of the module where the quality of the module could be improved.

The MEV reports are completed and presented at the first programme committee meeting after the finish of the module and discussed in an open forum that allows for peer consultation as well as input from student representatives. Following approval of any actions at the committee meetings, the completed MEVs are posted on student notice boards to close the feedback loop to students and are used to inform the annual review of the programme.

At the module level, evaluation therefore includes the perspectives of students, staff, and the external examiner. It is driven predominantly from an internal perspective in conjunction with that of the external examiner. This approach ensures that module inputs, process, and outputs are all assessed. Quality dimensions used in the audit include quality as exceptional, consistency, and fitness for purpose. However, there is also the potential for quality as a transformational process depending upon the actions taken by individual module leaders and the extent to which the feedback loop is closed. In this way, both quality assurance and enhancement are facilitated for all three system components (inputs, process, and outputs). Given the level of assessment, the changes that result from this type of audit are generally minor in comparison to programme wide initiatives. However, they can lead to the development of innovative approaches to teaching and learning, for example the introduction of formative self-assessment tasks by students has resulted in greater achievement of learning outcomes. The creation of the MEV which pulls together all three stakeholder assessments and provides both quantitative and qualitative feedback creates a holistic review of individual modules and also offers potential for benchmarking longitudinally. At the module level therefore, the MEV is the quality management instrument that enables both quality assurance and enhancement to take place.

Internal programme evaluation

While the validity of student evaluation at a module level is well recognised, there is also support for field or programme based evaluation (Wilson, Lizzo, and Ramsden, 1997) in order to facilitate a more comprehensive assessment of students’ perceptions of the quality of their experience. This type of feedback generally entails students assessing their overall satisfaction with the perceived quality of various aspects of provision including the design of the curriculum, the quality of the teaching, learning, and assessment methods, and the resources available to support these. Softer aspects of their experience that impact upon students’ overall satisfaction may also be included. Approaches to this type of evaluation can be undertaken across the different levels of study (e.g. first-year, continuing students, or graduating students). Once again, the degree to which this broader feedback can be used for quality enhancement purposes will depend on the manner in which it is incorporated into a review process and used to determine change.

With increasing student diversity and more competitive markets, HLTM staff recognised the need for a more comprehensive approach to monitoring the quality of provision for students. Thus in 1998, the ‘student experience’ survey was launched which draws on the well-recognised works of Ramsden (1991) and Harvey et al. (1997). The student experience survey is conducted three times a year across different student cohorts, and seeks both quantitative and qualitative feedback from students on their perceptions of the quality of their programme and their experience of other aspects of the department and the university. While quantitative ratings facilitate performance comparability on a longitudinal basis, the qualitative feedback provides insight into the student evaluations to help inform decision-making on the quality of provision (Veal, 1997).

Findings from each survey are discussed at programme committee meetings (which include student representatives), and any necessary actions to be taken in response to student feedback are identified and agreed. Where further investigation is necessary, this is usually done through the use of focus groups facilitated by an external researcher. A full written report is then produced and made available to students in order to ‘close the feedback loop’. Harvey et al. (1997) suggest this is a vital process for successfully involving students in quality management and the department believes that this demonstrates a commitment to taking student feedback seriously. The findings and actions are then reported within the annual programme review.

While it is solely students as internal stakeholders that inform this particular quality evaluation, the approach does monitor the quality of all system components: input, process, and output. In addition, all quality dimensions are included. Quality as transformation is particularly driven through student feedback at regular intervals throughout their programme of study and the efforts undertaken to close the feedback loop so that students can see the results of their feedback on the quality of their experience. In this way the department is developing an instructional environment ‘that promotes students’ perception of quality to result in better learning and development’ (Tam, 2002, p. 212). The longitudinal nature of the survey and quantitative assessment provide for quality assurance, while qualitative data supports enhancement initiatives that can be beneficial to both students and staff. For example, within HLTM, the survey informed changes to induction programmes and personal tutor systems to enhance student support. The process also allows for the impact of any recommendations implemented to be monitored and adjusted if necessary in the future in line with continual quality improvement practices. The potential for quality enhancement at the programme level is thus far greater than that at the modular level.

External programme evaluation

Since 2001 the Department has also participated in an annual Course Experience Survey run by the Subject Centre for Hospitality, Leisure Sport and Tourism, previously part of the Learning and Teaching Support Network (LTSN) and now part of the Higher Education Academy – www.hlst.heacademy.ac.uk

This survey collects views from graduating students regarding their experience of teaching and learning during their entire course, so includes aspects of each of the inputs, process and outputs of their experience. The majority of the data collected is quantitative in nature, but there is also an opportunity for students to comment more qualitatively on their experience.

The results are compiled by the Subject Centre for Hospitality, Leisure, Sport, and Tourism, and course directors receive a confidential report for each of their programmes of study, and also the overall results at a national level (see www.hlst.heacadeny.ac.uk/projects/sceq.html)

In this way current national trends relating to student needs and perceptions are identified on a longitudinal basis and fed back to participating course teams. The survey therefore sets the individual course performance within the context of national provision and identifiable trends.

As such there is potential for programme teams to enhance the quality of their provision. The intention is that the survey results can be used to benchmark performance against the national results over time and also to identify both positive and negative aspects of the student experience in comparison to national benchmarks. Feedback from survey participants suggests that the results are reported to course committees in order to inform programme changes and enhance the quality of student provision, and this information thus complements internal quality assurance mechanisms.

This audit provides indicators for the exceptional, consistency, fitness for purpose, and value for money quality dimensions. There is also potential for the transformative dimension, but this depends upon the way in which the information is utilised within a department. A limitation in this approach is the participation only of graduating students, so the feedback loop is only closed for staff and not for students. Furthermore, as the survey is conducted externally to departments, whilst there is potentially more objectivity in the analysis of results, there is unlikely to be equivalent ownership by staff in comparison to internal evaluation activities.

In HLTM, the findings from the this survey are used to benchmark the departmental programmes against national averages over time and to complement internal evaluations. The results are fed into the annual review process and discussed at committee meetings. However, due to the lack of qualitative data provided by this survey in comparison to that collected in the internal student experience survey, the latter is most likely to be used for quality enhancement activities and to inform change as it provides a richer source of data.

External examiner evaluation

In the UK it is required that institutions and programme teams appoint external examiners to review and evaluate the provision of HE courses. The role of the external examiner is determined by the QAA. It is perceived that ‘external examining provides one of the principal means for the maintenance of nationally comparable standards within autonomous higher education institutions’ (QAA Code of Practice, pg. 1). The particular nature of the external examiner role is then further defined by the requirements of the individual institutions. At Brookes this requires the external examiner to provide an annual assessment of the provision at a programme level in an annual report to the institution (not the school or department), and for this to be formally responded to by the programme team.

The precise nature of institutional engagement with external examiners will determine the degree to which programme teams and individual members of staff actively use feedback from external examiners. At Brookes, as in some other institutions, external examiners may also give feedback at the modular level as described above providing feedback to the module leader or programme team on issues such as module content, assessment tasks, and marking.

External examiner comments are likely to be largely qualitative in nature, although they will also review quantitative data, such as cohort performance and module grade averages, within their role. A key benefit of their contribution is that they are in a position to provide an informed and generally objective view on performance within both modules and an entire programme of study, and also against experience from other institutions, all be it from an individual perspective. In a situation where the majority of evaluation conducted is taken from the internal perspective of students and staff, this external view is valuable. External examiners are usually appointed for a period of four years and are thus also in a position to offer a longitudinal perspective on performance and development. Thus whilst there is clearly a quality assurance aspect to the role, there is also potential for a contribution to quality enhancement activities.

The quality dimensions contributed to by external examiners include quality as exceptional through the commendation of good practice, and consistency, and fitness for purpose in relation to performance of other similar programmes for both individual modules and the entire course. It is likely that the external examiner will be more concerned with the outputs of the education system rather than the inputs and process, although this will be largely dependant on the nature of the relationship between the department and the individual. For example, in HLTM the external examiner reviews and provides feedback to module leaders on the assessment method and learning outcomes for modules, prior to distribution to students, so arguably also has knowledge of these inputs into the system.

Annual programme review

The main purpose of this review is to ‘provide opportunities for staff and students to appraise actions taken in the light of the evidence provided by a variety of performance indicators and in relation to the University’s Mission and specific aims or targets contained in the School’s Planning Agreement’ (Oxford Brookes University Quality and Standards Handbook, 2003, p. 2). Performance indicators include both internal feedback from students and staff on the quality of provision drawing on MEVs and any other internal feedback, as well as internal data on student recruitment, progression, and achievement, and cohort analysis. These are compiled with external reports and advice, such as the annual External Examiner report, any external evaluations such as by the QAA, and advice from professional or statutory bodies in a review undertaken for each academic year.

As required at Brookes, HLTM conducts an annual review of provision following the close of an academic year. These reviews incorporate the MEVs, the internal student experience survey results as well as the Subject Centre survey results, external examiner reports, and internal data on recruitment, progression and achievement as identified above. Course managers are responsible for using these assessments and data to provide a qualitative self-assessment on changes made to the programmes, the quality of learning opportunities provided and the adequacy of learning resources. This review is then presented at the first committee meeting in the new academic year. Student representatives are present and actions undertaken during the year are discussed. Any actions to be undertaken as a result of the review are agreed so the feedback loop is once again closed. HLTM reviews are then combined into a school-wide review, which is then presented at the Board of Studies meeting.

The annual review is thus a summative compilation of internal and external quality evaluations taken throughout the year for varying purposes and using different criteria, although there is greater reliance on internal stakeholder assessments as a norm. The external stakeholder voice is only represented by the external examiner normally, and occasionally by industry bodies or other external stakeholders. As such there is a greater reliance on internal assessment and internal data to interpret the assessment. Depending on the professionalism of those involved, there might also be an inclination to use the data to support a positive self-evaluation. While all system components are measured, the focus is on outputs drawing on quantitative internal statistics and then on inputs in relation to these. All quality dimensions are present in this process but quality as transformation does not necessarily relate to empowerment and customer satisfaction as per Harvey and Knights (1996) definition. The emphasis on the inputs and outputs suggests that there is more reliance on quantitative data to support evaluations and as such this evaluation tool leans towards quality assurance more than enhancement. As the reviews become incorporated into one school-wide document however, there is less likelihood that they support enhancement initiatives in a meaningful way.

Subject level

In the mid 1990s, the general public in the UK started to demand greater accountability within higher education and this created a demand for valid, reliable, and comparable performance data on teaching quality (Wilson, Lizzo, and Ramsden, 1997). The QAA was established in 1997 to ‘provide an integrated quality assurance service for UK higher education’ (www.qaa.ac.uk/aboutqaa/qaaintro/contents.htm) The QAA is an independent body whose ‘mission is to promote public confidence that quality of provision and standards of awards in higher education are being safeguarded and enhanced’ (www.qaa.ac.uk/aboutqaa/aboutQAA.htm)

Prior to 2004, the QAA review process required quality assessment within higher education institutions at subject level by peer review using six criteria: curriculum design, content, and organisation; teaching, learning, and assessment; student progression and achievement; student support and guidance; learning resources; and quality management and enhancement (QAA, 1997). Unit 25 encompasses the cognate subject grouping of hospitality, leisure, sport, and tourism. Results of this review process were published providing both quantitative and qualitative evaluation of subject provision within institutions. The level at which these reviews were undertaken were therefore across departments or schools offering Unit 25 subjects, rather than at the programme or departmental level.

Within HLTM the review encompassed both hospitality and tourism programmes and at that time tourism was offered by three different schools within the University. As within other departments across the country, a great deal of time, effort, and financial support was required to produce a self-assessment document and the evidence to support this. Each of the six criteria was evaluated using both quantitative and qualitative data, and awarded a numerical score.

In terms of an evaluation mechanism this process draws on both internal and external stakeholder perspectives, but the overall assessment is by external agents using externally defined criteria. While these are known in advance, there is every likelihood that the system will be gamed (Bradach, 1998) and what gets measured will be fixed, at least in the short term. Peer review by a team of subject experts however, is expected to reduce any subjective bias in a review by a single assessor. The approach encompasses a review of inputs, process and outputs of the provision and is thus comprehensive, although it could be argued that more weight is given to outputs than the other two system components. For instance, in HLTM, many international students opt to undertake ordinary rather than honours degrees for financial reasons. While the processes in place for students therefore allow for this flexibility, the department was judged on its outputs (i.e. the percentage of students with ordinary degrees) within student progression and achievement.

It could be argued that all quality dimensions are included in the audit, yet quality as transformative is only really apparent under the ‘criteria quality management and enhancement’. With results disseminated to those involved in programme delivery so long after the event, there is less scope for quality as transformation. As quality criteria and process are imposed, there is also less likelihood of the process resulting in a culture of quality being developed as a result. As the system for undertaking this audit has now been changed, there is also no provision for benchmarking performance longitudinally. Furthermore as assessment took place across Unit 25, it is difficult to make comparisons across the subject provision of different institutions. Therefore there is considerably less potential to use these evaluations to inform and enhance provision within specific programmes of study at Unit 25 level.

QAA overview report

The most recent evaluation of hospitality, leisure, recreation, sport, and tourism subject provision for Unit 25 was completed in 2001 by the QAA. Following a review of individual departments, the QAA produced a Subject Overview Report for Hospitality, Leisure, Recreation Sport, and Tourism (www.qaa.ac.uk/revreps/subjrev/All/qo3_2001.pdf ). This report covered the same six criteria as those for individual programmes and provides an overview of the quality of provision for Unit 25 at a national level. The report summary states that:

‘The individual reviews confirm overall that the subjects are achieving their main aims and objectives, although with a need to address a number of important sector-wide issues Across all of the provision, the highest grades are in student support and guidance, followed by learning resources. The lowest average grades are in teaching, learning and assessment and in quality management and enhancement’ (p. 1)

The potential value of this assessment is the confidence that it provides to relevant stakeholders in higher education as a result of the external audits conducted. Although both the departmental and overview reports are publicly available via the QAA website, this information is most likely to be of relevant interest to involved stakeholders such as the funding councils, the university providers or accrediting bodies, rather than less involved external stakeholders such as prospective parents.

The Subject Overview Report provides a snapshot of national level subject area provision and an assurance of quality. The content is largely qualitative, with quantitative aggregation of departmental scores. Its key function is to provide assurance of quality at the Unit 25 subject level, and an indication of quality enhancement priorities for the subjects. As such the quality dimensions are primarily assuring value for money, fitness for purpose and to some extent exceptional and consistency aspects, although there is potential for subjects at a national level to use the information collectively through associations to identify aspects for enhancement.

Conclusions

It is apparent that in quality management within departments quantitative data is predominantly used, therefore there is a need to focus on qualitative sources of information in order to sufficiently inform change and create enhancement. In the case study, qualitative data is provided by students at the module level and also concerning their broader experience, however at the modular level the contribution to quality enhancement is relatively small, hence the majority of information suitable for quality enhancement comes from the broader student experience. There is a need therefore for those programmes which do not undertake such an evaluation to ensure that they obtain sufficient qualitative information in order to inform enhancement opportunities.

It is also evident that the majority of quality management information is obtained from internal sources. This approach is helpful to engender a culture of quality and is indicative of TQM processes. External information sources tend to concentrate on the contribution of external examiners. As this is often an individual voice there is potential for pressure and/or power and objectivity versus subjectivity issues in this contribution. Research is currently being undertaken within the UK to more comprehensively define the role of the external examiner in the UK (www.ltsn.ac.uk/generic centre//index.asp?id=19567)

This paper appears to support the need for this research and it will be interesting to find out how this role might be further enhanced in the future.

All methods of evaluation provide a snapshot of provision at any given time and at variable times throughout a programme of study. While this provides a one-off quality assurance tool, often further value can be gained through longitudinal application of the data to enable benchmarking to identify potential areas for enhancement. There is potential for a longitudinal approach for all the evaluation mechanisms with the exception of the QAA audit that will not be repeated in this format (but will take place at institutional rather than subject level). Whether this potential is realised is dependent on university committees and committees providing an appropriate framework for this and course teams taking ownership of evaluation findings and acting appropriately to enhance provision.

Quality assurance is provided through existing mechanisms to both internal and external stakeholders to some degree. Some methods have more potential than others for enhancement. This is dependent upon the nature of the data and the degree to which it can be interpreted by those external to the departments. There are current developments within UK higher education which aim to offer more information to parents and potential students as external stakeholders regarding the quality of provision at departmental level and initial information can be found at www.tqi.ac.uk.

The findings reveal that there is little opportunity for quality as transformational (such as that possible through the student experience survey) within the evaluations used, and effort must be directed to ensuring that quality management activities focus on this aspect of quality enhancement. In this it is important that efforts are made to ensure that such activity is made clear to students so that they will also have ownership in the quality processes adopted. In the case study department, the most widely applicable tool for quality enhancement purposes is the student experience survey and therefore it is possible that departments that do not use such a system might be more restricted in terms of the quality management information available to them.

Overall it would appear that there is a need to incorporate more external feedback with internal processes in order achieve enhancement potential and increase the objectivity of quality management initiatives. Institutions in designing their quality management systems have a role to play in this. By designing institutional system requirements to encourage the use of information sources external to the departments, for example through the use of staff from other departments, the current mechanisms could be enhanced. However, there is also an inherent danger in this in terms of providing programmes with another ‘hurdle to jump over’ and ensuring that ownership of the process remains within the department. It is clear that the best value overall from quality management activities appears to be at the programme level as this gives sufficient information to make significant changes, and the ownership necessary to support and encourage quality enhancement initiatives.

Biography and Contact Details

Nina Becket is Assistant Director of the Higher Education Academy Subject Network for Hospitality, Leisure, Sport, and Tourism
Oxford Brookes University
Gipsy Lane Campus, Headington,
Oxford OX3 0BP
Email: njbecket@brookes.ac.uk
Maureen Brookes is Senior Lecturer in Marketing in the Department of Hospitality, Leisure, and Tourism Management of the Business School
Oxford Brookes University
Gipsy Lane Campus, Headington,
Oxford OX3 0BP
Email: meabrookes@brookes.ac.uk

References:

Avdjieva, M. and Wilson, M. (2002), Exploring the Development of Quality in Higher Education, Managing Service Quality, 2 (6), pp. 372-383.

Borahan, N. G. and Ziarati, R. (2002), Developing Quality Criteria for Application in the Higher Education Sector in Turkey, Total Quality Management, 13 (7), pp. 913-926.

Bradach, S. (1998), Franchise Organisations, Boston: Harvard Business School Press.

Brookes, M. (2003), Evaluating the Student Experience: An Approach to Managing and Enhancing Quality in Higher Education, Journal of Hospitality, Leisure, Sport and Tourism Education, 2 (1), pp. 1 -9.

Brookes, M. and Downie, N. (2002), Managing Change: The Value of Student Feedback, EuroCHRIE Congress 2002, Barcelona.

Cheng, Y. C. and Tam, W. M. (1997), Multi-models of Quality in Education, Quality Assurance in Education, 5 (1), pp. 22-31.

Colling, C. and Harvey, L. (1995), Quality Control, Assurance and Assessment the Link to Continuous Improvement, Quality Assurance in Education, 3 (4), pp. 30-34.

Cullen, J., Joyce, J., Hassall, T., and Broadbent, M. (2003), Quality in Higher Education: from Monitoring to Management, Quality Assurance in Higher Education, 11 (1), pp. 30 34.

Elton, L. (1992), Quality Enhancement and Academic Professionalism, The New Academic, 1 (2), pp. 3-5.

Eriksen, S. D. (1995), TQM and the Transformation from an Elite to a Mass System of Higher Education in the UK, Quality Assurance in Education, 3 (1), pp. 14-29.

Harvey, L. (1995), Beyond TQM, Quality in Higher Education, 1 (2), pp. 123 146.

Harvey, L. and Knight, P. T. (1996), Transforming Higher Education, Buckingham: SRHE and Open University Press.

Harvey, Plimmer, Moon, and Geall (1997), Student Satisfaction Manual, Oxford: Oxford University Press.

Hewitt, F. and Clayton, M. (1999), Quality and Complexity Lessons from English Higher Education, International Journal of Quality and Reliability Management, 16 (9), pp. 838-858.

Hughes, P. (1988), The Challenge of Identifying and Marketing Quality in Education, The Australian Association of Senior Educational Administrators, Sydney, NSW. In Cheng, Y. C. and Tam, W. M. (1997), Multi-models of Quality in Education, Quality Assurance in Education, 5 (1), pp. 22-31.

Jackson, N. (1996), Internal Academic Quality Audit in UK Higher Education: part III the idea of partnership in trust, Quality Assurance in Education, 4 (4), pp. 37-46.

Jackson, N. (1997), Internal Academic Quality Audit in UK Higher Education: part II implications for a national quality assurance framework, Quality Assurance in Education, 5 (1), pp. 46-54.

Jackson, N. (1998), Academic Regulation in UK Higher Education: part 1 current practice and conceptual frameworks, Quality Assurance in Education, 6 (1), pp. 5-18.

McKay, J. and Kember, D. (1999), Quality Assurance Systems and Educational Development: part 1 the limitations of quality control, Quality Assurance in Education, 7 (1), pp. 25-29.

Motwani, J. and Kumar, A. (1997), The Need for Implementation of Total Quality Management in Education, International Journal of Education Management, 11 (3), pp. 131-135.

Oldfield, B. and Baron, S. (1998), Is Servicescape Important to Student Perceptions of Service Quality? Research Paper, Manchester Metropolitan University.

Owlia, M. S. and Aspinwall (1996), A Framework for the Dimensions of Quality in Higher Education, Quality Assurance in Education, 4 (2), pp. 12-20.

Oxford Brookes University (2003), Quality and Standards Handbook, October, Oxford Brookes University.

ONeil, C. (1997), Student Ratings at Dalahousie, Focus, 6 (5), pp. 1 8.

Powell, A., Hunt, A., and Irving, A. (1997), Evaluation of Courses by Whole Student Cohorts: a case study, Assessment and Evaluation in Higher Education, 22 (4), pp. 397 404.

Pounder, J. (1999), Institutional Performance in Higher Education: is Quality a Relevant Concept?, Quality Assurance in Education, 7 (3), pp. 156-163.

Quality Assurance Agency (1997), Subject Review Handbook, October 1998 to September 2000, December, Quality Assurance Agency.

Quality Assurance Agency (2000), Code of Practice for Assurance of Academic Quality and Standards in Higher Education – External Examining, January, Quality Assurance Agency.

Ramsden, P. (1991), A Performance Indicator of Teaching Quality in Higher Education: the Course Expereince Questionnaire, Studies in Higher Education, 16 (2), pp. 129 -150.

Roffe, I. M. (1998), Conceptual Problems of Continuous Quality Improvement and Innovation in Higher Education, Quality Assurance in Education, 6 (2), pp. 74-82.

Skrianthan, G. and Dalrymple, J. (2003), Developing Alternative Perspectives for Quality in Higher Education, International Journal of Education Management, 17 (3), pp. 126-136.

Tam, M. (2002), University Impact on Student Growth: a Quality Measure?, Journal of Higher Education Policy and Management, 24 (2), pp. 211-218.

Veal, A. J. (1997), Research Methods for Leisure and Tourism: A Practical Guide, 2nd Edition, London: Pitman Publishing.

Wiklund, H., Klefso, B., Wiklund, P., and Edvardsson, B. (2003), Innovation and TQM in Swedish Higher Education Institutions Possibilities and Pitfalls, The TQM Magazine, 15 (2), pp. 97 -107.

Wilson, K., Lizzo, A., and Ramsden, P., (1997), The Development, Validation and Application of the Course Experience Questionnaire, Studies in Higher Education. 22 (1), pp. 33 53.

Yorke, M. (1994), Enhancement-led Higher Education?, Quality Assurance in Education, 2 (3), pp. 6-12.

Websites:

The LTSN for Hospitality, Leisure, Sport and Tourism www.hlst.heacademy.ac.uk

The Quality Assurance Agency www.qaa.ac.uk

The LTSN Generic Centre External Examining Project – www.ltsn.ac.uk/genericcentre/index.asp?id=19567

The HEFCE Teaching Quality Information Site www.tqi.ac.uk

Creative Commons License
This paper is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Double Blind Review
This paper has been subject to a double blind peer review by at least two reviewers. For more information about our double blind review process please visit: http://bejlt.brookes.ac.uk/about/double-blind-review/

How to cite this paper.
Tagged with:
Posted in Academic Paper

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Subscribe to BeJLT

Get email alerts when there is a new issue.
* = required field

Send this page to Kindle

your kindle user name:
(you@kindle.com, without @kindle.com)
Approved E-mail:
(Approved E-mail that kindle will accept)
Kindle base email kindle.com | free.kindle.com
(Use kindle.com to download on wispernet or wifi, use free.kindle.com for wifi only.)
using kindle.com may incur charges)