Learning Outcomes and Assessment: developing assessment criteria for Masters-level dissertations

Authors

Abstract

This paper discusses the development of appropriate and effective criteria for the assessment of Masters dissertations. It identifies the features of dissertations which present difficulties in the assessment process and considers the problems faced by assessors, relating these difficulties to the literature. It presents a literature-based approach to developing assessment criteria, related to the learning outcomes of the dissertation modules. The new approach used a grid, and distinguished between first- and second- order criteria. Although the new approach has yet to be thoroughly evaluated, first indications were that it captured the learning outcomes more fully and that it led to greater consistency and agreement between assessors.

Diane Seymour

Introduction

This paper discusses the development of assessment criteria for the Masters programmes in International Hotel and Tourism Management and in International Travel and Tourism Management at Oxford Brookes University. In common with other taught Masters level programmes, a required component is the dissertation. The dissertation is widely viewed as the culmination of a students achievements: a piece of independent research in which knowledge, understanding, originality of thought, ability to analyse, evaluate and apply subject-specific theories and concepts is demonstrated. Student’s are expected to formulate their own research question, to gather and select material to answer their question and to set out their findings in an appropriate academic style. Since students choose their own topics, albeit with guidance, within a multi-disciplinary subject area, the assessment tasks and criteria need to be flexible enough to capture this divergence. Thus the ‘assessment tasks are designed to enable students to demonstrate individuality and diversity’ (Brown, Race, and Smith, 1995, p. 19). It is not therefore possible to prescribe a single type of approach to the investigation. In addition, the particular form which a dissertation takes is influenced by the dominant discipline forming the theoretical foundation for the investigation. However, students are expected to demonstrate the generic abilities to write in a clear and accessible language, to adhere to conventional academic practice, and to sustain a coherent line of argument that is directed towards a topic of interest to International Hotel and Tourism Management or International Travel and Tourism Management.

All of these features present particular difficulties in the assessment of the dissertation. Firstly, the requirement to demonstrate the abilities to analyse, evaluate, and to present a coherent extended argument means that the dissertation cannot be marked simply by assigning marks to each section (e.g. introduction, aims and objectives, literature review, methodology, analysis of findings, discussion and conclusion) and adding them up. Secondly, the dissertation is an extended piece of work (15,000 words); therefore it is likely the quality may vary between the different parts, again making an overall judgement difficult. Finally, since the dissertation is a blend of subject-specific and generic skills, the assessor must try to distinguish between these and decide on the relative weight to attach to each.

The problem

In common with other modules the assessment strategy for the dissertation module measures students’ achievement of the learning outcomes (see Appendix 1), or at least that is what we told ourselves. We developed a set of criteria against which to judge the dissertation learning outcomes (see Appendix 2). A process was put in place which required the submitted work to be independently assessed against the criteria by the supervisor and an internally appointed examiner. The two markers would then meet to agree the final grade.

Several problems were associated with the use of these criteria, many of which are also revealed in the literature on assessment. Firstly, the list of criteria were difficult to interpret in any consistent way and this led to different assessors arriving at different final grades (Saunders and Davis, 1998). An additional problem is that different assessors valued the different criteria as more or less important, a problem which also has been identified in the literature (Webster et al., 2000). Secondly, the different sections of criteria were not mutually exclusive, which meant that some aspects of the dissertation were assessed under more than one heading (for example structure appears in several places). Thirdly, there was no way of weighting the different criteria. If a student did well on one or two and badly or not so well on others, how was the final mark to be agreed? Is presentation as important as, say the literature review? There was therefore sometimes a lack of consistency between the assessments made by different assessors. Further, various studies have demonstrated that there are differences between experienced and non-experienced supervisors’ assessment grades and between those awarded by different subject specialists in multi-disciplinary departments. Thus some disagreements over grading reflected whether or not the lecturer had specific subject knowledge, whether or not they were experienced assessors and how long they spent on the assessment, a problem also identified in the literature (Blackburn and Saunders, 2001; Saunders and Davis, 1998). The combination of these difficulties in applying the criteria resulted in holistic rather criterion-based marking, where assessors ‘had a feel’ for the grade a dissertation deserved and manipulated the comments against the criteria in order to arrive at the ‘right’ mark. This was compounded by an absence of grade descriptors against each criterion, leaving the question of the difference between different grades of dissertation very much open to question. When supervisors and internal examiners met to discuss the final mark, it was therefore often a process of negotiation, including special pleading for consideration of factors not included in the criteria, for example, how hard the student had worked, how conscientious they had been and so on.

New approach

The literature on the development and use of assessment criteria was examined for examples of good practice, and for the discussion of experiences in other subject areas and other educational institutions. It became clear that a variety of approaches are adopted, ranging from global assessment where dissertations are assessed for a general impression of their adequacy (and the criteria are used as a sort of checklist to ensure that all areas have been considered), to analytic assessment, where work is examined against each criterion to which a proportion of the total mark has previously been allocated, the marks then being added up to arrive at a final mark (McDonald and Sansom, 1979). Neither of these approaches seemed to offer a solution to the difficulty of ensuring consistency. In addition, some of the analytical approaches to assessment resulted in long lists of items to be considered within each criterion, which would seem to increase the likelihood of inconsistent assessments rather than reduce it (for example Saunders and Davis, 1998). Informal discussion with colleagues suggested that there was a shared view that in addition to specific criteria such as the quality and range of the literature review, the methodological approach and so on, there was a holistic criterion that reflected the strength of the overall argument, the coherence of the dissertation and the level of critical awareness which was crucial in distinguishing between the different grades of the final assessment of the dissertation. This approach has been adopted with success elsewhere in other subject areas (Pilkington and Winch, 2000) and was developed for use on the Masters dissertations.

A grid was produced (Appendix 3) which reworked the existing criteria in order to produce more logical groupings within each criterion category. It contains a first-order criterion which captures the holistic importance of the strength and internal consistency of the overall argument, the use and integration of information gathered, and a critical awareness of strengths and weaknesses. Grade descriptors were developed to distinguish between Distinction, Merit, Pass, Refer and Fail grades. Five second-order criteria were then developed, each with its own grade descriptors. The first-order criterion plays a key role in arriving at the final grade for the dissertation. A dissertation cannot be awarded a class above that given to this criterion, whatever the performance in the other categories. The second-order criteria are designed to pick up other features of the dissertation which relate to the learning outcomes. A dissertation must receive a grade for at least three of these five criteria that matches the grade achieved in the first-order criteria in order to be awarded a final grade at that level. Less than three would mean that a lower grade would be awarded, though we felt that this was unlikely to occur given the holistic importance of the first-order criterion. As assessment form was developed from the grid (see Appendix 4), which lists the criteria on the left hand side and then has room for typed comments against each criterion on the right. Assessors were asked to insert comments to justify the grade awarded for each criterion.

The method of blind double marking was continued, with each assessor completing the assessment form fully, assigning a grade to each of the criteria. There was then a meeting between the two assessors to decide the final mark. It was agreed that should there be initial disagreement over the grade awarded for the first-order criterion that cannot be resolved, a third assessor would be appointed.

How did it work?

This was piloted for the first time this term, and so there has not yet been time to evaluate the new approach fully. What follows is based on an examination of the completed assessment forms and on informal one-to-one conversations with some of the assessors. Assessors were asked to talk about their impressions of using the grid generally and for any problems they experienced. They were also asked whether they found that they had arrived at a final grade more easily and whether they found they were in broad agreement with the second assessor at the meeting to decide the final mark.

Examination of completed forms

The new scheme appeared to result in fewer significant disagreements over the final grade. In only one case was it necessary to appoint a third assessor. In the vast majority of cases there was also agreement about the grades awarded for each of the second-order criteria.

Assessors’ reactions and experiences

Assessors reported some initial difficulties of interpretation and understanding of the role of the first-order criterion and some assessors did not at first insert grades into the second-order criteria and had to be asked to go back and do this before meeting to agree the final mark. These initial difficulties were resolved by discussions with the module leader or with myself, as and when necessary. They did say that once they got used to using the new scheme they found it much easier to arrive at judgements of the dissertations, and many were struck by the level of agreement assessors reached independently on each of the criteria. Once the grades were arrived at, assessors tended to specify ‘high merit’, ‘low pass’, etc, which they said made it extremely easy to identify any areas of difference between assessors and discuss the reasons behind the difference in assessment. The discussion of the final percentage mark to be awarded in the overall grade was then easily achieved.

Issues arising

The literature emphasises the importance of the ownership and shared understanding of any criteria used for assessment both among assessing staff (Saunders and Davis, 1998; Pilkington and Winch, 2000) and students (Price and Donovan, 2003). Ideally therefore, there should have been extensive discussion of the new assessment grid before the pilot took place. However, this did not occur as there was insufficient time before the current round of assessment, and the module leader for the dissertation was so dissatisfied with the existing framework that it was decided to operationalise the new one straightaway. Additionally, it was felt that the criteria themselves were not significantly changed, rather, they were simply grouped together differently, and of course there was the prioritising of the first-order criterion. However, this was not thought likely to cause difficulty since assessors had in any case been using, albeit implicitly, a holistic impression of the importance of argument and critical awareness to arrive at their final judgement. The new format was therefore circulated with instructions, and individual discussion with assessors were held during the assessment period to clarify any issues. This was clearly not ideal, however, and more widespread discussions are planned for later this term in which assessors will explore their understanding and use of the criteria.

Future plans for development

The next cohort of students will be invited to engage with the assessment criteria through workshops at which they will be asked to use the criteria to assess work previously submitted and assessed. We hope that by actively engaging with the criteria they will come to a better understanding of what is required for the dissertation and that this will enhance their performance.

Conclusions

We feel that the new scheme more accurately reflects the learning outcomes for the dissertation module and that the new assessment grid makes the criteria for assessment and the level of achievement for each grade much more transparent. This should help both staff and students. The dissertation is the culmination of a years study and is intended to be a vehicle which students can use to demonstrate their achievements. It is therefore crucial that the mark awarded is a fair reflection of the extent to which students demonstrate their achievement of the learning outcomes. The old system did not manage to do this. There are undoubtedly problems to be ironed out with the new system. In particular the way in which it was introduced was not ideal and I would not recommend introducing a change like this without extensive prior consultation and discussion. On the other hand, there never does seem to be time to arrange such meetings, and if this means that change cannot be introduced, then that is not an ideal situation either! What has resulted from this exercise is a heightened interest in discussion of the assessment of the dissertation that we hope will result in improvements to the grid and a greater feeling of shared ownership after meetings at the end of this term.

Biography

Diane Seymour is Senior Lecturer in Sociology in the Business School where she is also Academic Conduct Officer for undergraduate programmes. She was International Students’ Tutor in the former School of Hotel and Restaurant Management and Equal Opportunities Advisor.

Contact Details

Diane Seymour
The Business School
Department of Hospitality, Leisure, and Tourism Management
Oxford Brookes University
Gipsy Lane Campus
Headington
Oxford OX3 0BP
Telephone: 01865 483822
Email: dseymour@brookes.ac.uk

References:

Blackburn, A. and Saunders, M (2001) Dissertation supervision and assessment practice. Paper delivered at Assessment Conference, Oxford Brookes University

Brown, S. and Glasner, A (eds) (1999) Assessment Matters in Higher Education Buckingham: Open University Press

Brown, S., Race, P. and Smith, B. (1995) 500 Tips on Assesment. London: Kogan Page

Freeman, R. and Lewis, R. (1998) Planning and Implementing Assessment. London: Kogan Page

Gibbs, G., Habeshaw. S and Habeshaw, T. (1988) 53 Interesting Ways to Assess your students. Bristol: TES Ltd

McDonald, R and Sansom, D (1979) Uses of assignment attachments in assessment. Assessment in Higher Education. Vol. 5 No. 1, pp. 45-55

Pepper, D and Webster, F (1998) The Assessment of Undergraduate Dissertations in the School of Social Sciences and Law Oxford Brookes Occasional Paper

Pilkington, A. and Winch, C (2000) Assessment Strategies and Standards in Sociology University College Northampton

Price, M. and Donovan, B. (2003) Improving Students’ Learning by Developing Their Understanding of Assessment Criteria and Processes Assessment and Evaluation in Higher Education (awaiting publication)

Saunders, M. and Davis, S. (1998) The use of assessment criteria to ensure consistency of marking: some implications for good practice. Quality Assurance in Education. Vol. 6 No. 3 pp. 162-171

Webster, F., Pepper, D, and Jenkins, A. (2000) Assessing the undergraduate dissertation. Assessment and Evaluation in Higher Education. Vol. 25, No. 1

Appendix 1

Learning outcomes

Knowledge and understanding

You will be able to:

  • Apply knowledge and skills acquired throughout the programme
  • Display an independent approach to critical analysis and evaluation and engage in debates relevant to the area of study undertaken
  • Demonstrate the development of coherent and sustained arguments leading to logical conclusions
  • Reach accepted standards of written presentation

Professional/Disciplinary Skills

You will be able to:

  • Develop an in-depth knowledge and understanding of a relevant area of international hotel and tourism management theory and practice
  • Make informed choices appropriate to the chosen research topic form the alternative research methods available
  • Provide evidence of proficiency in the research methods applied to the chosen topic
  • Collect, analyse and interpret data
  • Effectively present the results of the research for both academic and industry audiences

Appendix 2

Original assessment criteria

Presentation:

  • Complies with Dissertation guidelines, correctly referenced, complies with specified length and submitted on time
  • Lucid exposition, focused, well structured and professionally presented
  • Appropriate academic style, free of spelling/typing mistakes and grammatical errors

Structure:

  • Use of a logical, coherent structure which presents the argument in a clear way
  • Includes an introduction, rationale for the study, aims, review and evaluation of theory and practice, discussion of methodology, presentation of findings, discussion and conclusions/recommendations

Literature Review:

  • Evidence of extensive research (breadth and depth) using a range of appropriate and up-to-date sources including books, journals, reports the Internet and trade press (where appropriate)
  • Evidence of evaluation and selectivity in the material presented
  • Identifies and develops concepts from analogous applications in other areas
  • Able to identify themes, structure ideas and present convincing argument

Analysis and Evaluation:

  • Demonstrates a clear understanding of the subject through critical interpretation and the analysis of issues
  • Evidence of the evaluation of concepts in relation to practice and theory
  • Analyses and evaluates analogous applications from other areas
  • Evaluates the strengths and weaknesses of the chosen approach to the study

Conclusions:

  • Reviews findings within the context of original research aims
  • Draws together the findings of primary research and literature review to develop conclusions about the subject and the contribution of the dissertation to knowledge in the subject area
  • Extends material into new areas, ideas and/or applications
  • Develops recommendations for the industry and possibly the development of theory
  • Identifies opportunities for further research

Appendix 3

MSc Dissertation Assessment marking criteria and grid.

It has been clear for some time that the criteria we use to mark dissertations needed reworking. Many of us found it difficult to use the old grid. From discussion with colleagues there seemed to be two major difficulties:

  1. The descriptors in each of the criteria boxes were not exclusive so it was sometimes difficult to avoid crediting, say structure in several different places.
  2. Where a dissertation was strong in some areas and weaker in others, the final grade rather depended on the particular academic values of the assessors.

Below is the new assessment grid for dissertation marking.

  • The criteria have been reworked to try to make a more logical collection in each category and grading guidelines for each criterion are suggested.
  • Markers will have a form which is blank apart from the list of criteria down the left hand side and will insert in the boxes the grade and the justifications for the grade for each criterion using the generic guidelines.
  • Specific percentages will be agreed at the meeting of supervisor and examiner. In order to overcome the problems of holistic marking (I know a Distinction when I see it), the criteria carry different weightings.
  • First order criterion: the first criterion, that of argument, is to be treated as a first order criterion. This means that a dissertation cannot be awarded a class of degree mark above that given to this category, whatever the performance in the other categories. This reflects the importance of the overall argument, its coherence and consistency, integration and use of information to sustain argument, logical progression and critical awareness etc. to work at this level.
  • The second order criteria are designed to pick up specific features of the dissertation which contribute to its overall coherence. A dissertation should receive a grade for three of these that reflects the standard achieved of the first order criterion in order to achieve that mark. Less than three would mean the grade below (though in practice I suspect that this is unlikely to happen very often.)

Marking procedures

At various meetings, colleagues have commented that the method used last year of blind marking meant that they were unable to gain from the meeting with the other assessor to agree a mark. However, the process of blind double marking (where the mark awarded by each assessor is unknown, and the final mark mediated by the module leader) is good practice and valued by the external examiner. This year a compromise is suggested. Each assessor should mark the dissertation and complete the form fully. There will then be a meeting between the two assessors to agree a final mark. Both copies of the form will be given to the module leader. Where there is initial disagreement over the grade (rather than the percentage) awarded for the first order criterion, and agreement is subsequently reached, a third form giving the justification for the final grade agreed should also be submitted. Where no agreement can be reached, the module leader will assign a third assessor.

Appendix 4

The New assessment and Marking grids are given below.

MSc Dissertations in Hotel, Travel and Tourism Management – Marking Criteria
First order criterion Distinction 75+% Merit 65-74% Pass 50-64% Refer 25-49% Fail 0-24%
Argument

Internal consistency

Use of information to sustain argument

Awareness of strengths and weaknesses

Extremely strong internal consistency making the project a convincing whole which addresses the original research question. Evidence of originality. Impressive use of information gathered to support argument. Critical awareness of strengths and limitations Evidence of internal consistency which relates to original question. Very good use of information gathered to support argument. Awareness of strengths and limitations Evidence of internal consistency which relates to original question but with some weaknesses in the integration of different sections. Use of information gathered but with some weaknesses in the integration of evidence. Some awareness of strengths and weaknesses Limited evidence of internal consistency which relates to the original with significant weaknesses in the integration of different sections. Limited use of information gathered to sustain the argument with significant weaknesses in the integration of evidence. Limited discussion of strengths and weaknesses. Lack of internal consistency. Very limited use of information gathered to sustain the argument with serious weaknesses in the integration of evidence. No awareness of limitation of the dissertation.
Second order criteria
Research problem

Formulation

Focus

Rationale

Very clearly formulated research question. Clear subject based focus. Excellent and convincing rationale. Clearly formulated research problem. Evidence of subject based focus. Clear and well thought through rationale. Competently formulated research problem with some evidence of subject focus. Competent rationale is provided Poorly formulated research question. Lacks subject focus. Rationale poorly articulated and justified. Incoherently formulated research question. Inadequate rationale.
Use of theory

Critical awareness of relevant theory

Analysis and evaluation

Grounding in theory

Extensive and critical awareness of and grounding in theory. Convincing evidence of ability to analyse, evaluate and apply theory. Clear and critical awareness of and grounding in theory. Very good evidence of ability to analyse, evaluate and apply theory. Generally clear awareness of and grounding in theory. Good evidence of ability to analyse, evaluate and apply theory. Some limited awareness of and grounding in theory. Little evidence of ability to analyse, evaluate and apply theory. Little awareness of and grounding in theory. Inadequate evidence of ability to analyse, evaluate and apply theory.
Literature review

Range of reading

Relation to research question

Independent research

Extensive reading which has been thoroughly critically evaluated and explicitly related to the research question.

Very good evidence of independent research for sources.

Wide reading with critical evaluation and clearly related to the research question. Good evidence of independent research for sources. Appropriate reading with some limited evaluation. Not consistently clearly related to the research question. Some evidence of independent research for sources. Reliance on limited sources, lack of evaluation. Poorly related to research question. Little evidence of independent research for sources. Over reliance on very restricted range of sources. Not related directly to research question. Very little evidence of independent research for sources.
Methodology

Appreciation of methodological issues

Rationale for research approach

Information gathering and analysis.

Awareness of strengths and weaknesses.

Very clear appreciation of relevant methodological issues. Excellent rationale for research approach adopted and the data collection methods used. Extremely systematic and appropriate information gathering and analysis. Critical awareness of the strengths and weaknesses of the approach taken. Very good appreciation of relevant methodological issues. Clearly presented rationale for research approach adopted and the data collection methods used. Very competent and appropriate information gathering and analysis. Some awareness of strengths and weakness of approach taken. Familiarity with key methodological issues. Competent rationale for research approach adopted and the data collection methods used. Competent information gathering and analysis. Some awareness of the strengths and weaknesses of the approach taken. Limited awareness of methodological issues. Defensible rationale presented for research approach adopted and the data collection method used. Weak information gathering and analysis but sufficient information gathered to allow for a possible reworking of data. Little awareness of strengths and weaknesses of approach taken. Little awareness of methodological issues. Inappropriate or non-existent rationale presented for the research approach and the data collection methods used. Poor and inappropriate information gathering and analysis, not capable of being reworked.
Presentation and expression

Referencing

Presentation

Use of language

Fully and appropriately referenced, well presented. Excellent use of language. Very good referencing, well presented and clear use of language. Generally well referenced, well presented and clear use of language but with some errors. Competent referencing but some inconsistencies. Adequately well presented. Clear use of language but with significant errors. Poorly referenced, poorly presented and very unclear language with serious errors.

MSc Dissertations in Hotel, Travel and Tourism Management Assessment grid

First order criterion Comments and justified grade
Argument

Internal consistency

 

Use of information to sustain argument

 

Awareness of strengths and weaknesses

Second order criteria
Research problem

Formulation

Focus

Rationale

Use of theory

Critical awareness of relevant theory

Analysis and evaluation

Grounding in theory

Literature review

Range of reading

Relation to research question

Independent research

Methodology

Appreciation of methodological issues

Rationale for research approach

Information gathering and analysis.

Awareness of strengths and weaknesses.

Presentation and expression

Referencing

Presentation

Use of language

Creative Commons License
This paper is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Double Blind Review
This paper has been subject to a double blind peer review by at least two reviewers. For more information about our double blind review process please visit: http://bejlt.brookes.ac.uk/about/double-blind-review/

How to cite this paper.
Tagged with:
Posted in Academic Paper

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Subscribe to BeJLT

Get email alerts when there is a new issue.
* = required field

Send this page to Kindle

your kindle user name:
(you@kindle.com, without @kindle.com)
Approved E-mail:
(Approved E-mail that kindle will accept)
Kindle base email kindle.com | free.kindle.com
(Use kindle.com to download on wispernet or wifi, use free.kindle.com for wifi only.)
using kindle.com may incur charges)