ASKe Manifesto seven years on: so what did change?

Abstract

Assessment continues to be a major challenge to institutions around the world. A challenge in terms of student satisfaction, a challenge in terms of resourcing (there are few economies of scale in assessment (Gibbs 2006) and a challenge in terms of transparency, reliability and validity to name but a few. One of the major problems is that there are very varied levels of understanding of assessment and its processes among stakeholders. This is unsurprising given the complex nature of assessment but it causes difficulties and unintended consequences especially when lack of understanding is found among all stakeholder groups: students, staff, management, quality assurance experts, government. If we are going to meet the challenges we face, the assessment literacy (see Price et al, 2012) of all these groups needs to improve.

As detailed in the original BeJLT paper (Price et al 2008) ASKe brought together a group of international assessment experts to envisage a new future for assessment; the result was the ASKe Assessment Manifesto. Through the Manifesto we hoped to stimulate debate across the sector and, by implication, influence ways of thinking about and practising assessment. So what happened? This paper will reflect on the impact of the Manifesto both locally and in terms of its reach beyond the institution, as well as considering its continued relevance.

Inside Brookes

It is pleasing for those of us involved in the development of the Manifesto that within Brookes it inspired and informed the Assessment Compact (a commitment between the University and the student body to a set of rights and responsibilities regarding assessment). The compact contextualises the Manifesto for use by Brookes and aims to bring about change rather than just consolidate current practice. It is both values and evidence-based, and seeks to reconceptualise thinking about assessment and feedback in the institution.

It was recognised that many of the ideas to be encapsulated in the Compact were complex and many staff and students had not previously been asked to think deeply about the assessment process. Engagement with, and commitment to, the Compact would inevitably mean some major changes to assessment processes, practices and attitudes. The Compact had to have academic integrity and also recognise the need for contextual interpretation and application of relevant principles (e.g. for different disciplines, courses etc.).

Consequently the final draft of the Compact was developed by a cross-university staff and student working group and its introduction and implementation supported by programme based ‘assessment design intensives’ (see here). Evaluation of the Compact’s impact has shown that over several years there has been some change in the way that many staff talk about and practice assessment. This in turn is having an impact on students’ understanding and approaches to assessment. In line with the Manifesto’s emphasis on holistic change in institutions’ approach to assessment, the Compact has also informed policies and practices removed from the direct teaching interface, including review of quality procedures e.g. the sort of questions validation panels should be asking, and development of a Brookes engagement survey based on the NSSE.   However higher education is in thrall to change and there are many competing agendas and initiatives which means Brookes will need to be vigilant in retaining its commitment to the Compact and the long term support needed to truly embed it and bring about the rewards envisaged. (For more detail on the development and introduction of the Compact see Rust et al, 2013)

Other institutions

Very quickly after its publication the Manifesto attracted the attention of institutions across the UK and beyond. Members of the ASKe team supported that interest through keynotes and workshops and found that the ideas contained within the Manifesto resonated with many practitioners from a huge range of disciplines. There was a need for support in how to change practice especially without any extra resource. ASKe’s growing series of 123 leaflets (see here) offer some practical advice and several institutions have developed their own Manifesto inspired initiatives with some publicly acknowledging ASKe’s work as an influence

‘The Weston Manor Manifesto has been discussed across our university at course team level right up to Academic Board. This document has offered a useful reference point for our assessment enhancement.’

Professor Susan Orr, Deputy Dean Faculty of Arts, National Teaching Fellow, York St John University 2005

Nationally

The impact of the Manifesto at national level has been in three main domains: influence on students through the National Union of Students (NUS), influence on staff through the Higher Education Academy and influence on government through a select committee.

Unfortunately, NUS representatives were unable to attend the event which generated the Manifesto but in 2008 they were developing a major campaign focused on assessment and feedback and they made use of the Manifesto to shape the ideas and direction of their thinking (Bols and Wicklow 2013). This enabled a useful and constructive relationship, aimed at bringing about changes in understandings of the assessment process, in which students could be change agents.

Influencing staff at a national level came first through subject centre interest with disciplines discussing the relevance of the Manifesto to their communities. Subjects as diverse as engineering, philosophy, business, geography, education, and law engaged with ASKe initiatives including the Manifesto (e.g. see McConlogue et al 2009). In particular the  Hospitality, Leisure, Sport and Tourism centre, and the Law centre took up the challenge of exploring the practicalities of fulfilling tenet 6, seeking to set up ‘forums  for the development and sharing of standards within …. disciplinary and professional communities’ (see Rust, 2009).

More recently the Higher Education Academy has endorsed the Manifesto by adopting it as the basis of a new initiative called ‘A Marked Improvement’, designed to support HEIs to transform their assessment practices. The Manifesto’s tenets are used both to set out the new ways necessary to think about assessment and also to frame a self-evaluation tool for HEIs and programme leaders to assess their own practices and areas for development.  Early reports from the initial pilot scheme, working with eight HEIs, suggests that it has been very successful in changing the vocabulary and nature of the discourse about assessment in those institutions, and in influencing changes in both policy and practice.

The third and completely unanticipated opportunity to influence Government thinking came in 2009 with the setting up of the Innovation, Universities, Science and Skills Select Committee inquiry into ‘Students and Universities.’ We prepared a submission to the committee including the ideas and detail from the manifesto and in 2010 we were called to give oral evidence. It was clear from the final report that the committee accepted many of our arguments, as illustrated in their “serious grounds for concern” about assessment methodologies and statistical practices (p116).  Unfortunately, with the change in Government in 2010, the select committee report had virtually no practical impact.

Internationally

Interest from around the world has come from a variety of sources, quality assurance agencies (e.g. Australian Universities Quality Agency and Norwegian Agency for Quality Assurance in Education), institutions looking for guidance about how to improve assessment and feedback (e.g. University of Sydney, University of Hong Kong, University of Trondheim, BI Oslo) and other projects and centres seeking to change assessment in HE (e.g. see National Academy for Integration of Research, The Manifesto also directly influenced David Boud who sought to develop an impetus for change within the Australian HE sector, both in terms of methodology and message. He drew on assessment experts worldwide to develop Assessment 2020 (see here) and support understanding and adoption of its ideas through a series of events for leaders of HEIs across Australia.

Another initiative also in Australia, inspired by, and founded on, Tenet 6 of the Manifesto is a nationwide project seeking to calibrate and establish common assessment standards used by examiners. The project, “Achievement Matters”, (see http://achievementmatters.com.au) started in 2010 and uses a ‘cultivated community’ approach.  Initially funded by the Australian Learning and Teaching Council and focussed on the subject of Accountancy it has recently been announced that it is being widened to include three further projects in Economics, Finance, and Tourism, Hospitality and Events.

Conclusion

So we think we can claim that the Manifesto and the ideas behind it did stimulate debate and that debate has continued and developed among practitioners and within the research community (e.g. Bloxham and Boyd 2011, Nicol et al 2014, Yang and Carless 2013). It seems that the ideas within the Manifesto remain as, or perhaps even more, relevant today as 7 years ago. The original paper concluded that ‘a review and evaluation of the allocation of time and resources within learning, teaching and assessment in higher education and the establishment of dialogic processes to assure standards’ was necessary. So have we got there yet? No we can’t claim as much as that; it is still work in progress, but one with momentum.

Assessment and feedback remain at the top, or close to the top, of agendas for higher education institutions, kept there by largely by student (dis)satisfaction rather than an overwhelming acceptance, or even acknowledgement, of the arguments underpinning the Manifesto. However, hopefully this reflection on the impact of the Manifesto shows that the ideas are resonating with a diverse and significant groups and it looks as if the momentum will continue. Those institutions which have tried numerous simple initiatives aimed at dampening dissatisfaction through a focus on the ‘service’ students receive have failed to see results in terms of sustainable change in measures of satisfaction. Consequently, more of them are now beginning to look at the idea of holistic change, despite it being more challenging to embed and with a need for evaluation over the long term.

In addition there is discussion about whether the right questions are being asked about satisfaction.  Underpinning the Manifesto is an assumption about the importance of assessment literacy not only for staff but also students. If we can support students to more easily understand and see the link between assessment and learning, understand its multiple purposes, recognise the imprecision of assessment standards within the complexity of learning in higher education, alongside supporting them in working with those standards, their perception of satisfaction and learning is likely to change and they will be able to more effectively contribute to discussion and debate in disciplinary communities.

Seven years on the Manifesto seems to be weathering the storm of the dynamic environment of higher education and it has even been lauded by some as an important influence on change agents in the sector.  Professor Sally Brown, in selecting her top 20 influential books for educational developers, asserts that the producers of the Manifesto have been

‘highly influential in developing a climate in which assessment innovations and authentic approaches become recognised as inevitable for promoting effective student learning’ (Brown, 2013 p 326)

So it seems that the Manifesto has had an impact, caused some change and has not yet outlived its usefulness. What will be the position in another seven years?

References

Bloxham, S. and Boyd, P. (2011). Accountability in grading student work: securing academic standards in a 21st century quality assurance context, British Educational Research Journal, iFirst Article, pp1-20.

Bols, A. and Wicklow, K (2013) Feedback – what students want. In Merry, S.,Price,M., Carless, D. and Tarras, M. Reconceptualising feedback in Higher Education: developing dialogue with students (pp19-29) London: Routledge

Brown, S (2014) The 20 books that influenced educational developers’ thinking in the last 20 years: opinion piece, Innovations in Education and Teaching International, 50 (4) pp321-330

Gibbs, G (2006) How assessment frames student learning. In K. Clegg & C. Bryan (eds) Innovative assessment in higher education (pp23-26). London:Routledge

House of Commons Innovation, Universities, Science and Skills Committee (2009) Students and Universities: Eleventh Report of Session 2008–09 Vol. 1. London: The Stationery Office.

McConlogue, T., Mueller, J. and Shelton, J (2009) Challenges of developing engineering students’ writing through peer assessment. Available at: http://www.heacademy.ac.uk/assets/documents/subjects/engineering/EE2010/90_GP_McConlogue.pdf

Nicol,D., Thompson, A. and Breslin, C (2014) Rethinking feedback practices in higher education: a peer review perspective. Assessment and Evaluation in Higher Education, 39 (1) pp102-122

Price, M., O’Donovan, B., Rust, C. & Carroll, J. (2008) Assessment Standards: A Manifesto for Change, Brookes eJournal of Learning and Teaching, 2 (3) available at: http://bejlt .brookes .ac .uk/article/assessment_standards_a_manifesto_for_change

Price, M., Rust, C, O’Donovan, B and Handley, K (2012) Assessment Literacy: the foundation for improving student learning, Oxford: Oxford Centre for Staff and Learning Development

Rust, C. (2009) Assessment standards: a potential role for Subject Networks, Journal of Hospitality, Leisure, Sport, and Tourism Education, 8 (1) 124-128  Available at: http://www.heacademy.ac.uk/johlste/vol8/vol8no1

Rust, C. Price, M. Handley, K. O’Donovan, B and Millar, J. et al, (2013) An assessment compact: changing the way an institution thinks about assessment and feedback. In Merry, S., Price, M., Carless, D. and Tarras, M. Reconceptualising feedback in Higher Education developing dialogue with students (pp19-29) London: Routledge

Yang, M. and Carless, D (2013) The feedback triangle and the enhancement of dialogic feedback processes. Teaching in Higher Education 2013, 18 (3) p285-297

Margaret Price

Margaret Price is a Professor of Assessment and Learning and Director of ASKe Pedagogy Research Centre. She has been a visiting scholar at University of Sydney and Hong Kong University, is an advisor on projects at institutions in Scandinavia and the UK, and she was awarded a National Teaching Fellowship in June 2002. Margaret is involved in a wide range of research and development of practice projects both within Brookes and at a national and international level. She seeks to encourage innovation and foster evidence-based assessment practice within the HE sector.

Chris Rust

Chris Rust is Professor of Higher Education, and Associate Dean (Academic Policy) at Oxford Brookes University. Previously he was Head of the Oxford Centre for Staff and Learning Development, and Deputy Director of the Human Resource Directorate, and has also been Deputy Director for two Centres for Excellence in Teaching and Learning – ASKe, and the Reinvention Centre. He has researched and published on a wide range of subjects, especially assessment. He is a Senior Fellow of the Staff and Educational Development Association, a Fellow of the RSA and was one of the first 14 Senior Fellows of the UK Higher Education Academy.

Berry O’Donovan

Berry O’Donovan is a Principal Lecturer Student Experience in the faculty of Business at Oxford Brookes. She researches and publishes on assessment and feedback and the intellectual development of management undergraduates, and has been centrally involved in the bidding for and implementation of many externally funded projects, including the £4.5 million ASKe Centre for Excellence. Berry describes herself as a ‘bundled’ academic who is involved in research, teaching and project management. Her teaching centres on compulsory introductory and capstone joint-honours modules with a focus on synoptic learning in business and management. She leads a cross-institutional project on student engagement and externally leads a development programme for Directors of Teaching and Learning (Associate Deans) for the UK’s Association of Business Schools.

Creative Commons License
This paper is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Double Blind Review
This paper has been subject to a double blind peer review by at least two reviewers. For more information about our double blind review process please visit: http://bejlt.brookes.ac.uk/about/double-blind-review/

How to cite this paper.
Posted in Academic Paper

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Subscribe to BeJLT

Get email alerts when there is a new issue.
* = required field

Send this page to Kindle

your kindle user name:
(you@kindle.com, without @kindle.com)
Approved E-mail:
(Approved E-mail that kindle will accept)
Kindle base email kindle.com | free.kindle.com
(Use kindle.com to download on wispernet or wifi, use free.kindle.com for wifi only.)
using kindle.com may incur charges)