An international perspective: Not just a top cited article

Authors

Mark Freeman


It was no surprise to hear that Assessment Standards: A Manifesto for Change, by Margaret Price, Berry O’Donovan, Chris Rust and Jude Carroll et al (2008), was one of the most-cited papers in BeJL&T’s first decade. This commentary seeks to describe how the article is more than just popular but a proxy for so much more…the veritable tip of the iceberg.

The paper epitomises a number of themes evident in the Assessment Standards Knowledge exchange (ASKe) and the authors’ work in and beyond ASKe and the United Kingdom. The first relates to a long running theme around the critical role of social constructivism in learning, not just for students but also for academics’ teaching, academic leaders of programs, organisational units and even higher education institutions. In the paper the authors not only stand on the shoulders of others by drawing on previous published work to show where assessment is broken, but they also capture the voice of a group of 40 international assessment experts in the 2007 forum (Weston Manor Group).  Reflections and tacit knowledge were exchanged in dialogue to construct a consensus of where assessment practice is deficient before finally synthesising six clear and unambiguous tenets as solutions. Having referred to these tenets in at least 100 public presentations I can vouch for them from personal experience – and my favourite is tenet 6!

Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities.

Dialogue and exchange are part of the DNA of the approach to supporting change in learning and teaching, not just at the core of this paper, or the authors and other members of the ASKe centre for excellence in teaching and learning, but in multiple aspects of Oxford Brookes University’s work over the years.   The first notable example of the wider role is the work of the Oxford Centre for Staff and Learning Development (OCSLD), one of the largest and most established providers of staff and educational development in the United Kingdom. Not only does it provide internal support for academics around educational development and pedagogical research, the OCSLD also annually run up to 200 different courses on 40 different topics for universities in the UK and overseas.  These academic development courses focus on teaching, learning, assessment and e-learning for both teaching academics and academic managers and leaders. To achieve this the OCSLD involves a large national network of specialist consultants.

Furthermore, the OCSLD has spawned a range of highly influential academic researchers and change agents particularly around assessment as well as learning and teaching more generally. Professor Graham Gibbs, its inaugural director, is one such recognised international expert whose books, research articles and change programs continue to be highly valued and have an impact.  Publications like Implications of ‘Dimensions of quality’ in a market environment (Gibbs, 2012) and projects like Transforming the Experience of Students through Assessment (TESTA), both commissioned by the Higher Education Academy.  Another significant earlier project was Teaching more students commissioned by the Polytechnic and Colleges Funding Council.  All have been influential in the UK but they have also been highly useful for academics and academic leaders facing similar environments beyond the UK where higher education sectors face simultaneous challenges of greater quality accountabilities, demands for greater participation including by socio-economically disadvantaged groups, reductions in public funding and increased competition for international enrolments.

A second notable example of Oxford Brookes University’s wider role where this social constructivist approach has been evident relates to the HEA Subject Centres. From my perspective, the subject centres were the jewel in the HEA crown, until the funding constraints led to their demise, because they enabled frontline academics to more easily engage in learning and teaching development and innovation.  Subject centres were a place where discipline academics could feel safe if they didn’t know the education jargon. Subject centres hosted a range of opportunities for dialogue with colleagues from the discipline who understood their context and jargon and who could translate what might otherwise appear as ‘edubabble’ or collaborate to explore a solution, or lean on someone who had been down that road before and learn from their experience. While there were only 26 subject centres across the UK, it is entirely consistent that Oxford Brookes University punched above its weight and was host to not one but two of these subjects centres – one for business, management, accountancy and finance, and the second for hospitality, leisure, sport and tourism. Both subject centres also launched and hosted their own pedagogical journals and produced various other media for engaging frontline academics in conversations around teaching, learning and assessment.

Returning to the paper, its authors have walked the talk, particularly around assessment, in a range of other UK and international contexts. Since its publication in 2008, all four authors of the Assessment Manifesto BeJLT article have travelled to Australia to facilitate a series of assessment-related events at various universities in the state capital cities. These events were sponsored and organised through the Teaching and Learning Network of the Australian Business Deans Council, a network of associate deans responsible for learning and teaching in their respective business schools. The events primarily took place in business schools but invitations to engage taken up by academics beyond business schools. While the formal workshop series were pitched at frontline academic staff to support evidence-based improvements in assessment (and feedback) practices, meetings with program directors and associate deans were also held to assist more strategic conversations. These focussed on enabling and supporting changes in assessment and feedback to occur. Appointments with individuals or small groups of interested academics were also encouraged. Some of these resulted in more fine-grained advice and ongoing collaboration, at times leading to further research outcomes as in the case of Bell, Mladenovic and Price (2013).

And then there’s Berry O’Donovan’s work with the Association of Business Schools, not least as lead facilitator on the innovative annual development program for directors of learning and teaching in business schools. Or Chris Rust’s work with the prestigious international business school accreditation organisation, the Association to Advance Collegiate Schools of Business, who has been providing leadership around systematic assurance of learning since 2003. Or Jude Carroll’s work around the world in regard to assessment and academic integrity.

A second theme, throughout the paper and beyond, is their commitment to challenging the status quo and promoting change and continuous improvements using evidence. The 2008 paper clearly shows that assessment practices left much to be desired. Recognising engagement with change is more likely to occur if the assembled evidence raises a healthy dissatisfaction with the status quo, after laying this foundation they provided a manifesto for change around six simply expressed but profound tenets.

A more recent example is that of Bloxham and Price (2013) who call into question the very efficacy of national systems for assuring academic standards, arguing that “external examining rests on assumptions about standards which are significantly open to challenge”.  This theoretical analysis was then recently extended by empirical work that uncovered outcomes with serious potential to challenge the status quo particularly in relation to the reliance on external examiners to assure academic standards and provide advice and recommendations to higher education institutions.  Jointly commissioned by the Quality Assurance Agency (QAA) and the Higher Education Academy (HEA), the research involved 24 experienced examiners from four disciplines (chemistry, history, psychology and nursing) to consider five 2i/2ii borderline examples of student work. Of the 20 assignments used in the research, only one was assigned the same rank (highest or joint highest) by all six examiners in a discipline.  Furthermore, nine (of 20) assignments were ranked both worst and best by different examiners. This led to a somewhat understated conclusion (QAA, 2014).

This research suggests that there are some issues that need to be addressed in terms of the level of engagement with the guidance, and the sector should take that seriously.

Yet this is not to say that the authors have had an unambiguously clear and long journey of success. And with true humility they have been the first to point it out.  In one particular case, in the early stage of their assessment research agenda, they recognised where their own and others’ work was leading up a blind alley. Reiterated in Price et al (2008), they recognised the excessive and unhelpful focus of a comprehensive marking criteria grid, also known as an assessment rubric, which failed to establish a common standard.

However, the dominant model of sharing of standards in higher education relies on explicit criteria and level descriptors which have been shown to be inadequate because many aspects of quality cannot be communicated through explicit criteria (Rust et al, 2003)

The final theme, and most important from my perspective, relates to the willingness of the ASKe members to cultivate a global assessment community.  Evidence for this can be as simple as the fact that the paper and BeJLT is an open-access journal, available online to interested academics anywhere.  Second, and most importantly, is their personal willingness to roll up their sleeves and engage collaboratively to explore solutions to these global assessment challenges.  One personal example epitomises this willingness. Following the national Bradley Review (2008), the Australian Government flagged in 2009 a series of major changes to the higher education sector. To accompany a major expansion in participation they flagged a national regulator with responsibility to ensure all higher education providers meet threshold standards and invited the academic community to participate and engage in the process of determining their future.

Discipline communities will ‘own’ and take responsibility for implementing teaching and learning standards (working with professional bodies and other stakeholders where appropriate) within the academic traditions of collegiality, peer review, pre-eminence of disciplines and, importantly, academic autonomy. (DEEWR, 2009, p. 32)

The Australian Learning and Teaching Council had seconded me from the University of Sydney in 2009 as the discipline scholar for business, management and economics cluster, one of nine such clusters charged with developing a set of threshold learning outcomes for a demonstration discipline. Having read (and shared) the Assessment Manifesto article and engaged with Chris Rust and Margaret Price already, including in their tours of Australian business schools, I made sure to collaborate with a group of discipline leaders during 2010, to develop discipline standards for accounting, and structure their development around multiple forums for dialogue (Freeman and Hancock, 2011).  In the end, some 2,100 people participated nationally over four cycles of engagement, representing all but two Australian universities, and including people from 21 private and other providers, and 20 other key stakeholders including professional and peak bodies.

Following the development of discipline standards – benchmark statements in QAA speak – for accounting, as the demonstration discipline, two further activities unfolded using the same approach reflected in tenet 6. First, I worked in a similar fashion with discipline leaders to develop standards in marketing (2011-12) and economics (2012-13), and I am currently working with finance, and also tourism, hospitality and events. The second and more interesting extension was the accounting follow-on project, Achievement Matters: External Peer Review of Accounting Learning Standards.This four-year project involves using external peers to assess student learning outcomes benchmarked against the original learning standards developed in 2010. It heavily relies on the pioneering work by Chris Rust with academics from the Hospitality, Sport, Leisure and Tourism subject centre first promulgated in Rust (2009).  One crucial difference to almost all other national processes for assuring learning involving external reviewers or examiners is that reviewers engage in the multiple forums to reach consensus around the standards (see also Sadler, 2012). These forums serve a crucial professional development role around assessment literacy and sharing of good practice in assessment design as well as increasing the confidence in reviewers’ judgements around threshold standards. More detail is available in Watty et al. (2013) and at http://achievementmatters.com.au.

To conclude my reflections on Assessment Standards: A Manifesto for Change, one of the most-cited papers in BeJL&T’s first decade, I wish to note two things. First, I want to note a sense of deep appreciation, not just to Margaret Price, Berry O’Donovan, Chris Rust and Jude Carroll and others from ASke, but also to Oxford Brookes University for providing the space and encouragement for so much good work.

And second, … carpe diem. 

References

Bell A, Mladenovic R and Price, M. (2013) Students’ perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars’, Assessment and Evaluation in Higher Education, 38 (7) pp. 769-88

Bloxham, S and Price, M. (2013) External examining: fit for purpose, Studies in Higher Education, i-first DOI 10.1080/03075079.2013.823931

Freeman M and Hancock, P. (2011) A Brave New World: Australian Learning Outcomes in Accounting Education, Accounting Education: An International Journal, 20 (3) pp. 265-73

Giibs, G. (2012) Implications of ‘Dimensions of quality’ in a market environment, Higher Education Academy, [Available online http://www.heacademy.ac.uk/assets/documents/evidence_informed_practice/HEA_Dimensions_of_Quality_2.pdf]

Price, M., O’Donovan, B., Rust, C and Carroll, J. (2008) Assessment Standards: A Manifesto Change, Brookes EJournal of Learning and Teaching. 2 (3)

Rust, C. (2009) Assessment Standards: A Potential Role for Subject Networks. Journal of Hospitality, Leisure, Sport and Tourism Education. 8 (1): 124–128. [Available online http://www.heacademy.ac.uk/assets/hlst/documents/johlste/vol8no1/81Perspectives.pdf]

Sadler, R. (2012) Assuring Comparability of Achievement Standards in Higher Education: From Consensus Moderation to Calibration, Assessment in Education: Principles, Policy and Practice. 20 (1) pp. 5-19

Watty, K., Freeman, M., Howieson, B., Hancock, P., O’Connell B., De Lange, P. and Abraham, A. (2013) Social moderation, assessment and assuring standards for accounting graduates, Assessment and Evaluation in Higher Education, 39 (4) pp. 461-478

Mark Freeman

Mark Freeman has been active leading and researching learning and innovation within business higher education and professional practice for 25 years.  Currently he provides leadership across the sector as Australian Business Deans Council Scholar and within the University of Sydney Business School as Director Accreditation. His work has been recognised through multiple national awards.

Creative Commons License
This paper is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Double Blind Review
This paper has been subject to a double blind peer review by at least two reviewers. For more information about our double blind review process please visit: http://bejlt.brookes.ac.uk/about/double-blind-review/

How to cite this paper.
Posted in Academic Paper

Leave a Reply

Your email address will not be published. Required fields are marked *

*

On this page

Subscribe to BeJLT

Get email alerts when there is a new issue.
* = required field

Send this page to Kindle

your kindle user name:
(you@kindle.com, without @kindle.com)
Approved E-mail:
(Approved E-mail that kindle will accept)
Kindle base email kindle.com | free.kindle.com
(Use kindle.com to download on wispernet or wifi, use free.kindle.com for wifi only.)
using kindle.com may incur charges)