This paper discusses the findings of a pilot study that set out to explore learners’ initial expectations and experiences of using ePortfolios (electronic portfolios). The popularity of ePortfolios is growing rapidly in tertiary education across all subject areas (Stefani et al., 2007; Strivens, 2007). However, early studies have revealed that learner reaction to this tool is mixed, especially when the ePortfolio is used to support personal development planning (PDP). If we are to implement ePortfolios across an institution, with appropriate levels of resourcing, then we need to know much more about learner experiences with ePortfolios and how best we can support their use.
Background to ePortfolios
ePortfolios have the potential to promote learning and encourage personal development by supporting
- the learning process
- the product of learning
- the transition of learners at various stages of their lifelong and life-wide journey (Barrett and Carney, 2005; ISLE, 2005; JISCinfoNet, 2008; Ward and Grant, 2007).
An ePortfolio is a web-based system that facilitates the development, collection and organisation of digital resources or artefacts, such as blogs, photographs and multimedia (Oradini and Saunders, 2007). These resources are likely to be drawn from a range of learner experiences over a period of time and will include those from both formal and non-formal learning opportunities (Beetham, 2005; Funk, 2004; Siemens, 2004).
Learner engagement with ePortfolios
Early studies indicate that learner response to ePortfolios, particularly in the first year of study in Higher Education (HE), is varied, with a notable lack of student engagement and ownership (Cosh, 2007; Oradini and Saunders, 2007; Peacock and Gordon, 2007; Pond, 2007; Tosh et al., 2005). This lack of student engagement may reflect tutor and student confusion about the roles and purpose of the ePortfolio in the learning environment. It could also be linked to the new skills and knowledge that are required by both tutors and learners. For example, tutors will need to explore how and in what ways to integrate PDP supported by ePortfolios in their programmes. Learners too will need to acquire new skills, for example, developing their online professional digital identity (Cambridge, 2008).
On the other hand, this lack of engagement could be linked to students’ high expectations of technology in their learning: the Net Generation is extremely discriminating and will not tolerate poor implementation of any Web 2.0 tool (Bakker et al., 2007; ECAR, 2007; Ipsos MORI, 2007). An additional consideration is that, until recently, much research has focussed on the learner response to e-learning tools, such as Virtual Learning Environments (VLE), which are often used for the e-administration of learning. Such tools are controlled and managed by tutors whilst ePortfolios, on the other hand, are controlled and managed by learners and are much more personal.
Other barriers to learner engagement could reflect poor system usability and accessibility. As implementation of ePortfolios progresses in the tertiary environment, students will be expected to learn how to use an ePortfolio system quickly in order to perform a wide range of tasks—an ePortfolio interface must be intuitive and easy to use regardless of the users’ level of experience of Web 2.0 tools. Usability is concerned with evaluating whether a system is usable and fit for purpose, and principles to consider when assessing usability of a product include learnability, flexibility, robustness, throughput, user satisfaction, efficiency, and memorability (Faulkner, 1998; Nielsen, 1993; Preece et al., 2002; Shneiderman and Plaisant, 2005). Accessibility focuses on assisting users to perceive, understand, navigate, interact, and contribute when encountering Web 2.0 systems (W3C, 1994–2006). However, access to learning technology is important for all learners regardless of ability and characteristics. Unlike e-learning in general, there is very limited research and guidance in this area for ePortfolios (TechDis, n.d.).
Further barriers to learner engagement with ePortfolios could mirror those identified by research into other e-learning tools, such as online assessment. For example, a study by Rosen and Maguire (1990) estimated that more than 50% of students in higher education were computer anxious; current research indicates that this remains an issue and has even been exacerbated by the widespread use of computers in many degree programmes (Mcilroy et al., 2007). Whilst it is often assumed that most of our students are ‘digital natives’—technologically capable and confident—the level of discomfort encountered by students using technology has often been underestimated (Brosman, 1999; Currant et al., 2008).
It appears that we know little about the learners’ perspective of using ePortfolios. If we are to engage learners to make full use of ePortfolios from the beginning of their studies, we need to know far more about their experiences and expectations of this tool. The aim of this study was to begin to explore learners’ expectations and experiences of using ePortfolios with the intention of gaining a rich picture of how the ePortfolio is used by these learners. From this it was hoped that a deeper understanding of learners’ needs could be gained and how student engagement could be facilitated. In particular this study aimed to ascertain how usable and accessible learners found the ePortfolio to be.
ePortfolios at Queen Margaret University
PebblePad, a commercial software application used by many Higher Education institutions and by some professional bodies, was introduced as the institutional ePortfolio at QMU in 2005. Similar to other ePortfolio systems, this has tools such as web-based portfolios (webfolios), online journals (blogs), competency checkers, online CVs and forms, and activity logs (see Figure 1). The application also facilitates sharing of assets to selected audiences, as well as publishing to the Internet.
Figure 1: the PebblePad ePortfolio. (© 2008, Pebble Learning Ltd. Used with permission.).
The study followed a mixed-method approach and used elements of ‘naturalistic’ inquiry (Denzin and Lincoln, 2005). Once ethical approval had been gained from the institution’s Research Ethics committee, participants were sought from four subject areas: Radiography, Inter-professional Health Education, MSc Professional Education, and MSc Physiotherapy, where the ePortfolio is used for a wide range of purposes within the learning environment at QMU. For example, Radiography students are required to develop reflective webfolios to demonstrate their learning and personal and professional development in a placement setting, whilst learners undertaking Inter-professional Health Education are required to keep an online journal (blog), to reflect on their experiences of problem solving and group processes. In some instances tutors provide a template to assist learners in visualising the required structure for the webfolio.
Data were collected through interview, observation and by questionnaire. Tables 1 and 2 illustrate the participant sample and data collection methods.
Table 1: participant sample (qualitative methods)
|Academic year||1||1||1||Post-grad level 1||1||4||4||4|
|Subject area||IPHE (OT)||IPHE (N)||IPHE (R)||D||IPHE (D)||R||R||R|
|Data collection method||Interview (x2)Cooperative evaluation (x2)||Interview (x1)Cooperative evaluation (x1)
Email interview (x1)
|Interview (x2)Cooperative evaluation (x2)||Interview (x1)||Interview (x1)Cooperative evaluation (x1)
Email interview (x1)
|Group interview (x1)Email interview (x1)||Group interview (x1)||Group interview (x1)|
Note: P = participant. GP = group participant. IPHE = Inter-professional Health Education. OT = occupational therapy. N = nursing. R = radiography. D = dietetics.
Table 2: participant sample (questionnaire)
|1||Female||37||Professional Education||2||Webfolio; forms; gateway|
|5||Female||34||Radiography||3||Webfolio; gateway; blog|
|6||Female||37||Professional Education||1||Webfolio; gateway|
|8||Female||37||Professional Education||1||Experience; webfolio; gateway|
|9||Female||42||Professional Education||1||Achievement; experience; meeting; blog; webfolio; forms; profile tool; CV; gateway.|
|10||Female||22||Radiography||4||Experience; webfolio; blog; CV; gateway.|
|11||Female||44||Radiography||4||Achievement; webfolio; blog.|
|14||Female||29||Professional Education||2||Action plan; webfolio; blog; gateway.|
|15||Female||45||Professional Education||1||Action plan; experience; webfolio; gateway.|
|16||Female||48||Professional Education||1||Webfolio; gateway.|
Face-to-face, semi-structured Interviews were conducted with learners at an early stage, shortly after introduction to the ePortfolio, and again at a later stage, once learners had been using the tool for a period of time. Interviews were conducted in order to gain access to personal accounts of actual experience and through this it was hoped to gain a deeper understanding of the main issues (see Table 3). Conducting interviews in two phases allowed comparisons to be made between data gathered, enabling a more complete picture of expectation and experience to emerge. Three participants were unable to take part in the second interview but were able to respond to questions via email.
Table 3: main areas of interview questioning
|Stage 1: initial expectations|
|Stage 2: actual experience|
Interview transcripts were analysed using NUD.IST (Non-numerical Unstructured Data Indexing Searching and Theorizing), where data were coded, categorised and studied for emerging themes. Data gathered at early stages were compared with data gathered at later stages to assess throughput—the ease and efficiency of ePortfolio use once the student has become familiar with the tool.
Cooperative evaluation observations
This is a technique used in the field of Human-Computer Interaction to obtain detailed feedback from users and is particularly useful in identifying usability issues (Monk et al., 1993). These observations were conducted in conjunction with the interviews and involved the researcher in sitting alongside the learner at the computer, observing their interactions with the eportfolio and questioning as required. During this investigation learners were asked to work through a task and were encouraged to ‘think aloud’. It was felt that observing use of the ePortfolio in this way would facilitate a better understanding of learners’ abilities in using an ePortfolio, their methods of engagement with the tool and would highlight any difficulties in using the tool. Evaluations were audio recorded with participants’ permission and supplemented with field notes. Transcripts were analysed with the aid of NUD.IST and studied for emerging themes.
A questionnaire was piloted on paper and then developed using Survey Monkey, an online survey tool. This was used to gather both quantitative data (relating to age, gender, course, and technical ability) and qualitative data, which explored:
- attitudes towards the use of an ePortfolio;
- perception of impact on the learning experience;
- user satisfaction and ease of use.
To improve rigour, the researchers analysed the data gathered via these various methods, separately and together, to crosscheck the analysis. A table of key themes was developed to bring together data from the different methods of investigation.
Several themes emerged:
Understanding and expectations
Learners’ understanding and expectations regarding ePortfolios were mixed. Some learners were unsure of the role of ePortfolios, while others perceived an ePortfolio to be a memory aid similar to an online diary. For example, one participant reported: ‘I don’t really know what to expect…just generally somewhere to write down what happened, bit like a diary really, what I would normally write down on paper’ (interview participant 2, phase 1).
Both interview and questionnaire participants acknowledged the tool could be useful for storing resources, recording evidence of learning, presenting a collection of work, and assisting with personal development. For example, one questionnaire respondent perceived it to be: ‘An online resource for storage of information relating to my course of study and career’ (respondent 12).
Reasons for use
Questionnaire respondents reported that their main reasons for using the ePortfolio were due to a requirement of their programme of studies and related to a summative assessment process: for example, ‘we were required to use it as part of our course’ (respondent 3) and ‘…required for the summative assessment’ (respondent 8).
Other reasons for using ePortfolios identified by interview participants and questionnaire respondents included:
- keeping a diary of experience whilst on placement;
- a place for reflecting on placement experiences;
- somewhere to release personal stress;
- a resource for continuing professional development.
One participant reported using it to ‘…do action plans to help with a couple of assignments’ (interview participant 2, phase 2). However, one learner specifically commented that she would only use the ePortfolio if it was going to be useful for her to do so.
Usability and accessibility
Here opinions were mixed; some learners found the ePortfolio tool intuitive and very easy to use, while others found it challenging and time-consuming to learn to use. One student commented that it was: ‘easy to get into…you can go in and change things…it’s all in one place and it’s interactive’ (Interview participant 1, phase 1). However a questionnaire respondent reported:
‘I found it difficult to use and would have preferred not to have had to learn to use it in my final year…I didn’t find using…intuitive, although it wasn’t very difficult to use… I think the problems I encountered were probably more due to the fact I received very little explanation and training in using it’ (respondent 10).
No negative issues regarding accessing the ePortfolio were raised during interview and only three questionnaire respondents mentioned difficulty with access to the ePortfolio outside the institution. For example, one learner reported:
‘Unfortunately I did not have access to a computer at home…I am sure there are some other students who are faced with a less than up-to-date system at home or one that struggles with new applications, this adds to the stress of trying to use the ePortfolio’ (respondent 6).
Several concerns were raised by learners, including the following:
- privacy issues;
- sharing with others by mistake;
- continued access to the tool following graduation;
- time, especially time required to learn how to use the tool and then remembering how to use the tool;
- obscure terminology used in the program.
Regarding the concern of time needed to learn the system, one questionnaire respondent remarked: ‘The ePortfolio has massive potential but some students need much more time to become comfortable with the system…I also feel that if I do not continue to use the system I will quickly forget how it works’ (respondent 6).
Improving learner engagement
Suggestions were made by participants regarding how ePortfolios could be implemented in the future to increase learner engagement:
- more encouragement from tutors;
- more practical ‘hands-on’ sessions ;
- time specifically set aside for exploring the tool;
- raised awareness of the tool across the institution;
- more explanation about the benefits for learners.
In particular, one learner suggested:
‘I think that any tuition MUST be timed so that it occurs close to the point that all students are really ready to use it on a daily basis – otherwise – they do not use it for a few weeks – AND ALL IS LOST. And it has to be re-introduced – all over again, wasting the student/tutor time’ (questionnaire respondent 8 – emphasis is respondent’s).
Discussion and recommendations
These early findings provide us with some initial insights into learner experiences of ePortfolios. Learners were, in the main, quite positive about the tool and found it relatively easy to use and access—there were few examples of computer anxiety. However, learners wanted support and time to learn how to use the tool and to explore how it related to their studies. Many of these results reflect much that we would expect when implementing any type of e-learning. For example, tutors need to explain the purpose, role and benefit of the e-tool to learners. This will require tutors to demonstrate how the tool relates to their students’ learning in their specific subject area (Alexander and Boud, 2002). Students will also need protected time to become familiar with the tool, especially those who are not comfortable with technology in the learning environment.
However, there are three findings that are specific to ePortfolios and which will impact on how institutions implement ePortfolios, and how they resource that implementation:
The complexity of the tool
Unlike many other e-learning tools, the ePortfolio has the potential to support multiple roles in the learning environment. It is also a very personal and private tool, managed by learners, not tutors. As a consequence, tutors and learners often start by using the tool in a piecemeal way, for example, focusing on assessment or transition and then expanding their use of the ePortfolio. This mirrors the way in which tutors have used VLEs within the curriculum, focussing on one tool of the VLE and then gradually expanding this (Mayes, 2002). Unfortunately with ePortfolios, this piecemeal implementation may have a detrimental effect since the ePortfolio is only used by learners as a tool to fulfil a specific module/programme requirement, such as an assessment. As a result, the learner does not see the potential for using an ePortfolio to support personal development planning, lifelong learning and employability. Consequently, guidance, support and training are required for tutors to grasp the many opportunities available through an ePortfolio and to implement them in an holistic way.
Data protection and ePortfolios
Learners were particularly anxious that the private materials they were storing within their ePortfolios could not be accessed by others, and that only materials that they had selected to share could be accessed by specified individuals. To use this tool as a private, personal development tool, learners needed assurance that their materials would not and could not be accessed by others, and that they could not accidentally share their materials with others. For institutions this may require a review on procedures regarding data protection and ePortfolio, which then needs to be communicated across the institution. Guidance on data protection and ePortfolio is available through the JISC (see Charlesworth and Home, 2005).
Post-graduation (alumni) access to the ePortfolio
Having committed resources to using the ePortfolio whilst studying, learners wanted to continue access after graduation. Since interoperability between ePortfolio systems is still someway in the future (JISC, 2007), institutions should assume that learners will require long-term access post-graduation. This will have both resource and procedural implications for an institution.
It is worthwhile noting that few learners raised issues related to accessibility and usability. Nevertheless institutions need to continue to work with developers of ePortfolio systems to ensure that they are accessible and usable for all learners.
Limitations of the research
Throughout the data collection we have sought to demonstrate that our methods were reproducible, reliable and consistent. This is the first stage in a three-year research project and it is acknowledged that our research is somewhat limited, especially by the small number of participants, and that our participants are using only one ePortfolio program (PebblePad)—although the ways in which this is being used varies across subjects. We seek to expand the research in the next stage, to develop a richer picture of learner engagement with ePortfolios, by liaising with tutors and learners at an earlier stage of the research and to provide incentives to encourage learner participation.
In this initial study, many of the learners were positive about their experience with ePortfolios. However, a number of issues were raised that could have a long-term impact and subsequently need to be addressed prior to implementation. These include:
- substantial staff development, which will be required to support tutors when implementing an ePortfolio (module and eventually programme-wide) to ensure learner engagement;
- protected time within the curriculum will be needed for tutors to explain the use and purpose of this complex tool to learners and to support this on a continued basis throughout a programme;
- sufficient hardware will be required to ensure alumni access and institutional processes should be reviewed to ensure adherence to data protection policies for ePortfolios.
It is also suggested that further research is undertaken into accessibility, usability, computer anxiety and ePortfolios.
This initial study is part of the fourth cohort for Inter/National Coalition for Electronic Portfolio research. The authors would like to thank the Cohort for its support and also the learners who kindly participated in this research.
Susi Peacock and Sue Murray work in the Centre for Academic Practice, Queen Margaret University. Susi is a senior lecturer in e-learning and leads the institution’s implementation of e-learning. She has presented and published on staff development and the student perspective on e-learning. Sue Murray is a research assistant and her doctorate explored the use of a web-based timetable developed for children with high-functioning autism. Her research interests focus on information management and the usability of web-based resources.
Centre for Academic Practice, Queen Margaret University, Edinburgh, EH21 6UU
Phone: 0131 474 0000
Dr. Sue Murray
Centre for Academic Practice, Queen Margaret University, Edinburgh, EH21 6UU
Phone: 0131 474 0000
Alexander, S. and Boud, D. (2002), ‘Learners still learn from experience when online’, in J. Stephenson, (Ed.) Teaching and learning online: pedagogies for new technologies. London: Kogan Page, pp. 3–15.
Bakker, G., Sloep, P. and Jochems, W. (2007), ‘Students and instant messaging: a survey of current use and demands for higher education’, ALT-J, vol.15, no. 2, pp. 143–53.
Barrett, H. and Carney, J. (2005), Conflicting paradigms and competing purposes in electronic portfolio development. Retrieved on 12 November 2007 from: http://electronicportfolios.org/portfolios/LEAJournal-BarrettCarney.pdf.
Beetham, H. (2005), e-Portfolios in post-16 learning in the UK: developments, issues and opportunities, JISC e-learning and pedagogy strand of the JISC elearning programme. [Online]. Retrieved on 27 November 2007 from: http://www.jisc.ac.uk/uploaded_documents/eportfolio_ped.doc.
Brosman, M. (1999), ‘Computer anxiety in students: should computer-based assessment be used at all?;, in S. Brown, J. Bull and P. Race (Eds.) Computer-assisted assessment in higher education. London: Kogan Page, pp. 47–54.
Cambridge, D. (2008), ‘Audience, integrity, and the living document: eFolio Minnesota and lifelong and lifewide learning with ePortfolio’, Computers & Education vol. 51, pp. 1227–46.
Charlesworth, A. and Home, A. (2005), Data Protection, Lifelong Learner Record Systems & ePortfolios: A Short FAQ. Retrieved on 26 August 2008 from: http://www.jisc.ac.uk/media/documents/programmes/buildmlehefe/data_protection_faq.pdf.
Cosh, J. (2007), ‘PDP in the faculty of art, law and social sciences at Anglia Ruskin University’, PDP-UK Newsletter, Issue 10, The Centre for Recording Achievement.
Currant, N., Currant, B., Whitfield, R., and Hartley, P. (2008), Defining ‘Generation Y’: towards a new typology of digital learners. Retrieved on 21 November 2008 from: http://www.elp.ac.uk/downloads/Defining%20Generation%20Y%20Bradford.pdf.
Denzin, N. K., and Lincoln, Y. S. (Eds.) (2005), The Sage handbook of qualitative research. 3rd ed. London: Sage.
Educause Centre for Applied Research (ECAR). (2007), The ECAR study of undergraduate students and information technology, 2007. Retrieved on 6 December 2007 from: http://connect.educause.edu/library/abstract/TheECARStudyofUnderg/45075.
Faulkner, C. (1998), The essence of human-computer interaction. Harlow, Essex: Prentice Hall.
Funk, K. (2004), ‘Brief or new: student learning portfolios: balancing tradition with innovation’, Occupational Therapy in Health Care vol. 18, no. 1/2, pp. 99–105.
Ipsos MORI (2007), ‘Student expectations study. Key findings from online research and discussion evenings held in June 2007 for the Joint Information Systems Committee’, Retrieved on 6 December 2007 from: http://www.jisc.ac.uk/media/documents/publications/studentexpectations.pdf?.
ISLE (2005), Individualised support for learning through ePortfolios. Retrieved on 10 March 2008 from: http://www.jisc.ac.uk/media/documents/programmes/elearningsfc/sfcbookletisle.pdf.
JISC (2007), ‘JISC Centre for Educational Technology Interoperability Standards’, Retrieved on 26 August 2008 from: http://jisc/whatwedo/services/jisccetis.aspx.
JISCinfonet (2008), ePortfolio infoKit. Retrieved on 18July 2008 from: http://www.jiscinfonet.ac.uk/InfoKits/effective-use-of-VLEs/e-portfolios.
Mayes, T. (2002), ‘Learning technology and learning relationships’, in J. Stephenson (Ed.) Teaching and learning online: pedagogies for new technologies. London: Kogan Page, pp. 16–26.
Mcilroy, D., Sadler, C. and Boojawon, N. (2007), ‘Computer phobia and computer self-efficacy: their association with undergraduates’ use of university computer facilities’, Computers in Human Behavior vol. 23, pp. 1285–99.
Monk, A., Wright, P., Haber, J. and Davenport, L. (1993), Improving your human-computer interface: a practical technique. Hemel Hempstead: Prentice Hall.
Nielsen, J. (1993), Usability engineering. London: Academic Press.
Oradini, F. and Saunders, G. (2007), Introducing e-portfolios across a paper dominated university. Retrieved on 18 July 2008 from: http://newsletter.alt.ac.uk/e_article000925026.cfm?x=bbrBpT6,b1Sq4BSv,w.
Peacock, S. and Gordon, L. (2007), Final report for ISLE Project, section 5.1: Identification of staff training needs and development of staff programme. Edinburgh: Queen Margaret University.
Pond, J. (2007), ePortfolio introduction for the SABE students in levels 4 & 6 as part of the professional institutional requirement of CPD. Retrieved on 11 December 2007 from: http://www.recordingachievement.org/downloads/20070316JPondWmin.pdf.
Preece, J., Rogers, Y. and Sharp, H. (2002), Interaction design: beyond human-computer interaction. New York: John Wiley.
Rosen, L. and Maguire, P. (1990), ‘Myths and realities of computerphobia: a meta analysis’, Anxiety Research vol. 3, pp. 175–91.
Shneiderman, B. and Plaisant, C. (2005), Designing the user interface: strategies for effective human-computer interaction. 4th ed. Boston: Pearson Education.
Siemens, G. (2004), ePortfolios. Retrieved on 9 November 2007 from: http://www.elearnspace.org/Articles/eportfolios.htm.
Stefani, L., Mason, R. and Pegler, C. (2007), The educational potential of e-portfolios. Routledge: London.
Strivens, J. (2007), A survey of ePdp and ePortfolio practice in UK higher education. HEA. Retrieved on 12 November 2007 from: http://www.recordingachievement.org/downloads/KD-HEAeportfoliosurvey.pdf.
TechDis (n.d.), e-Portfolios. Retrieved on 12 March 2008 from: http://www.techdis.ac.uk/index.php?p=9_13.
Tosh, D., Light, T., Fleming, K. and Haywood, J. (2005), ‘Engagement with electronic portfolios: challenges from the student perspective’, Canadian Journal of Learning and Technology vol. 31, no. 3. Retrieved on 18 July 2008 from: http://www.cjlt.ca/content/vol31.3/tosh.html.
Ward, R. and Grant, S. (2007), What is an ePortfolio? Centre for Recording Achievement. Retrieved on 12 November 2007 from: http://www.recordingachievement.org/eportfolios/keydocs.asp.
W3C (1994–2006), Introduction to web accessibility. Retrieved on 10 March 2008 from: http://www.w3.org/WAI/intro/accessibility.php.