“Open learning is a vision of an educational system in which education is accessible to every individual and with minimal restrictions.” (Alfonso, 2017; p. 1)
This chapter shares the experience of the Faculty of Management and Development Studies (FMDS) in developing the FMDS Online Examination System. It also tries to examine the issues and challenges on the development and implementation of the online examination system encountered by the teachers, assessment expert, developer, as well as the students enrolled in graduate online courses.
The Faculty of Management and Development Studies (FMDS), as one of the three Faculty of Studies of the University of the Philippines Open University, recognizes the importance of an assessment system that is accessible, credible, and reliable. Assessment system is one of the important topics which occupies a major place in the concerns of state and private universities due to the increasing number of admitted students (Alsamarai, et al 2014).
FMDS is a collective of various disciplines that deal with international health, nursing, environment and natural resources management, land valuation, land use and planning, research and development, social work, public management, and Association of Southeast Asian Nation (ASEAN) studies. As a Faculty with a varied and wide range of programs, the FMDS caters to students across the country and in different parts of the world. As FMDS opens its doors to serve the global community of learners, scholars, and workforce, it is also committed to adhere to the highest standards of academic excellence, service, and social responsibility. With this commitment in mind, the FMDS faculty and staff strive to explore ways to improve its instructions including its assessment system. The rapid advancement of technology in recent years has had a positive effect on the learning processes, especially the assessment component. This prompted FMDS to maximize the affordances of modern Information and Communication Technology (ICT) to help address the diverse needs of its students in terms of assessment.
According to Idemudia and colleagues (2014), “The incorporation of Information and Communication Technologies instruments in education and training has developed new paths of delivering, accessing, and the progression of helpful awareness.” At present, there are approximately 300 FMDS students based abroad who benefit from the Online Examination System (OES). The utilization of the electronic examinations system is considered a good indicator of the development of methods used to evaluate the performance of students in the universities (Alsamarai, et al 2014). Furthermore, the FMDS OES is the first Faculty-wide examination system that has been developed.
FMDS recognizes the importance of developing an online exam system (OES) that considers the varied locations of its students. It also recognizes the need to properly plan assessment practices to meet global standards. OES is now an essential tool among universities as it is more convenient to use for the students and faculty members alike (Sarrayrih, 2016). According to Ana and Bukie (2013), an online exam system is an effective way to measure the student’s knowledge.
FMDS also acknowledges many challenges of the traditional paper-and- pencil type of examination. Aside from entailing a tedious process, paper- and-pencil type of examination runs counter to the principles adhered to by UP Open University as an open and distance e-Learning (ODeL) institution. Among the challenges of a paper-and-pencil type of examination are: it is time-consuming and costly in terms of printing and reproducing exam papers; it is risky to send out to different parts of the world due to a chance of it being lost; and it is delayed in terms of sending the answered questionnaires to the checkers. Unlike computer- or web-based tests, paper-and-pencil type of examination does not have real-time updating of exam, random item selection, exam banks, and automatic scoring (Millsap, 2000) thus making feedback a lot harder and slower.
Before the establishment of the UP Open University’s examination unit in 2017, UP Open University did not have a centralized office or unit that takes care of examination arrangements. Administration of exams was usually done through the following: individual initiative of faculty-in-charge (FIC) through MyPortal, UP Open University’s learning management system (LMS), or through other online platforms; online proctoring through document sharing software or system (i.e., Skype, Google Docs and other document sharing platform); and sit-down, proctored exam in Learning Centers (LCs) and Testing Centers.
The FMDS OES was developed based on a recommendation by the Taskforce to develop assessment framework and guidelines for FMDS. The taskforce was constituted on 13 January 2015 by the Dean as part of the thrust of FMDS toward quality assurance of its program offerings. The development and implementation of the FMDS OES was part of its Diploma in Research & Development (DRDM) program.
The FMDS OES was developed by a team who worked together in collecting or gathering data related to FMDS exam administration. The assessment team consisted of the following:
a. Team Leader. The team leader was responsible for initiating the e-assessment system, coordinating efforts, and monitoring progress of the OES;
b. Assessment Expert. The assessment expert was responsible for ensuring that assessment principles are followed;
c. Technical Experts. The technical experts were responsible for developing the e-assessment system, and for giving technical feedback or comments; and
d. Faculty-in-charge (FIC). The FIC works with the learner and is responsible for pilot testing the system and gathering feedbacks.
FMDS first used the OES in 2015. It was developed primarily for the following reasons: to minimize the sending out of several paper-based exams to offshore students, thus cutting down the expenses on the part of the university; to increase the administrative efficiency since the tasks of printing, reproducing, and sending out of exam papers are no longer carried out; to decrease the workload of FICs in terms of marking exams especially for multiple-type of exam since scores are automatically computed by the system; and to reduce the risk of exam leakage.
The development of the first online examination (See Figure 1) version had the following features: a. Creating or Formulating Exam (e.g., essay, multiple choice, True or False)
- Administering exam
- Exam marking
- Feedbacking
The first version of the OES had applications — MyODBC connector and the Online Exam application–that needed to be downloaded and installed in the student’s computer (See Figure 2). These applications were needed to run the OES. This version also had the screen lock feature which disallowed students to open other computer programs and/or access other websites, thus strengthening the credibility and integrity of the online examination.
The second version of the OES, on the other hand, required students to download the Online Exam application only (See Figure 3). It used a combination of desktop and web-based applications. Face-to-face proctoring could also be used in this version.
Upon logging in in the second version of OES, the screen lock feature would automatically activate and the students could not browse other web pages (See Figure 4).
Meanwhile, the third version was purely web-based. Proctoring could also be done online via Skype (See Figure 5). This version of OES had more practical features than the first two versions.
As part of its continuing effort to improve the examination system, the FMDS re-developed the OES into an Enhanced Online Exam System (EOES) to further improve the quality of the first OES. It ensures the integrity of examination through new features such as the use of new webcam image capture and improved interface.
The development of the EOES aims to develop a more credible exam system through the use of webcam image capture to monitor the examinee. This feature captures the image of the examinee every five minutes. It also aims to increase the integrity of the OES with an improved layout and interface. The OES creates better mobility with improved scheduling to cater students from different time zones. These features of the OES were programmed to ensure reliable, valid, and fair examination, so as security and integrity.
As stated in the Educational Testing Services (2013), assessment practices must be reliable, valid, and fair. Reliability produces consistent scores among different testing conditions or versions used while validity measures the traits such as knowledge or skills which the test intended to measure. Lastly, fairness illustrates that no group of students is either advantaged or disadvantaged due to factors such as language and/or content of the questions.
Currently, FMDS is using the EOES (2016-2017). It has new system features, interface, and system functions and access. Some of the added features are the Student View, Time Zone Scheduling, Exam Schedule E-notification, and Image Capture and Psychometric Analysis.
The administrator or FIC can see the student’s view of the examination. This feature can help the administrator or FIC know whether the questions and items are properly laid out prior to final scheduling of examination. Additionally, the time zone scheduling feature eases the scheduling process since it gives a selection of different time zones, thus taking into consideration the fact that UP Open University students are geographically dispersed. Students are automatically notified of their exam schedule through Exam Schedule e-notification which serves as a reminder to students of the date and time of exam schedule. Meanwhile, the image or video capture continuously monitors the student while taking the test. The image captures the frontal view of the student to ensure that the student herself or himself is the one taking the test. IP address can also be checked.
The new system access includes the Administrator Access, Faculty Access, Proctor Access, and Student Access to ensure and uphold the security of the OES since each type of access has a certain limit of access and functions. For instance, the Administrator Access has a hold for the overall access to all features specified and the maintenance of the system (See Figure 6). The Faculty Access can only add or edit an exam, add or edit exam schedules, add or delete examinees, assign the proctor, view exam results and the captured images, and lock or unlock an exam. Furthermore, the Proctor Access can only lock or unlock an exam. Lastly, the Student Access can view exam schedules and answer the exam.
The Student View feature allows the FIC or the one encoding the exam to see how it looks from the student’s end (see Figure 7). Exam time is automatically converted based on the student’s time zone in the Time Zone Scheduling field. Manual converting or computation is not needed anymore which was also a problem in the previous versions because of wrong time conversion. In this version, the system does the computation. As seen in Figure 8, another new feature of the current version is the email notification of exam schedule. An email blast consisting of the exam schedule and some reminders will be sent to the examinees. Meanwhile, one of the new and most important features of this version is the Image Capture. When the FIC chooses the “Webcam Required” feature, the student will not be able to open the exam until the webcam is enabled from the student’s end. Also, image of the student is saved in the system every five minutes.
Despite the marked advantages of the EOES, there is still a need to improve on video capture and monitoring. The frontal view captured continuously may not be enough to maintain the integrity of the examination. Hence, it is recommended to have a wider angle of video monitoring, so it can monitor the entire screen of the computer or laptop and surroundings . It can even sense the movement of the keyboard and mouse. Additionally, continuous screenshots of the screen are imperative to monitor whether the student is navigating to other sites and documents.
The FMDS OES has been a great help in reducing cost of papers and other materials used in reproducing paper-based exams and of shipping or courier fees. The risk of exams being lost is also reduced. The efficiency of the administrative work and staff is also increased. Despite the OES being “easy to use and efficient, the success of implementation depends on the culture of the campus and using appropriate change strategies” (Baleni, 2012).
The constant development and improvement of the OES through the years is attributed to the observations provided by students, FICs, and proctors. The problems encountered by the students in the first version were as follows: failed internet connection, empty exam file, and “frozen” screen. In the second version, students tended to forget their password; some of them used their LMS password. The second version had a wider reach. More students from different parts of the world started using it. The exam areas were Chicago, Bahrain, Dubai, Riyadh, and the Philippines. Since it was already web-based, majority of the problems encountered in the second version were related to internet connection failure and server glitches (See Figure 9).
The initial implementation of the FMDS OES was time-consuming. The assessment system must be carefully and continuously monitored before, during, and after examination for further observations and improvement of the system. Students must be briefed about the system, the technical skills required on their part, and what is to be expected. Lastly, proctors should also be given an orientation about the system and how they should ensure that students are all logged in; they should also remind students to call the proctor’s attention when experiencing technical problems. Meanwhile, close coordination among members of the team (i.e., learning center coordinators, proctors, FICs, programmer, troubleshooter/ technical support) must be implemented. The members of the team must be also oriented on the importance of security measures and be familiarized with the system or software. Also, problems or system glitches encountered must be documented and reported to the programmer.
Rigor of the Development Process
It takes time, patience, and commitment in developing and refining the FMDS OES. Time is required in pilot testing both in small and large groups of students. Added to this is the challenge of diversity and the big number of students who initially took the exam through the System. Diversity refers to the context of the students’ geographical location, cultural or personal factors, and technology related matters The geographical location also connotes time differences which require monitoring beyond Philippine time. Internet connectivity was also one of the early concerns and obstacles in some local places in the Philippines. Nonetheless, the use of internet in embassies and consulate offices was one of the earliest difficult problems of students based abroad. The interplay of cultural and personal factors such as the level of acceptability of this new mode of examination was one of the difficulties encountered. Shifting to online exam from paper-and-pencil was not easy to implement. There was a resistance both from the students and FICs. Detailed instructions and technical assistance paved the way to a certain degree of acceptability of this new mode of examination system. The flaws and problems encountered related to the implementation of the OES served as the basis for refinement. The attitude of persistence and positivity helped the team to go on and further refine the OES alongside the challenges previously mentioned. The FMDS OES is a work in progress as the team continues to strive for quality and further improvement.
Commitment to Openness and Innovation
Since the students are from different parts of the globe and many are part-time students, the FMDS OES is indeed one of the concrete translations of openness as this allows one to take the examination without going to examination centers. Students can take their examination at home or even in their workplace. This lessens the expenses of both the student and the institution without compromising integrity and quality especially in some countries where safety and security in travelling to examination centers are both issues for the UP Open University students. Further, the continuous video captures and viewing of students’ answers are two of the features that uphold the integrity of the OES.
The commitment to innovation is one of the springboards of this initiative which contributes to strengthening the mission and vision of UP Open University as a leading university in online learning.
Progressiveness
The FMDS OES can be used on either proctored or non-proctored exam. Previously, paper-and-pencil test was used and this required the dispatch of the examination paper to different parts of the country and the world. Thus, this involved the following factors: manpower, money, and time. However, with the OES, these factors were greatly reduced. Examination papers are no longer needed to be reproduced and sent to different parts of the country and the world. The student needs only to log in and take the examination. Consequently, the FIC can immediately check the exam without waiting for weeks as compared to paper and pencil tests.
The examination system is just the tip of the iceberg in assessment. The FMDS OES is quite different from other exam systems since it provides psychometric analysis, both basic and advanced. This includes item analysis, item difficulty, and item discrimination. This will help the FIC analyze items in a very critical manner and later improve the identified items. The advanced method such as adaptive-based testing is one way of maintaining the integrity of the exam. With this adaptive-based testing, each student may have her or his own sets of exam according to her or his ability. Thus, credible and reliable individual outputs can be produced. The development of an e-assessment system has proven to be a challenge. Such system must undergo intensive pilot testing and the examination carefully monitored to avoid problems. Students must be technologically adept to avoid problems, speculations, and fear of system malfunctioning during an exam. Also, invigilators need proper training. High level of coordination among team members is also required. Lastly, assessment type is limited but can be expanded to wider choices as the system is improved
It is highly recommended that a user manual be developed. The contents of which should include coordinating e-examinations; preparing learners to use the online exam system; administering online exams; providing technical support or troubleshooting for the exam system and guidelines for invigilators; and more importantly, the system must be continuously improved.
Alsamarai, S., Amawi, A., & Eljini, MAH. (2014). E-assessment systems: A comparison of the views of samples of students and professors. CSCanada 8(3). Retrieved from www.cscanada.net/index.php/mse/article/download/5566/6365
Ana, P. & Bukie, P.T. (2013). Design and Implementation of Online Examination Administration System for Universities. African Journals Online, 12(1). Retrieved from ttps://search.proquest.com/docview/1553422707/fulltextPDF/B6D69A76E0A9450BPQ/1?accountid=47253
Baleni, Z.G. (2013). Why and how academic staff adopt e-assessment in a Higher Education Institution (HEI). Beldhnis, H. (Ed.). ECEL 2012 The Proceedings of the 11th European Conference on e-learning p. 28. Netherlands: University of Groningen.
Millsap, C.M., (2000). Comparison of Computer Testing versus Paper and Pencil Testing Doctoral dissertation). Retrieved from ProQuest. (3044291)
Sarrayrih, M.A. 2016. Implementation and Security Development Online Exam, Performances and Problems for Online University Exam. International Journal of Computer Science and Information Security (IJCSIS), 14(1). Retrieved from https://www.researchgate.net/publication/296675368_Implementation_and_Security_ Development_Online_Exam_Performances_and_problems_for_ Online_University_Exam
Serrano, J. V. , Ramos, R. C. , and Cabrera C. A. G. (2018). Development of an Online Exam System: The FMDS-UPOU Experience. In M. F. Lumanta, & L. C. Carascal (Eds.), Assessment Praxis in Open and Distance e-Learning: Thoughts and Practices in UPOU (pp. 165-178). Los Baños, Laguna, Philippines: UP Open University
