Högskolan i Skövde

Within the European Union and other jurisdictions, government agencies are required to make digital offerings like websites, apps, or documents accessible such as by following the recommendations of the Web Content Accessibility Guidelines (WCAG). Using websites of Swedish public sector organizations and automated accessibility checking tools, we both assess to which degree those websites achieve the legal requirements and which limitations the used tools have. Our results show that the median number of unique violated success criteria is only two or less for most tools, but the tools cover only one sixth of all WCAG success criteria. Tools differ in their assessments due to several reasons, such as the size and complexity but also inconsistency of WCAG and related documentation (‘techniques’), and typical software engineering problems in the corresponding implementations. To improve the coverage and quality of tools we suggest the incorporation of scientific knowledge, for example on readability testing, and the creation of a benchmark suite similar to those in related fields like PDF/A validation.

The study program 'Network and System Administration' teaches students in computer administration and networking. Most of the courses in this program had written exams on paper, but this was seen as increasingly tedious. Students were no longer used to write texts on paper, resulting in incomprehensible handwriting and a demanding grading process.

The introduction of the learning management system (LMS) 'Canvas' in 2018 provided a software suitable for electronic examinations, as it included a quiz component, allowing teachers to use both closed-ended (automatically graded, multiple choice) and open-ended questions (free-text) in an examination.

Written exams were migrated from paper to Canvas. After an initial learning-and-making-mistakes phase, feedback from teachers and students was positive. Time spent decoding handwriting was now spent constructing questions (Ćwil, 2019, Table I). Students benefit from getting old exams as examples in Canvas and then taking exams in a familiar environment.

In 2024, a central solution using a dedicated software ('Inspera') was acquired.

While both tools are suitable for electronic examinations, there are differences: Canvas is an LMS, the quiz tool is only one of several components. Inspera is purpose-made for examinations including features such as more sophisticated automatic grading and free-text answer commenting. Exam grades in Inspera get automatically transmitted to Ladok; in Canvas, this is a manual process.

For selected courses, we used Inspera instead of Canvas. Attempting to migrate from quizzes from Canvas to Inspera failed despite both tools claiming to support QTI[1], as they implement incompatible QTI versions.

Inspera does not support the role 'examiner' who reviews other teachers' grading. Adding the examiner as another teacher did not help, as one teacher cannot see another teacher's grading. The review process had to be conducted by having the examiner sit in front of the teacher's computer.

After the results were published, a student noticed they got a negative number of points for a multiple-choice question: in multiple-choice questions, points are added for each correct choice and subtracted for each incorrect choice to discourage random selection. In Inspera, the minimum must be set explicitly for every question, which is easy to miss. The automatic grading could not be re-run, so we were required to review all submissions to manually recalculate grades and push updates to Ladok.

Finally, we noticed a number of usability issues: 

when using Inspera for IT-related courses, designing questions with source code or terminal output, often shown in monospaced colorized text, for which Inspera:

• For questions with gaps to fill out, minor but valid variations to answers cannot be described using regular expressions.
• In IT-related courses, source code or terminal output is shown in monospaced text, often colorized for readability. Asked to show text verbatim, Inspera uses a monospaced font but in red, confusing students.
• When designing multiple choice questions, checkboxes are drawn empty, without indication which alternatives are checked. To see the checkmarks, one has to switch to a different editing mode.
• Inspera makes inefficient use of available space on screen. There are UI elements where text is cut off despite available space. In the exam overview, there are large empty boxes for each question.

 [1] https://www.1edtech.org/standards/qti

References

Ćwil, M. (2019). Teacher's Attitudes towards Electronic Examination – a Qualitative Perspective. International Journal of Learning and Teaching, 5(1), 77–82. https://doi.org/10.18178/ijlt.5.1.77-82

A traditional course setup consists of a number of lectures, a number of assignments for students, and a final written exam at the course s end. The lectures' purpose is to present the theoretical foundations, which then get applied in the assignments. This model is criticized as it spends valuable in-person time to teach lower order skills (Bloom et al. (1956)'s taxonomy), but higher order skills get only applied in assignments where interactive support from teachers may not be available.

A possible solution comes from the concept of blended learning (Garrison & Vaughan, 2008), where in-person teaching is combined with online resources. One implementation of blended learning is flipped classroom teaching, which describes a pedagogical concept where students prepare before a lecture the (theoretical) concepts and then convene in the classroom to apply the learned theory. Direct student-teacher interaction allows a teacher to identify problems both of individual students and the whole class and then counteract them appropriately.

Inspired by success stories of flipped classroom applications (Mason et al., 2013) and accelerated by the Corona pandemic, a course in the network and system administration program was migrated from being a traditional course to a flipped classroom course. The course consists of a series of lectures that get examined by a final written exam and two practical assignments which were not changed as part of the migration and will not be further discussed. The process of this migration is presented here.

The course's lectures are strictly based on one course book. The course is hold annually in the lecture period of October to January and is given in English, which neither is the teacher s nor the students native language. Up to Autumn 2019, the lectures were held in a traditional format with seven classroom lectures at the course s beginning to avoid overlap with the practical assignments. Based on the course feedback, students appreciated the slides quality and the entertaining parts of the lectures. Other comments remarked that lectures contained much information, which, according to one student, made students fall asleep.

Due to the Corona pandemic, the course was held via Zoom in 2020, but no lecture was recorded. Students had cameras switched off and interaction happened primarily by students asking questions via text chat which the teacher answered via voice to the whole class. Due to the unfamiliar format and occasional technical issues, pressure to get through the material in time was higher than normal. Whereas most students had no issues with online lectures, this pressure was mentioned in several comments in the course evaluation.

As it was unclear how future health restrictions would affect teaching, the decision was made to record lectures before Autumn 2021. The videos were based on updated slides with some entertaining material removed to keep the videos shorter and to avoid copyright issues. Audio recording and subtitles were both based on a shared manuscript. For each chapter from the course book one video was produced and all 'ähms' and goofs were removed.

Student feedback on the recorded lectures was very positive. One student compared the teacher's voice with Bob Ross. The recorded lectures were complemented by four classroom meetings. At the course start, students were informed that they could ask questions on the lectures' topics. The result was disappointing: very few questions were raised by students and even after discussing previous year's exams only a fraction of the allocated time was used. Accordingly, attendance dropped and generally negative feedback was given in the course evaluation. One student remarked that the lectures material is not suitable for flipped classroom teaching as it is mostly about learning facts. Several students requested the return to classical lectures.

For 2022, in order to improve classroom meetings, a generic assignment was devised that would be used in each lecture, but applied to different chapters each time. Students were asked to assemble in groups and for a chosen chapter prepare a short summary and discuss one selected keyword in detail. After 20 minutes, one member of each group would present the group's results to the audience. Several problems became apparent: first, many students had not prepared the selected chapters, i.e. neither watched the videos nor read the course book. Second, as being scheduled as a 'lecture', students were disappointed that they had to work while the teacher was 'doing nothing'. Finally, many students had stage fright and did not want to talk in front of the class. This led to a drastic drop of attendance so that only a fraction of all students attended the last lecture.

Scientific literature on flipped classroom has probably a strong 'survivor' bias documenting successful implementations in specific, suitable setups (Lundin et al., 2018). The body of literature thus may 'trick' a teacher into believing that flipped classroom is some sort of pedagogical 'silver bullet'.

Courses that are about learning facts are less suitable for flipped classroom: there are no obvious higher order assignments and without own experience, students cannot comment on the presented facts.

There are groups of students that prefer in-person lectures and there are students that prefer recorded lectures. Providing only one type of lectures will leave one group dissatisfied.

Based on our experience, we are considering the following changes for the next course iteration (Autumn 2023):

Recorded lectures will be still provided to students as the material exists and is still relevant.

The four classroom lectures will be again used for real lectures, probably some kind of best of slide selection or bonus material. Students will still have the opportunity to ask for clarification of any relevant material.

The chapter summary assignment will be refactored into a series of off-line assignments and each submission will contribute a few points to the final exam

References:

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Kratwol, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals (Vol. Handbook I: Cognitive domain). David McKay Company, New York.

Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. John Wiley & Sons. https://doi.org/10.1002/9781118269558

Lundin, M., Bergviken Rensfeldt, A., Hillman, T., Lantz-Andersson, A., & Peterson, L. (2018). Higher education dominance and siloed knowledge: A systematic review of flipped classroom research. International Journal of Educational Technology in Higher Education, 15(1), 1–30. https://doi.org/10.1186/s41239-018-0101-6

Mason, G. S., Shuman, T. R., & Cook, K. E. (2013). Comparing the effectiveness of an inverted classroom to a traditional classroom in an upper-division engineering course. IEEE Transactions on Education, 56(4), 430–435. https://doi.org/10.1109/TE.2013.2249066

The use of appropriate fonts and file formats for long-term maintenance of digital assets is a challenge for organizations in the public sector. The article reports from a study which investigated the PDF/A conformance and font usage in PDF files provided by Swedish public sector organizations (PSOs). This article presents an analysis of the PDF files’ properties and font usage including a categorization of fonts’ licenses. This study is motivated by the PDF/A-1 standard’s requirement that ‘only fonts that are legally embeddable in a file for unlimited, universal rendering shall be used.’ Analyzing PDF sets from three PSOs, the finding shows that the proportion of files that claim or succeed at conforming to PDF/A greatly varies among the sets despite similar backgrounds. Although the most popular way to make use of fonts is by embedding a subset of the font data, for some fonts expected to be ‘always available,’ a considerable proportion of PDF files does not include any font data. This puts the onus of locating this data on the PDF reader which is problematic for long-term archival

Open source software (OSS) and open standards have become increasingly important for addressing challenges related to lock-in, interoperability and long-term maintenance of systems and associated digital assets. OSS projects operate under different conditions and many projects and organisations consider successful governance and strategic involvement with projects to constitute major challenges. Today, many companies seek to establish work practices which facilitate strategic engagement with OSS projects. Based on findings from collaborative research which draws from rich insights and extensive experiences from practice, the paper presents seven actionable strategies for organisations that seek to leverage long-term involvement with OSS projects. 

In the context of long-term archival of digital assets, file formats that are standardized and designed for longevity such as PDF/A are preferred. However, due to the complexity of and ambiguities in PDF standards, it is far from trivial to either create standard-conformant files or check the conformance of any given file. This study investigates the challenges when checking real-world PDF files from public sector organizations meant for long-term archival for PDF/A conformance. Results show that only a small set of PDF files claims to conform to the PDF/A-1b specification variant and even fewer files pass conformance checks by various conformance checking tools. Challenges for conformance checking tools include both ambiguities in the standards’ technical specifications and limitations in the implementation.

File formats that are standardized and designed for longevity, such as PDF/A, are preferred in the context of long-term archival of digital assets. However, it is far from trivial to either create standard-conformant files or check the conformance of any given file due to the complexity and ambiguities in PDF standards. This study aims to investigate challenges when checking PDF/A conformance for real-world PDF files from public sector organizations that are intended for long-term-archival. Findings show that only a small proportion of the PDF files claim to conform to the PDF/A-1b specification and even fewer files pass conformance checks performed through the use of various conformance checking tools. Challenges for conformance checking tools include both ambiguities in the technical specifications of standards and limitations in the implementation of software.

In the context of long-term archival of digital assets, file formats that are standardized and designed for longevity such as PDF/A are preferred. However, due to the complexity and ambiguities in PDF standards, it is far from trivial to either create standard-conformant files or check the conformance of any given file. This study aims at investigating the challenges when checking real-world PDF files from public sector organizations meant for long-term archival for PDF/A conformance. Results show that only a small set of PDF files claims to conform to the PDF/A-1b specification variant and even fewer files pass conformance checks by various conformance checking tools. Challenges for conformance checking tools include both ambiguities in the standards’ technical specifications and limitations in the implementation.

This industry paper reports on how strategic use of open source in company contexts can provide effective support for addressing the fundamental challenges of lock-in, interoperability, and longevity of software and associated digital assets. The fundamental challenges and an overview of an ongoing collaborative research project are presented. Through a conceptual model for open source usage in company contexts we characterise how companies engage with open source and elaborate on how the fundamental challenges can be effectively addressed through open source usage in company contexts.

The publication and preservation of doctoral dissertations is fundamental to progression of research in a field. It allows future generations to 'stand on the shoulders of giants'.

Archiving dissertations requires to store them in a way that they will be accessible even when technologies change. Classical paper publications fulfill those requirements but have disadvantages such as the need for physical storage or distribution. An increasingly popular alternative is to store dissertations as electronic documents such as PDF files.

In this study, we analyze the extent to which electronically available doctoral dissertations published in Sweden fulfill the basic requirements of standard conformance and applicability for long-term archiving.

We find that only a small proportion of dissertations' electronic publications follow the Swedish National Archive's requirements to conform to standards made for long-term archiving. Only at one Swedish university, a significant number of electronic publications does conform to PDF/A-1b. Interestingly, conformance to PDF/A-1b correlates to properly setting the title of a PDF document and choosing a paper format different from A4 (ISO 216). This suggests that the consequences of missed compliance for long-term archiving are not commonly known and authors have to make a conscious choice for PDF/A-1b compliance.