Teachers' needs were collected in April-May 2020 with a process like this
- teachers responded to a questionnaire and participated in one of two online meetings. Additionally, teachers have been able to write Github issues, though the opportunity has not been widely marketed yet, and the Github issues have not been included in this first round of prioritizing.
- The EDIT team read through all the ideas, commented on them and estimated their workloads and the knowledge required for the implementation
- Teachers were able to rank the ideas sorted in two lists (from 1 to 5, max 5 votes per list):
- 1) quite easy and quite minor tasks, possible to give e.g. for summer interns
- 2) large (months) tasks and tasks requiring expert knowledge
- Some results of the ranking:
- 9 teachers participated in ranking the ideas
- Each idea received at least 1 ranked vote
- Most popular idea(s) received rankings from 5/9 teachers (list 1) and even 6/9 teachers (list 2)
- In list 1, numbers of rankings per idea were distributed quite evenly: over half of the ideas received 4-5 votes. Top ideas in list 2 were more clearly separated.
- Based on the results, we have made an action plan for the summer 2020, and suggested some topics for the academic year 2020-2021 in the table below.
- Github issues will be created on each item (if one does not exist already), and the links to issues will be added in the table below.
- The issues scheduled for summer 2020 will mainly be allocated to summer interns. The issues suggested for the academic year 2020-2021 cannot be allocated yet.
- There are still some items to be clarified with teachers and the EDIT steering group. Some items are not possible to be implemented in near future.
Teachers' ideas and their implementation evaluation
Suggested categories for
- required knowledge: beginner, easy, moderate, expert (referring to who we can estimate to have the required skills; in practice, can we give it to summer interns)
- workload estimate: days, weeks, months
|Category||Nr||(Teacher) Idea||EDIT suggestion||workload estimate||required knowledge estimate||Actions and schedule||Preliminary allocation|
|student enrollment||1||(Mario) automatic enrollment based on oodi (→ sisu), like in MC|
should be done Spring 2021 to Sisu - we'll wait for Sisu.
(to note: we need an A+-Sisu-interface in general, for probably multiple needs, and the develpment needs to be started in early autumn)
GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/550
|2 weeks||moderate||Needed during academic year 2020-2021|
|material production||2||(Mario) enrollment questionnaire: dynamic questionnaire difficult to edit|
In order to make more convenient for the teacher and "a real feature" instead of a hack, we should create a new questionnaire engine and define new RST syntax/options for configuring more complex, dynamic behaviour.
GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/551
|months (for a new questionnaire service that replaces all old questionnaire implementations)||expert||voted low in priority; to be saved in the barrel of wishes|
|3||(Lauri) compilation of RST takes too much time|
option A: implement some kind of build caching to a-plus-rst-tools so that rst-tools do not require that all RST files are always built again every time. Currently, all RST files are built so that updates to the linked exercise config.yaml files are not lost. Currently, the built-in cache of Sphinx does not recognize YAML files and it would not include them in a new build if the RST file (that links to the YAML file) has no changes and is therefore not included in the new build.
option B: rewrite a-plus-rst-tools and design all features so that they align better with the core functionality of Sphinx. Perhaps, we could easily use the Sphinx build caching for YAML files if we had a separate custom YAML builder instead of the current hack that combines YAML files into the HTML builder. Additionally, the current implementation does not support the Sphinx Latex builder since HTML output has been added to components that are supposed to be builder-agnostic and work with all output formats. Basically, a-plus-rst-tools violate many Sphinx APIs, but it could be implemented in "the right way".
option C: we could recommend teachers some additional tools & instructions which would solve at least some problems. VS Code is potentially one of the most suitable editors which can compile individual files as fast as possible.
option D: Quick hack (immediate solution): remove "make touchrst" from the docker-compile.sh script. Compiling will be then as fast as it can be in Sphinx since it uses the build cache. Exercises may be broken since YAML files may be ignored in the build, but that does not matter while writing the reading materials. Docker-compile.sh only affects local testing and not the production server.
GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/561 (closed) →
|weeks or even months||expert|
Summer 2020: Instructions on D, recommend C in the instructions
Later, if needed, i.e. based on additional teacher feedback: Alternative B (should be done "at some point")
D and C: Jhosimar and Markku
|4||(Lauri) RST is clumsy as source language|
option A: use (pilot) LaRST (Latex-to-RST), which was implemented by Tomi Janhunen at Tampere University. It compiles Latex into the RST format used by a-plus-rst-tools. The special A+ features, such as exercises, are defined with custom Latex macros/environments. Tomi has not yet released the project openly, but the project should be close to finished. One should consider if LaRST supports enough A+ features to be useful or is something crucial still missing.
NOTE: if you are interested in piloting, please contact Tomi!
option B: Jhosimar is writing new instructions in the aplus-manual that could ease the pain of writing RST a little bit. There are some hotkeys or macros in the VS code editor that could help.
GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/562
summer 2020: B
later = when Tomi is ready to publish: A
|Markku (A), Jhosimar (B, A)|
|5||(Lauri) opportunity to set model solution publication timestamp (instead of automatic publication after DL)|
It is easy to add a new date setting and disallow access to the model solutions before that date. → students do not see the model solution before they submit themselves. This concerns early and late submissions. Of course, other students can share the model solution they see for their friends submitting late.
(If it is meaningful to implement this - We should consider what features are really needed before filling the system with custom date settings and switches for every small detail.)
GITHUB ISSUE: https://github.com/apluslms/mooc-grader/issues/73
Is it meaningful to add this feature; need to check with Lauri.
if yes: summer 2020
|6||(Sanna) easier way to add links to the left hand side menu|
This needs to be clarified with Sanna
There is already a menu for adding those links in A+ (under Edit course). It is unclear how it could be any easier than that. There are different types of links (LTI services, links to external web pages, links to pages inside the A+ course). Sanna should clarify what kind of difficulties she has had and in which use cases. (Admins must enable new LTI services once in the system before they can be used, which is maybe confusing for teachers. However, it is very rare to add new LTI services.)
Furthermore, menu links require domain to be registerd (even for non LTI services), so A+ knows metadata of the service. Logic: links in the menu are peers to links within application (A+), thus external links should tell the user they are leaving the site. This should happen to all A+ links (event those in chapter), but latter is not implemented. It could be possible to add possibility to add link to any site, but that should show the max level warning, as A+ can't tell the user that there is nothing to worry about.
In addition, it would be possible to add links to the course configuration (and that should be done at some point), which would then be used to configure the menu. Above point about allowing any links might be relevant in this case (or we can show an error in configuration step).
GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/563
|? (clarify usecase first)||available already?|
|7||(Tomi) need for personalized exercises (emphasized in the remote study situation)|
The need is to provide different exercises for each student, particularly in online exams. Ways of implementing this can include
For example, if the student should transform a logical formula, each student should receive their own formula (depending on the focus of the randomization needs, e.g. STACK might already provide some solutions). The randomization could be e.g. generated synchronously by using the student ID (user ID) as the random seed. Tomi did not want to try the existing personalized exercises feature of the MOOC-Grader because it is clunky and requires that the server admin runs the exercise instance generator in the server before the exercises are published to the students. Tomi is more interested in synchronous "on-the-fly" generation, but this requires that even opening the exercise description includes some custom, exercise-specific computation in the MOOC-Grader server. This affects the server's performance negatively and it is a challenge to execute the teacher's custom code safely at the time exercise description is loaded. Launching a container for safe code execution at that time might be way too slow. This is, per request it would not be meaningful → serverless as alternative: As a long term solution, we could use serverless code for form generation (see grader v2 ideas). In addition, the A+ protocol should be updated to support initial seed, which would be stored in A+ and returned to grader later on. The seed part can be hacked using signed data in the form (e.g. JWT). Though, every time server secret changes, the old forms can not be submitted or regraded.
To note: teachers have very different kinds of ideas what personalisation really means, so there are probably other use cases for personalized exercises.
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/552
|months (or 2 weeks for a simple, insecure, hacky implementation)||expert (or moderate for simple, insecure solutions)||It would be good to find a teacher who needs the feature, and ask their course assistant to implement the exercise → To be documented in the github issue.||Course assistant? Discussion with Jaakko|
The URLs to A+ course materials should include the language code (en/fi) so that people can make links to the specific language version of the content. Django has a framework for this need, but adding it to the old code will cause some surprises that have to be solved.
Google indexing is also currently broken because the same URL is used for basically different pages; the English and Finnish versions are completely separate from each other from Google's perspective.
The google indexing could be fixed by building the site index with language keys for goole and by adding support for `?lang=fi` in the urls.
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/520
Plan A: summer 2020
Plan B: postponed to later
summer intern: Veera
|material availability/access||9||(Mario and Tomi) possibility to set the material accessibility based on needs (public, only to enrolled students, individual students), like in MC|
Whole courses can currently be restricted: public, authenticated user, only internal or external user, only enrolled used.
Individual chapters can also be restricted to only internal or external users.
There is currently no support for student-specific access restrictions. A good implementation for that requires consideration of caching (for performance) and configuration (how are the chosen students defined). Is it enough to have student restrictions for whole modules? Or should it be specific to chapters? Note, usertags could be used for selecting show/hide groups.
Needs to be clarified, what the teachers actually would need in addition to what already exists.
summer 2020: add to instructions.
To be clarified, if something else needs to be done.
Instructions on the existing alternatives: Jhosimar
if need to impelement: summer intern?
|10||(Jan-Mikael) conditional exercises (student must compelete first exercise in order to see the next one), like in MC|
The need is to restrict access to exercises; for example,
All students do the same exercises. The idea is to restrict the order in which students can take the exercises.
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/553
|weeks/months||moderate/expert||At the earliest academic year 2020-2021, maybe 2021-2022||?|
|11||(Otto) it should be possible to select what is and what is not indexed|
Google can only index public course contents. If you want to hide some of the public content from Google indexing, then that would be a new feature. This should maybe be configured in the web server (robots.txt?) and not really in Django, but then how can we do it dynamically based on course configurations?
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/520
|days||moderate||summer 2020||summer intern: Veera|
|12||(Otto) automatic notification for user about arriving on pages of a past course|
A+ knows that a course has closed since course instances have opening and closing times. Any closed course instance could show a warning easily. Adding an automatic link to the newest course instance would be easy too.
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/554
|days||easy||summer 2020 (priority 2)||summer intern: Saara|
|13||(Otto) "More granular control of hiding/disabling exercises. When updating a grader / during overload / etc. We could show the exercise, and just store the submissions for future grading. If the grader is being maintained, we could still show the exercise normally"|
This needs to be clarified with Otto
Does the teacher really want to manually configure these details? (E.g., exercise is in a state that accepts submissions but nothing is sent to the grader.)
I don't think otto wants to control this him self, but for the system to be able to - Jaakko
In order to do this, A+ should have background tasks for sending a large batch of submissions to grading (or mass regrading). It can't be done synchronously during the HTTP request from a browser. This background task would support the mass regrading feature too (when you want to regrade all broken submissions from a certain time).
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/555
|weeks||moderate||voted low in priority; to be saved in the barrel of wishes|
|(small?) improvements to existing features for teachers||14||(Mario) access to the enrollment questionnaire results from the frontpage (like normal modules)|
Mario wants to see the link to exercise submissions in the course front page. Currently, that is possible for normal exercises, but not for enrollment exercises. The problem is that we do not want to show the link to the enrollment questionnaire in the front page, especially not to students. The course contents (table of contents and exercise results) are rendered from the content cache and unfortunately, the cache code is already an error-prone, complex mess.
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/449
|maybe days if we ignore code quality - otherwise, more time||moderate (cache is complex)||voted low in priority; to be saved in the barrel of wishes|
|15||(Sanna) the results table should include an option to get students' email addressess in the list|
The all results (updated in December 2019) retrieves the data from the A+ API. API includes users' emails in the data, so this should be easy. (or has this been implemented already? In the export file at least?)
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/556
|days||easy||summer 2020||summer intern: Ella|
|16||(Sanna) the round results visualization should show the (number of) students who are above the minimum requirements instead of always 90%|
The plan is to migrate the visualization to the points page (and thus remove the current page). Visualization would then show what ever teacher has filtered and selected in the points list. - Jaakko
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/557
|weeks||moderate (or easy if we are lucky)||summer 2020||summer intern: Ella|
|(small?) improvements to existing features for students||17||(Juha and Otto) rst support for "one-liner" programming exercises|
This could probably already be implemented with active elements. They support text inputs within chapters and asynchronous graders that run in containers like for real exercises. You just need try it out.
In addition, O1 already uses some one line exercises, where students fills in the blanks. - Jaakko
Needs to be specified (there should be a GH issue on this?)
ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/558
|days||easy (if active elements are adequate)||summer 2020: Instructions|
maybe summer intern
|New features on observing student behavior during the course||18||(Juha and Otto) time usage stats for Jutut (showing students' workload per chapter and per module)|
Isn't this what Riku was tasked to do in the spring 2020? He completed the project, but Jaakko has no time to review and merge it to MOOC-Jutut master. We don't yet know if Riku's code actually works and could be used in production.
The review of the code will take days of time and should be done by someone who will then continue with Jutut. We don't have anyone at the moment!
The open PR is at https://github.com/Aalto-LeTech/mooc-jutut/pull/11
|days||expert (code review requires experience)||PR commented spring 2020 - otherwise, this is a challenge!|
|19||(Otto) stats (histograms?) on multiple-choice questions to be included in the time usage stats|
Needs to be clarified with Otto
A+ already has some histograms for quiz data, though perhaps they are too unreadable or unclear in certain cases. Does Otto want these histograms in MOOC-Jutut? Why? The use case and teacher's needs must be discussed further.
Originally, this was supposed to be in Jutut, but then it wouldn't be usable with non Jutut exercises. Teemu L. implemented current simple implementation. Latest idea was to build an analytics service, where teacher would be able to configure which data is fathered and aggrecated and then the service would support export formats, e.g., grafana or power BI to visualization.
|days or weeks? Unclear use case||moderate?||voted low in priority; to be saved in the barrel of wishes|
|New features on assessment and grading||20||(Juha and Otto) scala grader is too slow (current dockerised containers vs dedicated servers before)|
The Scala grader is slow because Scala has a slow start up time. It needs to load large JAR libraries to memory at start up.
The old Scala grader in Goblin (using chroot sandbox) was fast because it could run a daemon in the background. Basically, Scala was always loaded in memory and there was no start up time for grading a submission. Implementing this with containers is a major challenge (or just impossible?) and requires deep understanding of Kubernetes. We start a new container for each submission for security and simplicity. Reusing the same container for many submissions is probably possible, but changes our architecture and is inherently less secure.
GraalVM was researched to solve the slow startup of scalac, but it's rather complicated with scala macros and junit uses mainly those.
Main design goals for Grader v2 is to plan container which can be reused for multiple submissions (like serverless). This work will take a lot of time and requires skills.
|weeks||expert (deep understanding of container technology)|
Depends on whether we have the skills in the team.
|21||(Otto) "Always storing the student solution even if the grading fails. Currently the graders can time out and no feedback (and sometimes no submission) is stored in A-plus. This happens especially with the git-based submissions where the grader is responsible for fetching the submission."|
-→ this is under construction (spring 2020 in the roadmap, will be partly postponed to the summer)
We can't solve the git problem though, If A+ can't connect to the grader, then the grader is unable to download the data from git. Background task in A+ would help so far that it would get the giturl to the grader as soon as the grader is working again.
That said, A+ does always store the submission, as long as it's not just an url.
spring 2020 → early summer 2020 (the first part)
|22||(Otto) grader time usage stats "which exercises are taking long to grade, where queues are forming etc"|
Jaakko is currently working on this as part of Grader "stability" stuff (spring and summer 2020). This should produce information, which can be used for analysis. Some of the analysis can be done in Grader and rest (e.g. history) by CS IT.
CS IT could set up some monitoring tools for Kubernetes that can track containers: how long they live, which container is needed the most (most submission), CPU and memory load of the container → compile statistics for each course or exercise
Something like this was already briefly discussed with CS IT. Perhaps Seppo could do something, but when will he have the time?
|weeks||expert (partly dependent on CS IT people?)||dependent on CS IT → 2020-2021?||CS IT?|
|23||(Lauri) teacher should be able to submit exercises on behalf of the student|
Good idea in serving the students. Comments
→ We need some use cases for this and then discuss the pedagogical meaningfulness, and if there are some other perspectives
Teachers can already change the submitters of existing submissions, but it does feel a little hacky. Additionally, A+ does not remember that the submission was actually submitted by the teacher. The history metadata would be nice to have if there are issues later on (which points did the student really earn and what was earned on false grounds).
A simple addition could be a new many-to-many field in the database that remembers the previous submitters of the submission.
A good solutions is planned for the next version of A+, where we log all actions done for a submission (e.g. regrading, manually changed points and so on). In this context, it should be trivial to mark a submission to be done by the teacher on behalf of the student.
|days||easy (depends on the real needs)||to be discussed in SG June 2020|
|24||(Lauri) need for flexibility in late submission penalty (also for individual exercises in addition to whole modules, and also for individual students)|
The late penalty is an available feature already. This idea would give teachers more flexilibity in using it. Comments:
Teachers have also requested that late submission penalties could be defined in different ways. The current percentage penalty from the earned points is not flexible. For example, sometimes a constant penalty is better (always -10 points) or an increasing penalty based on time (each late day increases the penalty).
→ this feature needs to be discussed in whole before doing anything
|weeks||moderate||voted low in priority; to be saved in the barrel of wishes → discussion in SG June 2020|
|25||(Sanna) opportunity to choose which rounds are used in calculating the grade (extra rounds can replace normal rounds)|
The use case needs to be clarified with Sanna
If Sanna refers to the all results page, you can already filter the table based on rounds and exercises.
Next step for end-of-course stuff, would be making the grade calculation something usable by courses (e.g. ironing out the current hidden prototype). This could be designed in integration with the points page.
GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/567
|? unclear use case||?|
first look: summer 2020
Can this be solved during 2020-2021 with the service platform?
|summer intern: Ella|
|26||(Sanna) Visualising and using Radar results should be strongly improved to support the teacher process in understanding the overall situation, in finding the problematic cases and reporting them to LES (pic attached - illustrates the non-helpful current situation)|
Radar is indeed slow and hard to use. We have no developers for it and the previous developers have left the EDIT team.
GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/566
|weeks to months||moderate||voted low in priority; to be saved in the barrel of wishes|
|27||(Tomi) additional support for remote proctoring||(what this means in practice, needs to be specified) → note also https://jira.eduuni.fi/browse/CSCIDEAPANKKI-106 (please vote!)||voted low in priority; to be saved in the barrel of wishes|
|Additional needs||28||(Panu) Running R code||ISSU IN GITHUB: https://github.com/apluslms/mooc-grader/issues/72||voted low in priority; to be saved in the barrel of wishes|
Perceived pros and cons of A+ on general level
- Perceived benefits from A+: it is flexible, customizable, easier (than MC) to maintain material, more potential than TIM
- Perceived defects in A+: It requires knowledge on RST and Github and every typo requires one minute of work, which is too much
- Discussed ideas
- need for a scalable platform (the Tampere situation; currently material in Weto and A+ - all material would be needed in the future)
- streamlining the material production process and faster compiler → these were commented on also in the survey
- personalization (multiple perspectives; some commented also in the survey)
- tracking the history of individual students
- personal learning paths
- conditional activities (assignments, exams) i.e. the second activity in the flow would appear only after the first one is completed
- personal activities (for example exam questions are shown only to students who have enrolled to the exam)
- randomization of questions within an assignment round