Teachers' needs were collected in April-May 2020 with a process like this
- teachers responded to a questionnaire and participated in one of two online meetings. Additionally, teachers have been able to write Github issues, though the opportunity has not been widely marketed yet, and the Github issues have not been included in this first round of prioritizing.
- The EDIT team read through all the ideas, commented on them and estimated their workloads and the knowledge required for the implementation
- Teachers were able to rank the ideas sorted in two lists (from 1 to 5, max 5 votes per list):
- 1) quite easy and quite minor tasks, possible to give e.g. for summer interns
- 2) large (months) tasks and tasks requiring expert knowledge
- Some results of the ranking:
- 9 teachers participated in ranking the ideas
- All ideas received at least 1 ranked vote
- Most voted ideas received rankings from 5/9 teachers (list 1) and even 6/9 teachers (list 2)
- In list 1, numbers of rankings per idea were distributed quite evenly: over half of the ideas received 4-5 votes. Top ideas in list 2 were more clearly separated.
- Based on the results, the EDIT team makes a suggestion for the summer (to be updated after our weekly meeting)
Perceived pros and cons of A+ on general level
- Perceived benefits from A+: it is flexible, customizable, easier (than MC) to maintain material, more potential than TIM
- Perceived defects in A+: It requires knowledge on RST and Github and every typo requires one minute of work, which is too much
- Discussed ideas
- need for a scalable platform (the Tampere situation; currently material in Weto and A+ - all material would be needed in the future)
- streamlining the material production process and faster compiler → these were commented on also in the survey
- personalization (multiple perspectives; some commented also in the survey)
- tracking the history of individual students
- personal learning paths
- conditional activities (assignments, exams) i.e. the second activity in the flow would appear only after the first one is completed
- personal activities (for example exam questions are shown only to students who have enrolled to the exam)
- randomization of questions within an assignment round
Teachers' ideas and their implementation evaluation
Suggested categories for
- required knowledge: beginner, easy, moderate, expert (referring to who we can estimate to have the required skills; in practice, can we give it to summer interns)
- workload estimate: days, weeks, months
|Category||Nr||(Teacher) Idea||EDIT suggestion||workload estimate||required knowledge estimate||suggested schedule||Preliminary allocation|
|student enrolment||1||(Mario) automatic enrolment based on oodi (→ sisu), like in MC|
should be done Spring 2021 to Sisu - we'll wait for Sisu.
(to note: we need an A+-Sisu-interface in general, for probably multiple needs, and the develpment needs to be started in early autumn)
|2 weeks||moderate||Academic year 2020-2021||intern?|
|material production||2||(Mario) enrolment questionnaire: dynamic questionnaire difficult to edit|
In order to make more convenient for the teacher and "a real feature" instead of a hack, we should create a new questionnaire engine and define new RST syntax/options for configuring more complex, dynamic behaviour.
|months (for a new questionnaire service that replaces all old questionnaire implementations)||expert||not prioritized; saved in the barrel of wishes|
|3||(Lauri) compilation of RST takes too much time|
option A: implement some kind of build caching to a-plus-rst-tools so that rst-tools do not require that all RST files are always built again every time. Currently, all RST files are built so that updates to the linked exercise config.yaml files are not lost. Currently, the built-in cache of Sphinx does not recognize YAML files and it would not include them in a new build if the RST file (that links to the YAML file) has no changes and is therefore not included in the new build.
option B: rewrite a-plus-rst-tools and design all features so that they align better with the core functionality of Sphinx. Perhaps, we could easily use the Sphinx build caching for YAML files if we had a separate custom YAML builder instead of the current hack that combines YAML files into the HTML builder. Additionally, the current implementation does not support the Sphinx Latex builder since HTML output has been added to components that are supposed to be builder-agnostic and work with all output formats. Basically, a-plus-rst-tools violate many Sphinx APIs, but it could be implemented in "the right way".
option C: we could recommend teachers some additional tools & instructions which would solve at least some problems.
option D: Quick hack (immediate solution): remove "make touchrst" from the docker-compile.sh script. Compiling will be then as fast as it can be in Sphinx since it uses the build cache. Exercises may be broken since YAML files may be ignored in the build, but that does not matter while writing the reading materials. Docker-compile.sh only affects local testing and not the production server.
|weeks or even months||expert||Alternative A: Autumn 2020?|
|4||(Lauri) RST is clumsy as source language|
option A: use (pilot) LaRST (Latex-to-RST), which was implemented by Tomi Janhunen at Tampere University. It compiles Latex into the RST format used by a-plus-rst-tools. The special A+ features, such as exercises, are defined with custom Latex macros/environments. Tomi has not yet released the project openly, but the project should be close to finished. One should consider if LaRST supports enough A+ features to be useful or is something crucial still missing.
option B: Jhosimar is writing new instructions in the aplus-manual that could ease the pain of writing RST a little bit. There are some hotkeys or macros in the VS code editor that could help.
|days||easy||summer 2020 1 (A+B)||Markku (A), Jhosimar (B, A)|
|5||(Lauri) opportunity to set model solution publication timestamp (instead of automatic publication after DL)|
It is easy to add a new date setting and disallow access to the model solutions before that date.
(If it is meaningful to implement this - We should consider what features are really needed before filling the system with custom date settings and switches for every small detail.)
|days||beginner||summer 2020: 2||summer intern?|
|6||(Sanna) easier way to add links to the left hand side menu|
This needs to be clarified with Sanna
There is already a menu for adding those links in A+ (under Edit course). It is unclear how it could be any easier than that. There are different types of links (LTI services, links to external web pages, links to pages inside the A+ course). Sanna should clarify what kind of difficulties she has had and in which use cases. (Admins must enable new LTI services once in the system before they can be used, which is maybe confusing for teachers. However, it is very rare to add new LTI services.)
Furthermore, menu links require domain to be registerd (even for non LTI services), so A+ knows metadata of the service. Logic: links in the menu are peers to links within application (A+), thus external links should tell the user they are leaving the site. This should happen to all A+ links (event those in chapter), but latter is not implemented. It could be possible to add possibility to add link to any site, but that should show the max level warning, as A+ can't tell the user that there is nothing to worry about.
In addition, it would be possible to add links to the course configuration (and that should be done at some point), which would then be used to configure the menu. Above point about allowing any links might be relevant in this case (or we can show an error in configuration step).
|? (clarify usecase first)||available already?|
|7||(Tomi) need for personalized exercises (emphasized in the remote study situation)|
The need is to provide different exercises for each student, particularly in online exams. Ways of implementing this can include
For example, if the student should transform a logical formula, each student should receive their own formula (depending on the focus of the randomization needs, e.g. STACK might already provide some solutions). The randomization could be e.g. generated synchronously by using the student ID (user ID) as the random seed. Tomi did not want to try the existing personalized exercises feature of the MOOC-Grader because it is clunky and requires that the server admin runs the exercise instance generator in the server before the exercises are published to the students. Tomi is more interested in synchronous "on-the-fly" generation, but this requires that even opening the exercise description includes some custom, exercise-specific computation in the MOOC-Grader server. This affects the server's performance negatively and it is a challenge to execute the teacher's custom code safely at the time exercise description is loaded. Launching a container for safe code execution at that time might be way too slow. This is, per request it would not be meaningful → serverless as alternative: As a long term solution, we could use serverless code for form generation (see grader v2 ideas). In addition, the A+ protocol should be updated to support initial seed, which would be stored in A+ and returned to grader later on. The seed part can be hacked using signed data in the form (e.g. JWT). Though, every time server secret changes, the old forms can not be submitted or regraded.
To note: teachers have very different kinds of ideas what personalisation really means, so there are probably other use cases for personalized exercises.
|months (or 2 weeks for a simple, insecure, hacky implementation)||expert (or moderate for simple, insecure solutions)||Autumn 2020?||Markku?|
The URLs to A+ course materials should include the language code (en/fi) so that people can make links to the specific language version of the content. Django has a framework for this need, but adding it to the old code will cause some surprises that have to be solved.
Google indexing is also currently broken because the same URL is used for basically different pages; the English and Finnish versions are completely separate from each other from Google's perspective.
The google indexing could be fixed by building the site index with language keys for goole and by adding support for `?lang=fi` in the urls.
|weeks||moderate||summer 2020: 1||summer intern?|
|material availability/access||9||(Mario and Tomi) possibility to set the material accessibility based on needs (public, only to enrolled students, individual students), like in MC|
Whole courses can currently be restricted: public, authenticated user, only internal or external user, only enrolled used.
Individual chapters can also be restricted to only internal or external users.
There is currently no support for student-specific access restrictions. A good implementation for that requires consideration of caching (for performance) and configuration (how are the chosen students defined). Is it enough to have student restrictions for whole modules? Or should it be specific to chapters? Note, usertags could be used for selecting show/hide groups.
|weeks||moderate||summer 2020: 1||summer intern?|
|10||(Jan-Mikael) conditional exercises (student must compelete first exercise in order to see the next one), like in MC|
The need is to restrict access to exercises; for example,
All students do the same exercises. The idea is to restrict the order in which students can take the exercises.
|weeks/months||moderate/expert||Academic year 2020-2021||?|
|11||(Otto) it should be possible to select what is and what is not indexed||Google can only index public course contents. If you want to hide some of the public content from Google indexing, then that would be a new feature. This should maybe be configured in the web server (robots.txt?) and not really in Django, but then how can we do it dynamically based on course configurations?||days||moderate||summer 2020: 2||summer intern?|
|12||(Otto) automatic notification for user about arriving on pages of a past course||A+ knows that a course has closed since course instances have opening and closing times. Any closed course instance could show a warning easily. Adding an automatic link to the newest course instance would be easy too.||days||easy||summer 2020: 2||summer intern?|
|13||(Otto) "More granular control of hiding/disabling exercises. When updating a grader / during overload / etc. We could show the exercise, and just store the submissions for future grading. If the grader is being maintained, we could still show the exercise normally"|
This needs to be clarified with Otto
Does the teacher really want to manually configure these details? (E.g., exercise is in a state that accepts submissions but nothing is sent to the grader.)
I don't think otto wants to control this him self, but for the system to be able to - Jaakko
In order to do this, A+ should have background tasks for sending a large batch of submissions to grading (or mass regrading). It can't be done synchronously during the HTTP request from a browser. This background task would support the mass regrading feature too (when you want to regrade all broken submissions from a certain time).
|weeks||moderate||not priorized; saved in the barrel of wishes|
|(small?) improvements to existing features for teachers||14||(Mario) access to the enrolment questionnaire results from the frontpage (like normal modules)||Mario wants to see the link to exercise submissions in the course front page. Currently, that is possible for normal exercises, but not for enrollment exercises. The problem is that we do not want to show the link to the enrollment questionnaire in the front page, especially not to students. The course contents (table of contents and exercise results) are rendered from the content cache and unfortunately, the cache code is already an error-prone, complex mess.||maybe days if we ignore code quality - otherwise, more time||moderate (cache is complex)||not priorized; saved in the barrel of wishes|
|15||(Sanna) the results table should include an option to get students' email addressess in the list||The all results (updated in December 2019) retrieves the data from the A+ API. API includes users' emails in the data, so this should be easy. (or has this been implemented already? In the export file at least?)||days||easy||not priorized; saved in the barrel of wishes|
|16||(Sanna) the round results visualization should show the (number of) students who are above the minimum requirements instead of always 90%|
The plan is to migrate the visualization to the points page (and thus remove the current page). Visualization would then show what ever teacher has filtered and selected in the points list. - Jaakko
|weeks||moderate (or easy if we are lucky)||not priorized; saved in the barrel of wishes|
|(small?) improvements to existing features for students||17||(Juha and Otto) rst support for "one-liner" programming exercises|
This could probably already be implemented with active elements. They support text inputs within chapters and asynchronous graders that run in containers like for real exercises. You just need try it out.
In addition, O1 already uses some one line exercises, where students fills in the blanks. - Jaakko
|days||easy (if active elements are adequate)||summer 2020: 1||summer intern?|
|New features on observing student behavior during the course||18||(Juha and Otto) time usage stats for Jutut (showing students' workload per chapter and per module)||Isn't this what Riku was tasked to do in the spring 2020? He completed the project, but Jaakko has no time to review and merge it to MOOC-Jutut master. We don't yet know if Riku's code actually works and could be used in production.||days||expert (code review requires experience)||summer 2020?||Jaakko|
|19||(Otto) stats (histograms?) on multiple-choice questions to be included in the time usage stats|
Needs to be clarified with Otto
A+ already has some histograms for quiz data, though perhaps they are too unreadable or unclear in certain cases. Does Otto want these histograms in MOOC-Jutut? Why? The use case and teacher's needs must be discussed further.
Originally, this was supposed to be in Jutut, but then it wouldn't be usable with non Jutut exercises. Teemu L. implemented current simple implementation. Latest idea was to build an analytics service, where teacher would be able to configure which data is fathered and aggrecated and then the service would support export formats, e.g., grafana or power BI to visualization.
|days or weeks? Unclear use case||moderate?||not priorized; saved in the barrel of wishes|
|New features on assessment and grading||20||(Juha and Otto) scala grader is too slow (current dockerised containers vs dedicated servers before)|
The Scala grader is slow because Scala has a slow start up time. It needs to load large JAR libraries to memory at start up.
The old Scala grader in Goblin (using chroot sandbox) was fast because it could run a daemon in the background. Basically, Scala was always loaded in memory and there was no start up time for grading a submission. Implementing this with containers is a major challenge (or just impossible?) and requires deep understanding of Kubernetes. We start a new container for each submission for security and simplicity. Reusing the same container for many submissions is probably possible, but changes our architecture and is inherently less secure.
GraalVM was researched to solve the slow startup of scalac, but it's rather complicated with scala macros and junit uses mainly those.
Main design goals for Grader v2 is to plan container which can be reused for multiple submissions (like serverless). This work will take a lot of time and requires skills.
|weeks||expert (deep understanding of container technology)|
Depends on when we have the skills in the team.
|21||(Otto) "Always storing the student solution even if the grading fails. Currently the graders can time out and no feedback (and sometimes no submission) is stored in A-plus. This happens especially with the git-based submissions where the grader is responsible for fetching the submission."|
-→ this is under construction (spring 2020 in the roadmap, will be partly postponed to the summer)
We can't solve the git problem though, If A+ can't connect to the grader, then the grader is unable to download the data from git. Background task in A+ would help so far that it would get the giturl to the grader as soon as the grader is working again.
That said, A+ does always store the submission, as long as it's not just an url.
|spring 2020 → summer 2021||Jaakko|
|22||(Otto) grader time usage stats "which exercises are taking long to grade, where queues are forming etc"|
Jaakko is currently working on this as part of Grader "stability" stuff (spring and summer 2020). This should produce information, which can be used for analysis. Some of the analysis can be done in Grader and rest (e.g. history) by CS IT.
CS IT could set up some monitoring tools for Kubernetes that can track containers: how long they live, which container is needed the most (most submission), CPU and memory load of the container → compile statistics for each course or exercise
Something like this was already briefly discussed with CS IT. Perhaps Seppo could do something, but when will he have the time?
|weeks||expert (partly dependent on CS IT people?)||dependent on CS IT - could this be allocated for 2020-2021 anyway?||CS IT?|
|23||(Lauri) teacher should be able to submit exercises on behalf of the student|
Good idea in serving the students. Comments
→ We need some use cases for this and then discuss the pedagogical meaningfulness, and if there are some other perspectives
Teachers can already change the submitters of existing submissions, but it does feel a little hacky. Additionally, A+ does not remember that the submission was actually submitted by the teacher. The history metadata would be nice to have if there are issues later on (which points did the student really earn and what was earned on false grounds).
A simple addition could be a new many-to-many field in the database that remembers the previous submitters of the submission.
A good solutions is planned for the next version of A+, where we log all actions done for a submission (e.g. regrading, manually changed points and so on). In this context, it should be trivial to mark a submission to be done by the teacher on behalf of the student.
|days||easy (depends on the real needs)||to be discussed in SG June 2020|
|24||(Lauri) need for flexibility in late submission penalty (also for individual exercises in addition to whole modules, and also for individual students)|
The late penalty is an available feature already. This idea would give teachers more flexilibity in using it. Comments:
Teachers have also requested that late submission penalties could be defined in different ways. The current percentage penalty from the earned points is not flexible. For example, sometimes a constant penalty is better (always -10 points) or an increasing penalty based on time (each late day increases the penalty).
→ this feature needs to be discussed in whole before doing anything
|weeks||moderate||not priorized; barrel of wishes|
|25||(Sanna) opportunity to choose which rounds are used in calculating the grade (extra rounds can replace normal rounds)|
The use case needs to be clarified with Sanna
If Sanna refers to the all results page, you can already filter the table based on rounds and exercises.
Next step for end-of-course stuff, would be making the grade calculation something usable by courses (e.g. ironing out the current hidden prototype). This could be designed in integration with the points page.
|? unclear use case||?||Can this be solved during 2020-2021 with the service platform?|
|26||(Sanna) Visualising and using Radar results should be strongly improved to support the teacher process in understanding the overall situation, in finding the problematic cases and reporting them to LES (pic attached - illustrates the non-helpful current situation)||Radar is indeed slow and hard to use. We have no developers for it and the previous developers have left the EDIT team.||weeks to months||moderate||not priorized; saved in the barrel of wishes|
|27||(Tomi) additional support for remote proctoring||(what this means in practice, needs to be specified) → note also https://jira.eduuni.fi/browse/CSCIDEAPANKKI-106 (please vote!)||not priorized; saved in the barrel of wishes|
|Additional needs||28||(Panu) Running R code||not priorized; saved in the barrel of wishes|