Page tree
Skip to end of metadata
Go to start of metadata

Teachers' needs were collected in April-May 2020 with a process like this

  • teachers responded to a questionnaire and participated in one of two online meetings. Additionally, teachers have been able to write Github issues, though the opportunity has not been widely marketed yet, and the Github issues have not been included in this first round of prioritizing.
  • The EDIT team read through all the ideas, commented on them and estimated their workloads and the knowledge required for the implementation
  • Teachers were able to rank the ideas sorted in two lists (from 1 to 5, max 5 votes per list):
    • 1) quite easy and quite minor tasks, possible to give e.g. for summer interns
    • 2) large (months) tasks and tasks requiring expert knowledge
  • Some results of the ranking:
    • 9 teachers participated in ranking the ideas
    • All ideas received at least 1 ranked vote
    • Most voted ideas received rankings from 5/9 teachers (list 1) and even 6/9 teachers (list 2)
    • In list 1, numbers of rankings per idea were distributed quite evenly: over half of the ideas received 4-5 votes. Top ideas in list 2 were more clearly separated.
  • Based on the results, the EDIT team makes a suggestion for the summer (to be updated after our weekly meeting)

Perceived pros and cons of A+ on general level

  • Perceived benefits from A+: it is flexible, customizable, easier (than MC) to maintain material, more potential than TIM
  • Perceived defects in A+: It requires knowledge on RST and Github and every typo requires one minute of work, which is too much
  • Discussed ideas
    • need for a scalable platform (the Tampere situation; currently material in Weto and A+ - all material would be needed in the future)
    • streamlining the material production process and faster compiler → these were commented on also in the survey
    • personalization (multiple perspectives; some commented also in the survey)
      • tracking the history of individual students
      • personal learning paths
      • conditional activities (assignments, exams) i.e. the second activity in the flow would appear only after the first one is completed
      • personal activities (for example exam questions are shown only to students who have enrolled to the exam)
      • randomization of questions within an assignment round

Teachers' ideas and their implementation evaluation

Suggested categories for

  • required knowledge: beginner, easy, moderate, expert (referring to who we can estimate to have the required skills; in practice, can we give it to summer interns)
  • workload estimate: days, weeks, months
CategoryNr(Teacher) IdeaEDIT suggestionworkload estimaterequired knowledge estimatesuggested schedulePreliminary allocation
student enrolment1(Mario) automatic enrolment based on oodi (→ sisu), like in MC

should be done Spring 2021 to Sisu - we'll wait for Sisu.

(to note: we need an A+-Sisu-interface in general, for probably multiple needs, and the develpment needs to be started in early autumn)

2 weeksmoderateAcademic year 2020-2021intern?
material production2(Mario) enrolment questionnaire: dynamic questionnaire difficult to edit

Mario probably refers to the hacky aplus-enrollment-questionnaire whose JavaScript code requires that some question keys are hardcoded in the JS. The JS adds the dynamic behaviour to the questionnaire: hiding/showing questions based on answers to the previous questions.

https://github.com/Aalto-LeTech/aplus-enrollment-questionnaire/

In order to make more convenient for the teacher and "a real feature" instead of a hack, we should create a new questionnaire engine and define new RST syntax/options for configuring more complex, dynamic behaviour.

months (for a new questionnaire service that replaces all old questionnaire implementations)expertnot prioritized; saved in the barrel of wishes

3(Lauri) compilation of RST takes too much time

option A: implement some kind of build caching to a-plus-rst-tools so that rst-tools do not require that all RST files are always built again every time. Currently, all RST files are built so that updates to the linked exercise config.yaml files are not lost. Currently, the built-in cache of Sphinx does not recognize YAML files and it would not include them in a new build if the RST file (that links to the YAML file) has no changes and is therefore not included in the new build.

option B: rewrite a-plus-rst-tools and design all features so that they align better with the core functionality of Sphinx. Perhaps, we could easily use the Sphinx build caching for YAML files if we had a separate custom YAML builder instead of the current hack that combines YAML files into the HTML builder. Additionally, the current implementation does not support the Sphinx Latex builder since HTML output has been added to components that are supposed to be builder-agnostic and work with all output formats. Basically, a-plus-rst-tools violate many Sphinx APIs, but it could be implemented in "the right way".

option C: we could recommend teachers some additional tools & instructions which would solve at least some problems.

option D: Quick hack (immediate solution): remove "make touchrst" from the docker-compile.sh script. Compiling will be then as fast as it can be in Sphinx since it uses the build cache. Exercises may be broken since YAML files may be ignored in the build, but that does not matter while writing the reading materials. Docker-compile.sh only affects local testing and not the production server.

weeks or even monthsexpertAlternative A: Autumn 2020?

4(Lauri) RST is clumsy as source language

option A: use (pilot) LaRST (Latex-to-RST), which was implemented by Tomi Janhunen at Tampere University. It compiles Latex into the RST format used by a-plus-rst-tools. The special A+ features, such as exercises, are defined with custom Latex macros/environments. Tomi has not yet released the project openly, but the project should be close to finished. One should consider if LaRST supports enough A+ features to be useful or is something crucial still missing.

option B: Jhosimar is writing new instructions in the aplus-manual that could ease the pain of writing RST a little bit. There are some hotkeys or macros in the VS code editor that could help.

dayseasysummer 2020 1 (A+B)Markku (A), Jhosimar (B, A)

5(Lauri) opportunity to set model solution publication timestamp (instead of automatic publication after DL)

It is easy to add a new date setting and disallow access to the model solutions before that date.

(If it is meaningful to implement this - We should consider what features are really needed before filling the system with custom date settings and switches for every small detail.)

daysbeginnersummer 2020: 2summer intern?

6(Sanna) easier way to add links to the left hand side menu

This needs to be clarified with Sanna

There is already a menu for adding those links in A+ (under Edit course). It is unclear how it could be any easier than that. There are different types of links (LTI services, links to external web pages, links to pages inside the A+ course). Sanna should clarify what kind of difficulties she has had and in which use cases. (Admins must enable new LTI services once in the system before they can be used, which is maybe confusing for teachers. However, it is very rare to add new LTI services.)

Furthermore, menu links require domain to be registerd (even for non LTI services), so A+ knows metadata of the service. Logic: links in the menu are peers to links within application (A+), thus external links should tell the user they are leaving the site. This should happen to all A+ links (event those in chapter), but latter is not implemented. It could be possible to add possibility to add link to any site, but that should show the max level warning, as A+ can't tell the user that there is nothing to worry about.

In addition, it would be possible to add links to the course configuration (and that should be done at some point), which would then be used to configure the menu. Above point about allowing any links might be relevant in this case (or we can show an error in configuration step).

? (clarify usecase first)
available already?

7(Tomi) need for personalized exercises (emphasized in the remote study situation)

The need is to provide different exercises for each student, particularly in online exams. Ways of implementing this can include

  • using one exercise including variables in e.g. calculations or formula, and randomizing the variables for each student
  • using multiple exercises and randomizing the whole exercises for each student

For example, if the student should transform a logical formula, each student should receive their own formula (depending on the focus of the randomization needs, e.g. STACK might already provide some solutions). The randomization could be e.g. generated synchronously by using the student ID (user ID) as the random seed. Tomi did not want to try the existing personalized exercises feature of the MOOC-Grader because it is clunky and requires that the server admin runs the exercise instance generator in the server before the exercises are published to the students. Tomi is more interested in synchronous "on-the-fly" generation, but this requires that even opening the exercise description includes some custom, exercise-specific computation in the MOOC-Grader server. This affects the server's performance negatively and it is a challenge to execute the teacher's custom code safely at the time exercise description is loaded. Launching a container for safe code execution at that time might be way too slow. This is, per request it would not be meaningful → serverless as alternative: As a long term solution, we could use serverless code for form generation (see grader v2 ideas). In addition, the A+ protocol should be updated to support initial seed, which would be stored in A+ and returned to grader later on. The seed part can be hacked using signed data in the form (e.g. JWT). Though, every time server secret changes, the old forms can not be submitted or regraded.

To note: teachers have very different kinds of ideas what personalisation really means, so there are probably other use cases for personalized exercises. 

months (or 2 weeks for a simple, insecure, hacky implementation)expert (or moderate for simple, insecure solutions)Autumn 2020?Markku?

8
  • (Juha) unique links to language-specific pages in multilingual courses
  • (Juha and Otto) indexing and search of public contents (google does not index properly)
  • Otto: "Finding our material should be easier"

The URLs to A+ course materials should include the language code (en/fi) so that people can make links to the specific language version of the content. Django has a framework for this need, but adding it to the old code will cause some surprises that have to be solved.

Google indexing is also currently broken because the same URL is used for basically different pages; the English and Finnish versions are completely separate from each other from Google's perspective.

The google indexing could be fixed by building the site index with language keys for goole and by adding support for `?lang=fi` in the urls.

weeksmoderatesummer 2020: 1summer intern?
material availability/access9(Mario and Tomi) possibility to set the material accessibility based on needs (public, only to enrolled students, individual students), like in MC

Whole courses can currently be restricted: public, authenticated user, only internal or external user, only enrolled used.

Individual chapters can also be restricted to only internal or external users.

There is currently no support for student-specific access restrictions. A good implementation for that requires consideration of caching (for performance) and configuration (how are the chosen students defined). Is it enough to have student restrictions for whole modules? Or should it be specific to chapters? Note, usertags could be used for selecting show/hide groups.

weeksmoderatesummer 2020: 1summer intern?

10(Jan-Mikael) conditional exercises (student must compelete first exercise in order to see the next one), like in MC

The need is to restrict access to exercises; for example,

  • first, the student can see only exercise 1
  • after completing exercise 1, exercises 2 and 3 open
  • after completing exercise 2, exercise 4 opens
  • after completing exercises 2-4, all the rest of the exercises are opened.

All students do the same exercises. The idea is to restrict the order in which students can take the exercises.

weeks/monthsmoderate/expertAcademic year 2020-2021?

11(Otto) it should be possible to select what is and what is not indexedGoogle can only index public course contents. If you want to hide some of the public content from Google indexing, then that would be a new feature. This should maybe be configured in the web server (robots.txt?) and not really in Django, but then how can we do it dynamically based on course configurations?daysmoderatesummer 2020: 2summer intern?

12(Otto) automatic notification for user about arriving on pages of a past courseA+ knows that a course has closed since course instances have opening and closing times. Any closed course instance could show a warning easily. Adding an automatic link to the newest course instance would be easy too.dayseasysummer 2020: 2summer intern?

13(Otto) "More granular control of hiding/disabling exercises. When updating a grader / during overload / etc. We could show the exercise, and just store the submissions for future grading. If the grader is being maintained, we could still show the exercise normally"

This needs to be clarified with Otto

Does the teacher really want to manually configure these details? (E.g., exercise is in a state that accepts submissions but nothing is sent to the grader.)

I don't think otto wants to control this him self, but for the system to be able to - Jaakko

In order to do this, A+ should have background tasks for sending a large batch of submissions to grading (or mass regrading). It can't be done synchronously during the HTTP request from a browser. This background task would support the mass regrading feature too (when you want to regrade all broken submissions from a certain time).

weeksmoderatenot priorized; saved in the barrel of wishes
(small?) improvements to existing features for teachers14(Mario) access to the enrolment questionnaire results from the frontpage (like normal modules)Mario wants to see the link to exercise submissions in the course front page. Currently, that is possible for normal exercises, but not for enrollment exercises. The problem is that we do not want to show the link to the enrollment questionnaire in the front page, especially not to students. The course contents (table of contents and exercise results) are rendered from the content cache and unfortunately, the cache code is already an error-prone, complex mess.maybe days if we ignore code quality - otherwise, more timemoderate (cache is complex)not priorized; saved in the barrel of wishes

15(Sanna) the results table should include an option to get students' email addressess in the listThe all results (updated in December 2019) retrieves the data from the A+ API. API includes users' emails in the data, so this should be easy. (or has this been implemented already? In the export file at least?)dayseasynot priorized; saved in the barrel of wishes

16(Sanna) the round results visualization should show the (number of) students who are above the minimum requirements instead of always 90%

The visualization page uses Teemu L.'s own JavaScript library for rendering the graphs. We need to check if the JS code can be made more customizable. Is the 90% limit hardcoded in JS or is it actually hardcoded in the data provided by A+ API?

The plan is to migrate the visualization to the points page (and thus remove the current page). Visualization would then show what ever teacher has filtered and selected in the points list. - Jaakko

weeksmoderate (or easy if we are lucky)not priorized; saved in the barrel of wishes
(small?) improvements to existing features for students17(Juha and Otto) rst support for "one-liner" programming exercises

This could probably already be implemented with active elements. They support text inputs within chapters and asynchronous graders that run in containers like for real exercises. You just need try it out.

In addition, O1 already uses some one line exercises, where students fills in the blanks. - Jaakko

dayseasy (if active elements are adequate)summer 2020: 1summer intern?
New features on observing student behavior during the course18(Juha and Otto) time usage stats for Jutut (showing students' workload per chapter and per module)Isn't this what Riku was tasked to do in the spring 2020? He completed the project, but Jaakko has no time to review and merge it to MOOC-Jutut master. We don't yet know if Riku's code actually works and could be used in production.daysexpert (code review requires experience)summer 2020?Jaakko

19(Otto) stats (histograms?) on multiple-choice questions to be included in the time usage stats

Needs to be clarified with Otto

A+ already has some histograms for quiz data, though perhaps they are too unreadable or unclear in certain cases. Does Otto want these histograms in MOOC-Jutut? Why? The use case and teacher's needs must be discussed further.

Originally, this was supposed to be in Jutut, but then it wouldn't be usable with non Jutut exercises. Teemu L. implemented current simple implementation. Latest idea was to build an analytics service, where teacher would be able to configure which data is fathered and aggrecated and then the service would support export formats, e.g., grafana or power BI to visualization.

days or weeks? Unclear use casemoderate?not priorized; saved in the barrel of wishes
New features on assessment and grading20(Juha and Otto) scala grader is too slow (current dockerised containers vs dedicated servers before)

The Scala grader is slow because Scala has a slow start up time. It needs to load large JAR libraries to memory at start up.

The old Scala grader in Goblin (using chroot sandbox) was fast because it could run a daemon in the background. Basically, Scala was always loaded in memory and there was no start up time for grading a submission. Implementing this with containers is a major challenge (or just impossible?) and requires deep understanding of Kubernetes. We start a new container for each submission for security and simplicity. Reusing the same container for many submissions is probably possible, but changes our architecture and is inherently less secure.

GraalVM was researched to solve the slow startup of scalac, but it's rather complicated with scala macros and junit uses mainly those.

Main design goals for Grader v2 is to plan container which can be reused for multiple submissions (like serverless). This work will take a lot of time and requires skills.

weeksexpert (deep understanding of container technology)

Spring 2021?

Depends on when we have the skills in the team.



21(Otto) "Always storing the student solution even if the grading fails. Currently the graders can time out and no feedback (and sometimes no submission) is stored in A-plus. This happens especially with the git-based submissions where the grader is responsible for fetching the submission."

-→ this is under construction (spring 2020 in the roadmap, will be partly postponed to the summer)

We can't solve the git problem though, If A+ can't connect to the grader, then the grader is unable to download the data from git. Background task in A+ would help so far that it would get the giturl to the grader as soon as the grader is working again.

That said, A+ does always store the submission, as long as it's not just an url.



spring 2020 → summer 2021Jaakko

22(Otto) grader time usage stats "which exercises are taking long to grade, where queues are forming etc"

Jaakko is currently working on this as part of Grader "stability" stuff (spring and summer 2020). This should produce information, which can be used for analysis. Some of the analysis can be done in Grader and rest (e.g. history) by CS IT.

CS IT could set up some monitoring tools for Kubernetes that can track containers: how long they live, which container is needed the most (most submission), CPU and memory load of the container → compile statistics for each course or exercise

Something like this was already briefly discussed with CS IT. Perhaps Seppo could do something, but when will he have the time?

weeksexpert (partly dependent on CS IT people?)dependent on CS IT - could this be allocated for 2020-2021 anyway?CS IT?

23(Lauri) teacher should be able to submit exercises on behalf of the student

Good idea in serving the students. Comments

  • Anni: This may conflict with the students' responsibility of their own learning (learning to take responsibility of their studies) (ref. Aalto Code of Academic integrity, ch. 2.3) and b) maintaining equality between students - should teachers accept submissions from multiple (one official and additional inofficial) channels? It might cause the teacher more work in the long run... → I would not encourage students to submit via parallel channels, that causes a mess. In my opinion, we need to teach students to learn to take their responsibility of their actions (e.g. being late).
  • Jaakko: As a principle, it should be always clear if a submission is actually done by the student, i.e., it's authored by the student. There shouldn't be a way to falsify the author (e.g. assistants shouldn't ever be able to craft a submission for a student). This doesn't mean, the system couldn't allow creating submissions which grant points for the student. Personally, I think the student should be always allowed to make a submission. Though, the submission wouldn't effect grading outside of the restrictions. This is currently out-in feature.

→ We need some use cases for this and then discuss the pedagogical meaningfulness, and if there are some other perspectives

Teachers can already change the submitters of existing submissions, but it does feel a little hacky. Additionally, A+ does not remember that the submission was actually submitted by the teacher. The history metadata would be nice to have if there are issues later on (which points did the student really earn and what was earned on false grounds).

A simple addition could be a new many-to-many field in the database that remembers the previous submitters of the submission.

A good solutions is planned for the next version of A+, where we log all actions done for a submission (e.g. regrading, manually changed points and so on). In this context, it should be trivial to mark a submission to be done by the teacher on behalf of the student.

dayseasy (depends on the real needs)to be discussed in SG June 2020

24(Lauri) need for flexibility in late submission penalty (also for individual exercises in addition to whole modules, and also for individual students)

The late penalty is an available feature already. This idea would give teachers more flexilibity in using it. Comments:

  • Anni: in principle, giving penalty for submitting late may conflict university grading regulations/Code of Conduct (needs to be checked at Aalto level, if there are instructions concerning this - at UH this is very clear). Teachers can choose not to accept late submissions, because of their annual workload scheduling, but if late submissions are allowed, they have the same max points than early submissions. Only if learning to submit in time is included in the course (and thus degree) learning objectives, published before the course/at the course start with all other learning objectives, submitting late may be used as an assessment criterium.
  • Markku: Adding late submission settings in the exercise level in addition to the module level is not very difficult, but it complicates the code. The late submission feature should be defined as a whole so that it makes sense. Adding small additions without thinking of the big picture has reduced the code quality in the past.

Teachers have also requested that late submission penalties could be defined in different ways. The current percentage penalty from the earned points is not flexible. For example, sometimes a constant penalty is better (always -10 points) or an increasing penalty based on time (each late day increases the penalty).

→ this feature needs to be discussed in whole before doing anything

weeksmoderatenot priorized; barrel of wishes

25(Sanna) opportunity to choose which rounds are used in calculating the grade (extra rounds can replace normal rounds)

The use case needs to be clarified with Sanna

Currently, A+ does not really have feature for computing course grades (except there is one limited and hidden feature for just that). Some courses use a JavaScript-based grade calculator that only shows a grade in the student's UI.

If Sanna refers to the all results page, you can already filter the table based on rounds and exercises.

Next step for end-of-course stuff, would be making the grade calculation something usable by courses (e.g. ironing out the current hidden prototype). This could be designed in integration with the points page.

? unclear use case?Can this be solved during 2020-2021 with the service platform?

26(Sanna) Visualising and using Radar results should be strongly improved to support the teacher process in understanding the overall situation, in finding the problematic cases and reporting them to LES (pic attached - illustrates the non-helpful current situation)Radar is indeed slow and hard to use. We have no developers for it and the previous developers have left the EDIT team.weeks to monthsmoderatenot priorized; saved in the barrel of wishes

27(Tomi) additional support for remote proctoring(what this means in practice, needs to be specified) → note also https://jira.eduuni.fi/browse/CSCIDEAPANKKI-106 (please vote!)

not priorized; saved in the barrel of wishes
Additional needs28(Panu) Running R code


not priorized; saved in the barrel of wishes
  • No labels