Page History

should be done Spring 2021 to Sisu - we'll wait for Sisu.

(to note: we need an A+-Sisu-interface in general, for probably multiple needs, and the develpment needs to be started in early autumn)

GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/550

Mario probably refers to the hacky aplus-enrollment-questionnaire whose JavaScript code requires that some question keys are hardcoded in the JS. The JS adds the dynamic behaviour to the questionnaire: hiding/showing questions based on answers to the previous questions.

https://github.com/Aalto-LeTech/aplus-enrollment-questionnaire/

In order to make more convenient for the teacher and "a real feature" instead of a hack, we should create a new questionnaire engine and define new RST syntax/options for configuring more complex, dynamic behaviour.

GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/551

option A: implement some kind of build caching to a-plus-rst-tools so that rst-tools do not require that all RST files are always built again every time. Currently, all RST files are built so that updates to the linked exercise config.yaml files are not lost. Currently, the built-in cache of Sphinx does not recognize YAML files and it would not include them in a new build if the RST file (that links to the YAML file) has no changes and is therefore not included in the new build.

option B: rewrite a-plus-rst-tools and design all features so that they align better with the core functionality of Sphinx. Perhaps, we could easily use the Sphinx build caching for YAML files if we had a separate custom YAML builder instead of the current hack that combines YAML files into the HTML builder. Additionally, the current implementation does not support the Sphinx Latex builder since HTML output has been added to components that are supposed to be builder-agnostic and work with all output formats. Basically, a-plus-rst-tools violate many Sphinx APIs, but it could be implemented in "the right way".

option C: we could recommend teachers some additional tools & instructions which would solve at least some problems. VS Code is potentially one of the most suitable editors which can compile individual files as fast as possible.

option D: Quick hack (immediate solution): remove "make touchrst" from the docker-compile.sh script. Compiling will be then as fast as it can be in Sphinx since it uses the build cache. Exercises may be broken since YAML files may be ignored in the build, but that does not matter while writing the reading materials. Docker-compile.sh only affects local testing and not the production server.

GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/561 (closed) →

Summer 2020: Instructions on  D, recommend C in the instructions

Later, if needed, i.e. based on additional teacher feedback: Alternative B (should be done "at some point")

D and C: Jhosimar and Markku

option A: use (pilot) LaRST (Latex-to-RST), which was implemented by Tomi Janhunen at Tampere University. It compiles Latex into the RST format used by a-plus-rst-tools. The special A+ features, such as exercises, are defined with custom Latex macros/environments. Tomi has not yet released the project openly, but the project should be close to finished. One should consider if LaRST supports enough A+ features to be useful or is something crucial still missing.

NOTE: if you are interested in piloting, please contact Tomi!

option B:Jhosimar is writing new instructions in the aplus-manual that could ease the pain of writing RST a little bit. There are some hotkeys or macros in the VS code editor that could help.

GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/562

summer 2020: B

later = when Tomi is ready to publish: A

It is easy to add a new date setting and disallow access to the model solutions before that date. → students do not see the model solution before they submit themselves. This concerns early and late submissions. Of course, other students can share the model solution they see for their friends submitting late.

(If it is meaningful to implement this - We should consider what features are really needed before filling the system with custom date settings and switches for every small detail.)

GITHUB ISSUE: https://github.com/apluslms/mooc-grader/issues/73

Is it meaningful to add this feature; need to check with Lauri.

if yes: summer 2020

This needs to be clarified with Sanna

There is already a menu for adding those links in A+ (under Edit course). It is unclear how it could be any easier than that. There are different types of links (LTI services, links to external web pages, links to pages inside the A+ course). Sanna should clarify what kind of difficulties she has had and in which use cases. (Admins must enable new LTI services once in the system before they can be used, which is maybe confusing for teachers. However, it is very rare to add new LTI services.)

Furthermore, menu links require domain to be registerd (even for non LTI services), so A+ knows metadata of the service. Logic: links in the menu are peers to links within application (A+), thus external links should tell the user they are leaving the site. This should happen to all A+ links (event those in chapter), but latter is not implemented. It could be possible to add possibility to add link to any site, but that should show the max level warning, as A+ can't tell the user that there is nothing to worry about.

In addition, it would be possible to add links to the course configuration (and that should be done at some point), which would then be used to configure the menu. Above point about allowing any links might be relevant in this case (or we can show an error in configuration step).

GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/563

The need is to provide different exercises for each student, particularly in online exams. Ways of implementing this can include

• using one exercise including variables in e.g. calculations or formula, and randomizing the variables for each student
• using multiple exercises and randomizing the whole exercises for each student

For example, if the student should transform a logical formula, each student should receive their own formula (depending on the focus of the randomization needs, e.g. STACK might already provide some solutions). The randomization could be e.g. generated synchronously by using the student ID (user ID) as the random seed. Tomi did not want to try the existing personalized exercises feature of the MOOC-Grader because it is clunky and requires that the server admin runs the exercise instance generator in the server before the exercises are published to the students. Tomi is more interested in synchronous "on-the-fly" generation, but this requires that even opening the exercise description includes some custom, exercise-specific computation in the MOOC-Grader server. This affects the server's performance negatively and it is a challenge to execute the teacher's custom code safely at the time exercise description is loaded. Launching a container for safe code execution at that time might be way too slow. This is, per request it would not be meaningful → serverless as alternative: As a long term solution, we could use serverless code for form generation (see grader v2 ideas). In addition, the A+ protocol should be updated to support initial seed, which would be stored in A+ and returned to grader later on. The seed part can be hacked using signed data in the form (e.g. JWT). Though, every time server secret changes, the old forms can not be submitted or regraded.

To note: teachers have very different kinds of ideas what personalisation really means, so there are probably other use cases for personalized exercises. 

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/552

• (Juha) unique links to language-specific pages in multilingual courses
• (Juha and Otto) indexing and search of public contents (google does not index properly)
• Otto: "Finding our material should be easier"

The URLs to A+ course materials should include the language code (en/fi) so that people can make links to the specific language version of the content. Django has a framework for this need, but adding it to the old code will cause some surprises that have to be solved.

Google indexing is also currently broken because the same URL is used for basically different pages; the English and Finnish versions are completely separate from each other from Google's perspective.

The google indexing could be fixed by building the site index with language keys for goole and by adding support for `?lang=fi` in the urls.

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/520

Plan A: summer 2020

Plan B: postponed to later

summer intern: Veera

Whole courses can currently be restricted: public, authenticated user, only internal or external user, only enrolled used.

Individual chapters can also be restricted to only internal or external users.

There is currently no support for student-specific access restrictions. A good implementation for that requires consideration of caching (for performance) and configuration (how are the chosen students defined). Is it enough to have student restrictions for whole modules? Or should it be specific to chapters? Note, usertags could be used for selecting show/hide groups.

Needs to be clarified, what the teachers actually would need in addition to what already exists.

summer 2020:  add to instructions.

To be clarified, if something else needs to be done.

Instructions on the existing alternatives: Jhosimar

if need to impelement: summer intern?

The need is to restrict access to exercises; for example,

• first, the student can see only exercise 1
• after completing exercise 1, exercises 2 and 3 open
• after completing exercise 2, exercise 4 opens
• after completing exercises 2-4, all the rest of the exercises are opened.

All students do the same exercises. The idea is to restrict the order in which students can take the exercises.

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/553

Google can only index public course contents. If you want to hide some of the public content from Google indexing, then that would be a new feature. This should maybe be configured in the web server (robots.txt?) and not really in Django, but then how can we do it dynamically based on course configurations?

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/520

A+ knows that a course has closed since course instances have opening and closing times. Any closed course instance could show a warning easily. Adding an automatic link to the newest course instance would be easy too.

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/554

This needs to be clarified with Otto

Does the teacher really want to manually configure these details? (E.g., exercise is in a state that accepts submissions but nothing is sent to the grader.)

I don't think otto wants to control this him self, but for the system to be able to - Jaakko

In order to do this, A+ should have background tasks for sending a large batch of submissions to grading (or mass regrading). It can't be done synchronously during the HTTP request from a browser. This background task would support the mass regrading feature too (when you want to regrade all broken submissions from a certain time).

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/555

Mario wants to see the link to exercise submissions in the course front page. Currently, that is possible for normal exercises, but not for enrollment exercises. The problem is that we do not want to show the link to the enrollment questionnaire in the front page, especially not to students. The course contents (table of contents and exercise results) are rendered from the content cache and unfortunately, the cache code is already an error-prone, complex mess.

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/449

The all results (updated in December 2019) retrieves the data from the A+ API. API includes users' emails in the data, so this should be easy. (or has this been implemented already? In the export file at least?)

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/556

The visualization page uses Teemu L.'s own JavaScript library for rendering the graphs. We need to check if the JS code can be made more customizable. Is the 90% limit hardcoded in JS or is it actually hardcoded in the data provided by A+ API?

The plan is to migrate the visualization to the points page (and thus remove the current page). Visualization would then show what ever teacher has filtered and selected in the points list. - Jaakko

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/557

This could probably already be implemented with active elements. They support text inputs within chapters and asynchronous graders that run in containers like for real exercises. You just need try it out.

In addition, O1 already uses some one line exercises, where students fills in the blanks. - Jaakko

Needs to be specified (there should be a GH issue on this?)

ISSUE IN GITHUB: https://github.com/apluslms/a-plus/issues/558

Instructions: Jhosimar

maybe summer intern

Isn't this what Riku was tasked to do in the spring 2020? He completed the project, but Jaakko has no time to review and merge it to MOOC-Jutut master. We don't yet know if Riku's code actually works and could be used in production.

The review of the code will take days of time and should be done by someone who will then continue with Jutut. We don't have anyone at the moment!

The open PR is at https://github.com/Aalto-LeTech/mooc-jutut/pull/11

Needs to be clarified with Otto

A+ already has some histograms for quiz data, though perhaps they are too unreadable or unclear in certain cases. Does Otto want these histograms in MOOC-Jutut? Why? The use case and teacher's needs must be discussed further.

Originally, this was supposed to be in Jutut, but then it wouldn't be usable with non Jutut exercises. Teemu L. implemented current simple implementation. Latest idea was to build an analytics service, where teacher would be able to configure which data is fathered and aggrecated and then the service would support export formats, e.g., grafana or power BI to visualization.

The Scala grader is slow because Scala has a slow start up time. It needs to load large JAR libraries to memory at start up.

The old Scala grader in Goblin (using chroot sandbox) was fast because it could run a daemon in the background. Basically, Scala was always loaded in memory and there was no start up time for grading a submission. Implementing this with containers is a major challenge (or just impossible?) and requires deep understanding of Kubernetes. We start a new container for each submission for security and simplicity. Reusing the same container for many submissions is probably possible, but changes our architecture and is inherently less secure.

GraalVM was researched to solve the slow startup of scalac, but it's rather complicated with scala macros and junit uses mainly those.

Main design goals for Grader v2 is to plan container which can be reused for multiple submissions (like serverless). This work will take a lot of time and requires skills.

Spring 2021?

Depends on whether we have the skills in the team.

-→ this is under construction (spring 2020 in the roadmap, will be partly postponed to the summer)

We can't solve the git problem though, If A+ can't connect to the grader, then the grader is unable to download the data from git. Background task in A+ would help so far that it would get the giturl to the grader as soon as the grader is working again.

That said, A+ does always store the submission, as long as it's not just an url.

spring 2020 → early summer 2020 (the first part)

Jaakko is currently working on this as part of Grader "stability" stuff (spring and summer 2020). This should produce information, which can be used for analysis. Some of the analysis can be done in Grader and rest (e.g. history) by CS IT.

CS IT could set up some monitoring tools for Kubernetes that can track containers: how long they live, which container is needed the most (most submission), CPU and memory load of the container → compile statistics for each course or exercise

Something like this was already briefly discussed with CS IT. Perhaps Seppo could do something, but when will he have the time?

Good idea in serving the students. Comments

• Anni: This may conflict with the students' responsibility of their own learning (learning to take responsibility of their studies) (ref. Aalto Code of Academic integrity, ch. 2.3) and b) maintaining equality between students - should teachers accept submissions from multiple (one official and additional inofficial) channels? It might cause the teacher more work in the long run... → I would not encourage students to submit via parallel channels, that causes a mess. In my opinion, we need to teach students to learn to take their responsibility of their actions (e.g. being late).
• Jaakko: As a principle, it should be always clear if a submission is actually done by the student, i.e., it's authored by the student. There shouldn't be a way to falsify the author (e.g. assistants shouldn't ever be able to craft a submission for a student). This doesn't mean, the system couldn't allow creating submissions which grant points for the student. Personally, I think the student should be always allowed to make a submission. Though, the submission wouldn't effect grading outside of the restrictions. This is currently out-in feature.

→ We need some use cases for this and then discuss the pedagogical meaningfulness, and if there are some other perspectives

Teachers can already change the submitters of existing submissions, but it does feel a little hacky. Additionally, A+ does not remember that the submission was actually submitted by the teacher. The history metadata would be nice to have if there are issues later on (which points did the student really earn and what was earned on false grounds).

A simple addition could be a new many-to-many field in the database that remembers the previous submitters of the submission.

A good solutions is planned for the next version of A+, where we log all actions done for a submission (e.g. regrading, manually changed points and so on). In this context, it should be trivial to mark a submission to be done by the teacher on behalf of the student.

The late penalty is an available feature already. This idea would give teachers more flexilibity in using it. Comments:

• Anni: in principle, giving penalty for submitting late may conflict university grading regulations/Code of Conduct (needs to be checked at Aalto level, if there are instructions concerning this - at UH this is very clear). Teachers can choose not to accept late submissions, because of their annual workload scheduling, but if late submissions are allowed, they have the same max points than early submissions. Only if learning to submit in time is included in the course (and thus degree) learning objectives, published before the course/at the course start with all other learning objectives, submitting late may be used as an assessment criterium.
• Markku: Adding late submission settings in the exercise level in addition to the module level is not very difficult, but it complicates the code. The late submission feature should be defined as a whole so that it makes sense. Adding small additions without thinking of the big picture has reduced the code quality in the past.

Teachers have also requested that late submission penalties could be defined in different ways. The current percentage penalty from the earned points is not flexible. For example, sometimes a constant penalty is better (always -10 points) or an increasing penalty based on time (each late day increases the penalty).

→ this feature needs to be discussed in whole before doing anything

The use case needs to be clarified with Sanna

Currently, A+ does not really have feature for computing course grades (except there is one limited and hidden feature for just that). Some courses use a JavaScript-based grade calculator that only shows a grade in the student's UI.

If Sanna refers to the all results page, you can already filter the table based on rounds and exercises.

Next step for end-of-course stuff, would be making the grade calculation something usable by courses (e.g. ironing out the current hidden prototype). This could be designed in integration with the points page.

GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/567

first look: summer 2020

Can this be solved during 2020-2021 with the service platform?

Radar is indeed slow and hard to use. We have no developers for it and the previous developers have left the EDIT team.

GITHUB ISSUE: https://github.com/apluslms/a-plus/issues/566