A+ training / workshop for summer interns 2020!
- Wed 3.6.2020 at 9.30-12.00
- in Zoom at https://aalto.zoom.us/j/63612675337
- How to use A+, how to create a course, where do I find instructions, and the questions I have on A+
- Additionally, in the start Jaakko will host a minimal intro to using git
Send an email to email@example.com → we can also support you via short online meetings!
- If you need a laptop, screen, mouse, keyboard, something else (you can of course use your own computer, as well!)
- You should have received an email message from HR sent on 20th May - please follow the instructions in the message!
- Pick up the equipment from CS IT according to the instructions and fill the Doodle beforehand (and if none of the times work for you, contact firstname.lastname@example.org)
- To enter the campus buildings, email@example.com needs to be informed at the latest on the previous working day by 16.00
- what your laptop should include:
- Ubuntu Linux
- CS IT should have installed Docker and added the user to the docker group. In addition, read the "installing Docker Compose" section below on this page.
- The preinstalled editors are Eclipse and Vim. You can can install other editors with instructions behind the next link to scicomp (section "Ubuntu packages").
- More instructions for using Aalto Linux and how to install software: https://scicomp.aalto.fi/aalto/linux/
- Aalto VPN
- Zoom and Teams for online meetings
- If you are not using the laptop provided by us, then macOS is easy alternative. With Windows, a Linux virtual machines is the best choice.
- Coding style guide: we don't have guide of our own, but in Python, the official PEP 8 is good: https://www.python.org/dev/peps/pep-0008/
- Contribution guide, (link to be added here!)
- Git and pull request guides are on this wiki page
- Definition of Done
- Eclipse (available by default)
try this https://snapcraft.io/atom and report to Jaakko on succes/failure
- Visual Studio Code (vscode),
try this https://snapcraft.io/code and report to Jaakko on success/failure
- Download .deb package: https://go.microsoft.com/fwlink/?LinkID=760868
sudo apt install ./<file>.debin the Downloads folder (or where the file is)
- others? inform preferably CS IT in advance
- Instructions on how to change the terminal to another
Links to git repositories
Our work is scattered under the apluslms and Aalto-LeTech organisations in github.com. The Acos server has its own organisation too.
You don't need to clone or fork all of them. Do that when you need to. All repositories are listed here, so you can find them when needed.
Installing Docker Compose
Docker Compose ("docker-compose") is another tool besides Docker ("docker"). CS IT should have installed Docker into Aalto work laptops for you (i.e. there should be a package "docker-ce" and a command "docker"). Unfortunately, Ubuntu repositories have outdated versions of docker-compose, so it should be installed manually. It can be installed into the user's home directory without administration privileges. Docker-compose is just a single binary executable file.
NOTE: this is not needed on macOS as docker-compose is included in the Docker for Mac application.
Installation can be completed by copy-pasting and executing following in a terminal window. If the last lines are ok, then do logout from the desktop and log back in to make the command work without full path.
Following example shows what it should look like. Note that the script is pasted one block at a time and only last part provides any feedback. Furthermore, if the version number is the same, then the hash on the last line should be the same for you too. However, if you get a newer version, then it will be different.
Everything should work now, but if you are unsure, you can compare your local files to these examples:
- The official Docker Compose installation guide: https://docs.docker.com/compose/install/#install-compose.
- About .zprofile and .profile with zsh (the default shell in Aalto linuxes): https://superuser.com/q/187639
Show exit codes in the terminal (OPTIONAL)
This configuration is not required for your environment to work for A+ development, but it might still be useful.
Sometimes crashed and failed commands go unnoticed. To help with that, you can include program exit codes in the prompt. If you drop this ..zshrc into your home folder, then your shell will start to look like the following image.
After downloading the that file, move it to your home folder, and then rename it from
.zshrc (yes, it starts with a dot).
NOTE: this only works for zsh, thus bash users require something different (e.g. macOS users).
Use nano (i.e. not vim) for git commits with nice colors (OPTIONAL)
In this part, we configure our terminal text editor to
nano. which might be easier to use than the typical default
vim. In addition, we add custom syntax coloring to git operations.
First, let's set our default editor
~/.profile should look something like the example file in the seciton Installing Docker Compose. You need to log out and in for above changes to take effect.
Second, let's configure nano to show some colors and to react to mouse clicks. To do that, create following files:
If you need to copy text from open nano session, then you need to press SHIFT while selecting text with the mouse. You could also disable mouse by commenting out the corresponding line. Or you can add line numbers. Both parameters have comments in the file above.
For the second file, you might need to create the folder
Now, when you issue a command, which uses editor (e.g.
git commit), you should see
nano with some nice colors.
Creating Python virtual environments
Python virtual environments allow you to install Python packages (libraries and tools) without interfering with the system-wide Python installation. Each virtual environment is isolated from each other, so you can, for example, install different versions of the same package in different environments. You typically create a new venv for each app that requires a lot of dependencies, for example, a venv for a-plus and a venv for the mooc-grader, assuming you develop both systems. We recommend that the venv directory is created outside the application directory, e.g., outside the a-plus source code directory.
Since we can run A+ and the MOOC-Grader in Docker containers, we typically only need virtual environments for running certain Django (manage.py) commands like "test", "makemigrations", "makemessages", and "compilemessages".
The example below creates a new venv in the directory venv_aplus under the current working directory.
The source command is used to activate the venv again at another time (in a new terminal window or after rebooting the machine): source venv_aplus/bin/activate
A+ local development setup
Course developers usually do not modify A+ source code and they do not need to clone the a-plus git repo. They can run the latest released versions of the A+ Docker containers in order to test their courses. The container version is set in docker-compose.yml.
A+ and MOOC-grader developers need to mount the source code to the Docker containers so that they can run their own version of the source code in the container. Example paths for the volumes field in docker-compose.yml (the full example docker-compose.yml file is also below):
- The first path before the colon is the local path that you need to fix for your own computer.
- A-plus is installed in
/srv/aplusin the run-aplus-front container. You can mount development version of the source code to
/src/aplus. The container will then copy it to
/srv/aplusand compile the translation file (django.mo). If you mount directly to
/srv/aplus, you need to manually compile the translation file beforehand, but on the other hand, Django can reload the code and restart the server without restarting the whole container when you edit the source code files.
- The same applies to the run-mooc-grader container, but the paths inside the container are
- In the run-mooc-grader container, the course directory should be mounted to
/srv/courses/default. If you mount the mooc-grader code to
/srv/grader, then you need to change the course directory mount to
aplus/local_settings.py is needed when you run Django manage.py commands (such as makemessages, compilemessages, makemigrations). It should contain the settings below. Note that when you run the run-aplus-front container, it does not use the
local_settings.py file from the root of the a-plus source code.
Git tutorials and guides
- Linux Shell tutorial from the CS IT: https://scicomp.aalto.fi/training/linux-shell-tutorial/
- We recommend that everyone knows how to use git in the command-line. GUI's support most basic features, but some more complex operations are typically missing.
- Basics in the first two chapters of the Pro Git book: https://git-scm.com/book/en/v2
- Set your name and email to git: https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration
- You might want to change the default text editor for writing commit messages. Nano is a simple editor if you are not very familiar with the command line:
git config --global core.editor "nano" (or see Use nano for git commits above).
- Reminder of git commands (and some workflows to manage changes):
- You must understand the following commands before you can do anything: git status, git log, git add, git commit, git pull, git push, git diff (--cached)
- Branches and merging (entire chapter 3): https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell
- Branches are crucially important for separating your work from the upstream master version while you are working on your feature in a topic branch. You must understand the commands: git fetch, git checkout, git branch, git merge
git lolfor visualizing the branches:
git config --global --add alias.lol "log --graph --decorate --pretty=oneline --abbrev-commit --all" (source)
- Rebasing: https://git-scm.com/book/en/v2/Git-Branching-Rebasing
- Rewriting history (squash, change commit messages, reorder commits, interactive rebase): https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History
- git reset can be used to:
- remove staged changes (reverse of git add)
- remove changes in working tree (git reset --hard), though use
git checkout filename, so you don't make mistakes
- "delete" your local commits from a branch when the local branch has a different history than the remote and you want to use the remote version (i.e. change where the branch points to)
- Git submodules (course repositories add the a-plus-rst-tools repo as a submodule): https://git-scm.com/book/en/v2/Git-Tools-Submodules
- Stashing, i.e., saving work and removing it from the work directory without committing: https://git-scm.com/book/en/v2/Git-Tools-Stashing-and-Cleaning
- GitHub.com pull requests: https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests
- Git tutorial from the DigitalOcean community: https://www.digitalocean.com/community/tutorials/how-to-contribute-to-open-source-getting-started-with-git
- Git does not remove commits, so if you lose changes, but did commit them at some point, you have ~30 days to get them back. Contact a git guru!
- Finally, check pull-request guides below.
This is NOT a tutorial. If you are unfamiliar with git or unsure of what you are doing, read some tutorials and/or ask for help. If you are not sure what a command does, read the documentation.
Reading documentation in the terminal:
git <command> --help OR
man git-<command> OR
man git <command>
COMMANDS EXAMPLE (where 'origin' is the upstream and 'fork' my remote repository)
git fetch <remote> git fetch origin
git checkout <branch to be updated> git checkout master
git merge --ff-only <remote>/<branch> <branch to be updated> git merge --ff-only origin/master master
# OPTIONAL: if you wish to update your fork as well
git push <your remote fork> <branch> git push fork master
# start with updating your local repo if it's not up to date
git checkout <branch or commit where you wish to begin your new branch>
git checkout -b <name of new branch>
# make wanted changes
git add <file> # do this for all wanted files, if want just part of a file, use -p
# if you want to add all tracked files, use -u
# write commit message
# Make new changes and commits as needed
git push <your remote> <branch>
If ready for a pull request, you can create it in GitHub
# start with cleaning the branch history if applicable (no PR yet, but there are fixups or things to squash)
git checkout <branch>
git rebase <branch to rebase onto> # Example: git rebase master
git checkout <branch>
# make the desired changes
git add <file> # for all the changed files
If an entirely new commit:
# write commit message
If a fixup:
git commit --fixup=<commit>
git push <your remote> <branch>
git checkout <branch to clean up>
git rebase -i <commit or tag or branch that is before the things you want to clean>
# if you have fixup-commits, you can also use:
# git rebase -i --autosquash <commit>
# reorder commits to wished order, mark commits that should be squashed to the previous ones with f or s appropriately, etc
# In Nano, CTRL+K (cuts the line you are currently on) and CTRL+U (pastes the line) may be very useful for reordering the commits
# if the pull request is done and approved, then we can force push the cleaned branch
git push -f <your remote fork> <branch>
- CTRL+Z pauses whatever you had in progress
- You can see what all you have on pause with the command
- If you have just one job on pause, you can return to it with the command
- If you have several jobs on pause, you can return to the job you wish with the command fg %2, where the number is the number of the job
- You can see what all you have on pause with the command
- CTRL+S stops the connection between the keyboard and the terminal, all keypresses are stored in a buffer
- CTRL+Q resumes the connection (and executes everything in the buffer)
- If you have written something that's in the buffer that you DO NOT want to be executed, you can just close the terminal tab or terminal
Django tutorials (Python web framework)
- Django is the Python web framework used in A+ and the MOOC-Grader. A web framework contains interfaces to functionalities that are often needed in the web (backend) development, such as HTTP requests and responses, databases (object-relational mapper) and rendering (HTML) templates.
- Web software development course at Aalto: https://mycourses.aalto.fi/course/view.php?id=24331 and https://plus.cs.aalto.fi/wsd/2019-2020/
- Basic tutorial: https://docs.djangoproject.com/en/2.2/intro/
- Topics that most people should probably understand:
- database, models, making queries
- HTTP requests, view classes, generic view classes
- user authentication
- Forms, form validation
- (HTML) templates, Django template language
- Migrations, how to generate migrations after you change the fields in model classes
- unit testing
- security (HTML escaping in templates, XSS, CSRF, SQL injection, access control): https://docs.djangoproject.com/en/2.2/topics/security/
Manual testing is always required as well even if you had a lot of automated tests. Mount your source code into the A+ and MOOC-Grader Docker containers and run them in a test course. Think of what can be tested automatically and what should be tested manually.
- It is easier to run these Django commands in your host machine than in a container. Thus, you need to create a Python virtual environment (instructions are located in this page) and install the requirements (pip packages) from the project (A+ or MOOC-Grader) into the venv.
- Running unit tests in a Django project: python manage.py test
- To save time, you can run only a certain test suite with parameters: python manage.py test exercise.tests.ExerciseTest.test_base_exercise_absolute_url
- New features should be accompanied with adequate unit tests. Create tests for at least the most important cases and also edge cases.
- We don't have enough automatic tests for our old code base, but at least we should test new features better and always include automatic tests.
Selenium tests run a browser such as Firefox and execute interactions in the web page according to the test code. They test if the page behaves correctly. Selenium tests are a form of functional testing that tests the system from the end users perspective instead of testing internal functions of the system like unit testing does.
- A+ has some old Selenium tests. They cover only a small part of the system.
- Selenium tests are a bit difficult to write and they may easily break when the HTML code of the pages is changed again later.
- Let's use Selenium tests for only the most important parts of the system, especially when bugs are hard to find manually. For example, end-of-course features, results/points page that teachers use for final grading of the course.
- A+ has some documentation about its Selenium tests: https://github.com/apluslms/a-plus/tree/master/selenium_test
Git and pull request guidelines
- git commit messages are important and they should explain the reason and background for the changes: https://mislav.net/2014/02/hidden-documentation/
- Pull Request Etiquette: https://gist.github.com/mikepea/863f63d6e37281e329f8
- Responsibilities of author (person who creates the pull request) and reviewer (who accepts it)
- Includes good reasoning why good commit messages are important (in addition to the "hidden documentation" above)
- Though, doesn't include how to give positive comments
- About the style of comments and behaviour on pull request https://medium.com/better-programming/pull-request-etiquettes-for-reviewer-and-author-f4e80360f92c
- There is a pull request template in the A+ repositories. It is used in the pull request description when you make a new pull request. The template reminds you to write clear a description and to discuss how the code has been and should be tested.
- The pull request title should be written in the imperative form: "Add this feature", NOT "Adds some feature" or "This PR adds a feature".
- The pull request description is written like normal text. Be clear and concise, but include all relevant information so that readers understand the topic and the need for the changes. Compare writing the description to writing email messages.
- Naming the git branch for the pull request: the name does not technically affect anything, but it is best to use very short and clear names that describe the topic of the pull request. You will more easily recognize your own git branches later when they have descriptive names. You may include the issue number in the branch name if you like that.
- You create new PRs for new features and fixes that logically belong together. So, one PR for one feature. The contents of one PR should constitute a logical set of changes for the whole feature. A pull request does not always add a new feature, since there are also bug fixes and other necessary changes without adding new features.
- You modify the PR by pushing new commits when you fix bugs and other issues that have been pointed out in the code review and testing.
- You should clean up the git history before creating the PR. While the PR is being reviewed, you make changes in new commits and do not squash so that reviewers can easily follow the latest changes (the newest commits). At the end when everything has been polished, you clean up the git history again (git rebase, squash etc.) and force-push.
- There can be more than one commit. Each commit should constitute a sensible set of changes that somewhat belong together. The code should usually always "work" after each commit, so don't leave the code in a completely broken state after a commit (syntax errors, immediate crashes when executing the code). Of course, not every detail can work after a single commit when large features are split into many commits, but there are usually logical subcomponents or such.
- Massive commits are hard to read when developers track changes afterwards (for example, while bug hunting). A large number of very small commits is also hard to digest.
Reviewing someone else's pull request:
- Start a review in the pull request (Github has a button for that) and add comments to the code lines that you want to discuss. You may, for example, suggest changes, ask about unclear things (that should maybe be implemented better or clarified with a comment), and point out possible bugs.
- It is possible to write comments in Github without starting a review, but we prefer to start a review when you are really reviewing the whole pull request. Individual comments could be used if you had to comment on something without reviewing the whole pull request.
- The pull request discussion tab is used for comments that are not tied to any specific line of code.
- When there are conversations on certain code lines, either the developer or the reviewer could resolve the conversation, which will collapse it (it can still be opened later, however). There is no strict rule who should resolve those conversations. If the developer can be certain that he/she has finished the requested changes correctly, then the developer could resolve it after pushing the changes. Otherwise, the reviewer or the manager could resolve the conversation after verifying that the code has been fixed correctly.
UPDATE 26.6.2020 based on discussion (TODO: clean up the pull request and issue guide)
- There should normally be a Github issue about something before you implement the changes and submit a pull request.
- Github issue should describe a problem and usecase (what the user needs or wants and why). The issue should not be a direct statement "do this".
- We have many old issues that are orders ("do this"), but then it is important to find out the actual problem. That can be discussed in the issue. The original idea ("do this in that way") may actually be bad and the problem should be solved with a different approach.
- In the issue, discuss ideas about how to solve the problem and what kind of implementation would be good. Describe the overall idea of the implementation before you start coding. This way, we can agree on the approach before you write a lot of code and submit a pull request.
- Since we have agreed about the approach for implementation before submitting the PR, we won't argue about the approach in the PR discussion.
- The PR discussion and review can concentrate on smaller issues and details in the code (like coding style, inefficient code, or security problems). The overall architecture should already be good since it has been agreed on in the issue discussion. Remember to link the PR to the issue.
- The PR discussion should stay on topic and discuss the changes of the PR. Discussion about the actual problem/feature is done in the issue, not the PR. Usually, the discussion about the problem should be completed before submitting the PR and there should be no need to debate about the original problem anymore in the PR discussion.
- If new concerns appear during the PR discussion that are not directly related to the original issue, then a new issue can be created from those. Let's keep the PR discussion on topic!
- Draft pull requests can be used for sharing coding ideas at an early stage. For example, if you want to discuss some code design details before finishing the code, then a draft PR is a good way to share the code for others and discuss about it.
- Draft PRs are not ready for normal, full reviews and no such reviews should be done on them.
- If you have a draft PR, you can force-push finished code to the git branch in the end and convert it to a normal pull request.
- Normal pull requests are ready for review. Reviews may suggest changes to the code, but massive changes should not be needed at this stage since the problem has been discussed in the issue beforehand.
- After reviews have been done and the fixed code is finished, the git history shall be cleaned. Small fixes should usually be squashed into the appropriate commits and the code may need to be rebased onto the upstream master.
Jaakko's comment on 26.6.2020:
Goal of the process is to ensure, there wouldn't be huge problems at the PR time, but that still can happen. Sometimes it might happen that we need to move back to the issue, discuss it with the new information in mind, and choose what should be done if anything. With a clear idea about the problem, use case and an implementation idea, that should happen too often though.
In short, the development, implementation or the review can find information, which was not available during the discussion about the issue. If that happens, I think a good idea would be commenting in the issue "The code/thing/idea (link to code if relevant) made me think that have we have not consider thing Y" or such. This can mean: PR is closed (if it's way of the new plan), PR is redirected, PR is accepted, but it doesn't close the issue and another PR is created to continue. Keep in mind that new issues should be created if that is more suitable.
(Interestingly, original paper about waterfall development model described a process, where you can return to previous steps if that is need )